Skip to content
Permalink
Browse files

[MINOR][DOCS] Fix for contradiction in condition formula of keeping i…

…ntermediate state of window in structured streaming docs

This change solves contradiction in structured streaming documentation in formula which tests if specific window will be updated by calculating watermark and comparing with "T" parameter(intermediate state is cleared as (max event time seen by the engine - late threshold > T), otherwise kept(written as "until")). By further examples the "T" seems to be the end of the window, not start like documentation says firstly. For more information please take a look at my question in stackoverflow https://stackoverflow.com/questions/54599594/understanding-window-with-watermark-in-apache-spark-structured-streaming

Can be tested by building documentation.

Closes #23765 from vitektarasenko/master.

Authored-by: Viktor Tarasenko <v.tarasenko@vezet.ru>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
  • Loading branch information...
Viktor Tarasenko authored and srowen committed Feb 13, 2019
1 parent 7f44c9a commit 5894f767d1f159fc05e11d77d61089efcd0c50b4
Showing with 1 addition and 1 deletion.
  1. +1 −1 docs/structured-streaming-programming-guide.md
@@ -922,7 +922,7 @@ late data for that aggregate any more. To enable this, in Spark 2.1, we have int
**watermarking**, which lets the engine automatically track the current event time in the data
and attempt to clean up old state accordingly. You can define the watermark of a query by
specifying the event time column and the threshold on how late the data is expected to be in terms of
event time. For a specific window starting at time `T`, the engine will maintain state and allow late
event time. For a specific window ending at time `T`, the engine will maintain state and allow late
data to update the state until `(max event time seen by the engine - late threshold > T)`.
In other words, late data within the threshold will be aggregated,
but data later than the threshold will start getting dropped

0 comments on commit 5894f76

Please sign in to comment.
You can’t perform that action at this time.