Wall-Clock based Windowing / Suppression #15864
Draft
+189
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The change adds the possibility to suppress events during windowing based on WALL_CLOCK_TIME instead of STREAM_CLOCK_TIME.
Problem Statement:
In our use case we needed this functionality, as the amount of events varies very much. Sometimes we get so few events, that the event won't be emitted at all unless a new event comes in to advance STREAM_CLOCK_TIME.
Solution:
We implemented a new possibility to use WALL_CLOCK_TIME instead of STREAM_CLOCK_TIME for suppression during windowing. It is based on a scheduled operation and punctuates the clock on each interval.
Testing:
For testing we extended the
streams/src/test/java/org/apache/kafka/streams/kstream/internals/SuppressScenarioTest.java
with a new scenario where the suppression is based on wall clock time.Committer Checklist (excluded from commit message)