Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wall-Clock based Windowing / Suppression #15864

Draft
wants to merge 3 commits into
base: trunk
Choose a base branch
from

Conversation

eichingertim
Copy link

@eichingertim eichingertim commented May 5, 2024

The change adds the possibility to suppress events during windowing based on WALL_CLOCK_TIME instead of STREAM_CLOCK_TIME.

Problem Statement:
In our use case we needed this functionality, as the amount of events varies very much. Sometimes we get so few events, that the event won't be emitted at all unless a new event comes in to advance STREAM_CLOCK_TIME.

Solution:
We implemented a new possibility to use WALL_CLOCK_TIME instead of STREAM_CLOCK_TIME for suppression during windowing. It is based on a scheduled operation and punctuates the clock on each interval.

Testing:
For testing we extended the streams/src/test/java/org/apache/kafka/streams/kstream/internals/SuppressScenarioTest.java with a new scenario where the suppression is based on wall clock time.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant