Refactor StreamOperator Hierachy to make Keys Explicit #2573
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The name is clashing with the soon-to-be-added TimerService/InternalTimerService which is meant as an interface for dealing with both processing time and event time. TimeServiceProvided is renamed to ProcessingTimeService to reflect the fact that it is a low-level utility that only deals with "physical" processing-time trigger tasks.
Before, this would allow handing in a custom ProcessingTimeService but this was in reality always TestProcessingTimeService.
This also introduces KeyedTwoInputStreamOperatorTestHarness which is similar to KeyedOneInputStreamOperatorTestHarness
This moves the event-time/processing-time trigger code from WindowOperator behind a well defined interface that can be used by operators (and user functions). InternalTimerService is the new interface that has the same functionality that WindowOperator used to have. TimerService is the user facing interface that does not allow dealing with namespaces/payloads and also does not allow deleting timers. There is a default implementation in HeapInternalTimerService that can checkpoint timers to a stream and also restore from a stream. Right now, this is managed in AbstractStreamOperator and operators can ask for an InternalTimerService. This also adds tests for HeapInternalTimerService. This adds two new user functions: - TimelyFlatMapFunction: an extension of FlatMapFunction that also allows querying time and setting timers - TimelyCoFlatMapFunction: the same, but for CoFlatMapFunction There are two new StreamOperator implementations for these that use the InternalTimerService interface. This also adds tests for the two new operators. This also adds the new interface KeyContext that is used for setting/querying the current key context for state and timers. Timers are always scoped to a key, for now. Also, this moves the handling of watermarks for both one-input and two-input operators to AbstractStreamOperators so that we have a central ground-truth.
aljoscha
force-pushed
the
refactor-stream-operators
branch
from
c9049c5
to
790ffdb
Compare
aljoscha
force-pushed
the
refactor-stream-operators
branch
from
790ffdb
to
0600d14
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is more of a preview PR, I'm not yet completely done with testing and making sure that everything works.
R: @StephanEwen