Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance sinks to support idleness. #2

Conversation

EronWright
Copy link

@EronWright EronWright commented Jun 3, 2021

What is the purpose of the change

Propagates the idleness signal to the SinkFunction and SinkWriter for connector-specific purposes such as watermark propagation across pipelines.

Brief change log

  • Add processStreamStatus() method to StreamOperator interface with default implementation
  • Update abstract stream operator to invoke processStreamStatus when the combined input changes.
  • Add markIdle() method to SinkFunction interface
  • Add markIdle() method to SinkWriter interface
  • Add support to sink operators for invoking markIdle() on sink functions.

Verifying this change

(TODO) This change adds tests for idleness callbacks, and verifies exception handling behavior.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDocs

@EronWright EronWright force-pushed the FLINK-22700-sink-watermarks-with-idleness branch from 4810620 to 47fd3af Compare June 10, 2021 19:10
@EronWright EronWright force-pushed the FLINK-22700-sink-watermarks branch from bf8ba2c to 4bea2c8 Compare July 5, 2021 00:09
@syhily syhily closed this Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants