Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/flink 18661 efo de registration #1

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

xiaolong-sn
Copy link

What is the purpose of the change

This is the milestone of FLIP-128 to add EFO support to the FlinkKinesisConsumer.
The de-/regisiterStream operations has been introduce.

JIRA issue

Note

This PR is blocked by:

Brief change log

  • ConsumerConstants
    • Add the describe stream and descrbie stream consumer operation parameters.
  • KinesisConfigUtil
    • Add the validation of the above parameter
  • FanOutStreamInfo
    • Add a new data class for storing enhanced fan-out consumer info.
  • FlinkDataFetcher
    • Add a KinesisProxyV2Interface in order to register stream consumers when the efo registration mode is set to eager.
  • ShardConsumer
    • Add a KinesisProxyV2Interface in order to de-/register stream consumers when the efo registration mode is set to lazy.
  • KinesisProxyV2Interface & KinesisProxyV2
    • Add the de-/register stream consumer operations here.
  • AwsV2Util
    • Add several methods to classify exceptions.

Verifying this change

This change added tests and can be verified as follows:

  • KinesisProxyV2Test
  • AwsV2UtilTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

dannycranmer and others added 18 commits July 8, 2020 09:07
…ctoring ShardConsumer to use PollingRecordPublisher
…' into feature/FLINK-18483-efo-1-test-coverage
… consumers to fit in the requirements of FLIP-128.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants