Skip to content

Conversation

@zhijiangW
Copy link
Contributor

@zhijiangW zhijiangW commented Nov 7, 2018

What is the purpose of the change

Before refactoring ChannelSelector interface mentioned in FLINK-10662, we found there are many duplicated implementations for ChannelSelector in stream and batch modes separately. The root cause is that they are locating in different modules, so can not reference each other.

The core runtime codes should be unified for both stream and batch modes as much as possible, then we can reduce the work overhead to maintain the duplicated codes, especially for the following refactor.

This PR is based on #7031 , then review the second commit for changes.

Next we would remove duplicated implementations for OutputEmitter and RoundRobinChannelSelector.

Brief change log

  • Refactor the existing partitioner package from flink-streaming-jave to flink-runtime.
  • Remove StreamRecord type explicitly from SerializationDelegate.
  • Modify related tests based on refactor changes.

Verifying this change

This change is already covered by existing tests.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (yes)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@zhijiangW
Copy link
Contributor Author

@pnowojski , this is the preparation for unifying implementations in both stream and batch modes. As I described in above purpose, I would submit the following PRs for de-duplication if you think this module refactor make sense.

@zhijiangW
Copy link
Contributor Author

@pnowojski , there are some problems for this issue now. I will update the codes later. Sorry for interrupting. I will cc you after ready.

@pnowojski
Copy link
Contributor

I'm closing this PR @zhijiangW for now. If you will want to re-open it, please feel free to do so.

@pnowojski pnowojski closed this Aug 23, 2019
@zhijiangW zhijiangW deleted the FLINK-10790 branch June 10, 2020 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants