-
Notifications
You must be signed in to change notification settings - Fork 13.8k
[FLINK-15308][runtime] Remove data compression for pipelined shuffle. #10627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit a8a9d94 (Thu Dec 19 10:25:28 UTC 2019) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. DetailsThe Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
c09e6a1 to
a2c34a0
Compare
CI report:
Bot commandsThe @flinkbot bot supports the following commands:
|
| * | ||
| * @return a retained copy of self with separate indexes | ||
| */ | ||
| public BufferConsumer copy() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should also adjust the method description above.
zhijiangW
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix @wsry!
Almost LGTM except for one tiny comment. In addition, can you add some messages in commit for the reason of removing pipelined compression?
a2c34a0 to
919abd2
Compare
|
Thanks for the review @zhijiangW . I have updated the PR. |
Current shuffle data compression implementation for pipelined mode introduce several special case handling including an additional synchronization which can complicate the network stack. So we decide to remove data compression for pipelined shuffle in version release-1.10 and the feature may be added back if there is a better solution in the future.
919abd2 to
9322dfa
Compare
What is the purpose of the change
This PR removes data compression for pipelined shuffle mode. The feature may be added in the future version of Flink if there is a better solution.
Brief change log
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation