New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-19385] Request partitions for each InputGate independently #13467
[FLINK-19385] Request partitions for each InputGate independently #13467
Conversation
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 9c4cc72 (Wed Sep 23 18:36:51 UTC 2020) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I wonder why we didn't do it before.
I was wondering if we can add a test for it? The ticket description looks like we could easily transfer it to this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also +1 for the changes.
In theory every input gate can recovery state and then request partitions independently without precision concern. The formal way somehow considered all the buffer resources used in recovery for loading states firstly before loading new data from upstreams. Then we can guarantee the recovery process can end ASAP to avoid over loading them again once failure happens while mix processing new data with state data. But indeed it is not a strong requirement. Just for clarifying the concerns from @AHeise .
I agree to add somehow UT if possible. Also it will be better to give some descriptions in commit message to understand the background why we need this change. It is for resolving the potential problem of channel selection priority in multiple gates, although we have not supported it in unaligned checkpoint yet.
Thanks for reviewing! |
Subsumed by #13351 |
What is the purpose of the change
Please see FLINK-19385 for motivation.
Verifying this change
The issue is covered by
StreamTaskSelectiveReadingITCase
: #13351 fails without this change.Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: noDocumentation