New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-22053][core] Allow NumberSequenceSource to have less splits than parallelism. #15449
Conversation
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 90e7028 (Wed Mar 31 08:52:28 UTC 2021) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me with few minor comments.
...-core/src/main/java/org/apache/flink/api/connector/source/lib/util/IteratorSourceReader.java
Outdated
Show resolved
Hide resolved
@@ -52,6 +64,8 @@ | |||
.setNumberSlotsPerTaskManager(PARALLELISM) | |||
.build()); | |||
|
|||
@Rule public StreamCollector collector = new StreamCollector(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use DataStream.executeAndCollect()
instead?
That is the main code users also use, let's give it as much exposure in tests as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have also changed it in the original test.
env.fromSequence(0, 100) | ||
.map( | ||
x -> { | ||
Thread.sleep(10); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
10ms is pretty long here, would 2ms also do it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have slighty increased it instead, but only apply it to the first element now.
LGTM. +1 to merge this. |
Merging this... (I need to build on top of this). |
…an parallelism. This closes apache#15449
What is the purpose of the change
NumberSequenceSource causes fatal exception when less splits than parallelism.
Brief change log
Verifying this change
Added IteratorSourcesITCase#testLessSplitsThanParallelism.
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation