Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-28380][runtime] Produce one intermediate dataset for multiple consumer job vertices consuming the same data #20351

Merged
merged 1 commit into from Aug 9, 2022
Merged

[FLINK-28380][runtime] Produce one intermediate dataset for multiple consumer job vertices consuming the same data #20351

merged 1 commit into from Aug 9, 2022

Conversation

wsry
Copy link
Contributor

@wsry wsry commented Jul 25, 2022

What is the purpose of the change

Currently, if one output of an upstream job vertex is consumed by multiple downstream job vertices, the upstream vertex will produce multiple dataset. For blocking shuffle, it means serialize and persist the same data multiple times. This ticket aims to optimize this behavior and make the upstream job vertex produce one dataset which will be read by multiple downstream vertex.

Brief change log

  • Produce one intermediate dataset for multiple consumer job vertices consuming the same data.

Verifying this change

This change added tests.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@wsry wsry requested a review from gaoyunhaii July 25, 2022 08:28
@flinkbot
Copy link
Collaborator

flinkbot commented Jul 25, 2022 •
edited

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

wsry added a commit to wsry/flink that referenced this pull request Jul 27, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
2779a5c 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Jul 27, 2022
@wsry
[hotfix][tests] Migrate tests relevant to FLINK-28380 to Junit5/AssertJ
578b2bd 
Migrated tests include StreamingJobGraphGeneratorTest, ForwardForConsecutiveHashPartitionerTest and ForwardForUnspecifiedPartitionerTest.

This closes apache#20351.
gaoyunhaii
gaoyunhaii reviewed Jul 28, 2022
View reviewed changes
Copy link
Contributor

@gaoyunhaii gaoyunhaii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wsry for the PR! I left some comments.

flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java Outdated Show resolved Hide resolved
...ming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java Outdated Show resolved Hide resolved
...ming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java Outdated Show resolved Hide resolved
...ming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java Outdated Show resolved Hide resolved
wsry added a commit to wsry/flink that referenced this pull request Jul 28, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
6e2d68e 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Jul 28, 2022
@wsry
[hotfix][tests] Migrate tests relevant to FLINK-28380 to Junit5/AssertJ
80dea31 
Migrated tests include StreamingJobGraphGeneratorTest, ForwardForConsecutiveHashPartitionerTest and ForwardForUnspecifiedPartitionerTest.

This closes apache#20351.
@wsry
Copy link
Contributor Author

wsry commented Jul 28, 2022

@gaoyunhaii Thanks for the review and feed back. I added a fixup commit. Please take another look.

reswqa
reswqa reviewed Jul 28, 2022
View reviewed changes
Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wsry Thanks for your nice work, I only left some minor comments about test.

...rc/test/java/org/apache/flink/runtime/executiongraph/BlockingResultPartitionReleaseTest.java Outdated Show resolved Hide resolved
...test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphConstructionTest.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertexTest.java Outdated Show resolved Hide resolved
...e/src/test/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopologyTest.java Outdated Show resolved Hide resolved
...e/src/test/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopologyTest.java Outdated Show resolved Hide resolved
...e/src/test/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopologyTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adapter/DefaultResultPartitionTest.java Outdated Show resolved Hide resolved
wsry added a commit to wsry/flink that referenced this pull request Jul 29, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
bc52dc7 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Jul 29, 2022
@wsry
[hotfix][tests] Migrate tests relevant to FLINK-28380 to Junit5/AssertJ
2b618e0 
Migrated tests include StreamingJobGraphGeneratorTest, ForwardForConsecutiveHashPartitionerTest and ForwardForUnspecifiedPartitionerTest.

This closes apache#20351.
@wsry
Copy link
Contributor Author

wsry commented Jul 29, 2022

@gaoyunhaii I have updated the PR.

wsry added a commit to wsry/flink that referenced this pull request Jul 29, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
d353fda 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Jul 29, 2022
@wsry
[hotfix][tests] Migrate tests relevant to FLINK-28380 to Junit5/AssertJ
92ac5ba 
Migrated tests include StreamingJobGraphGeneratorTest, ForwardForConsecutiveHashPartitionerTest and ForwardForUnspecifiedPartitionerTest.

This closes apache#20351.
@gaoyunhaii
Copy link
Contributor

Thanks @wsry for the PR! LGTM, let's have a final check after the commit merged.

wsry added a commit to wsry/flink that referenced this pull request Aug 5, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
4c4b592 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Aug 5, 2022
@wsry
[hotfix][tests] Migrate tests relevant to FLINK-28380 to Junit5/AssertJ
03cc1ab 
Migrated tests include StreamingJobGraphGeneratorTest, ForwardForConsecutiveHashPartitionerTest and ForwardForUnspecifiedPartitionerTest.

This closes apache#20351.
wsry added a commit to wsry/flink that referenced this pull request Aug 8, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
8f03049 
…consumer job vertices consuming the same data

This closes apache#20351.
wsry added a commit that referenced this pull request Aug 8, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
e48a303 
…consumer job vertices consuming the same data

This closes #20351.
wsry added a commit to wsry/flink that referenced this pull request Aug 8, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
9bb16e9 
…consumer job vertices consuming the same data

This closes apache#20351.
gaoyunhaii
gaoyunhaii approved these changes Aug 8, 2022
View reviewed changes
Copy link
Contributor

@gaoyunhaii gaoyunhaii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wsry for the PR! LGTM

wsry added a commit to wsry/flink that referenced this pull request Aug 8, 2022
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
8a63197 
…consumer job vertices consuming the same data

This closes apache#20351.
@wsry
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
84f1ac0 
…consumer job vertices consuming the same data

This closes apache#20351.
@wsry
Copy link
Contributor Author

wsry commented Aug 9, 2022

Tests passed: https://dev.azure.com/kevin-flink/flink/_build/results?buildId=624&view=results, merging.

@wsry wsry merged commit 60bc87c into apache:master Aug 9, 2022
@lincoln-lil lincoln-lil mentioned this pull request Aug 22, 2022
Closed
huangxiaofeng10047 pushed a commit to huangxiaofeng10047/flink that referenced this pull request Nov 3, 2022
@wsry @huangxiaofeng10047
[FLINK-28380][runtime] Produce one intermediate dataset for multiple …
ba57976 
…consumer job vertices consuming the same data

This closes apache#20351.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reswqa reswqa reswqa left review comments

@gaoyunhaii gaoyunhaii gaoyunhaii approved these changes

Assignees
No one assigned
Labels
component=Client/JobSubmission component=Runtime/Coordination component=Runtime/Network
Projects
None yet
Milestone
No milestone
4 participants
@wsry @flinkbot @gaoyunhaii @reswqa