Skip to content

Conversation

@pltbkd
Copy link
Contributor

@pltbkd pltbkd commented Feb 10, 2022

Parallelism of a transformation with default parallelism(-1) is set when transforming, using the default parallelism set in the environment. However, in SinkExpander#expand, the environment parallelism is set to -1 at the entrance, to verify if the parallelism of a expanded transformation is set. The environment parallelism will be restored when exiting the method, but at present the transform is called within this scope. If the parallelism of a sink is not set, the parallelism of the sink transformation and all transformations expanded from it will not be handled, so the JobGraph generated will have vertices with -1 parallelism, causing the assertion failure in AdaptiveScheduler.

This pr fixes the bug by putting the restoring of the environment parallelism ahead of transforming the sink transformations.

@flinkbot
Copy link
Collaborator

flinkbot commented Feb 10, 2022

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@flinkbot
Copy link
Collaborator

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 4a9ad6d (Thu Feb 10 07:33:52 UTC 2022)

Warnings:

  • No documentation files were touched! Remember to keep the Flink docs up to date!

Mention the bot in a comment to re-run the automated checks.

Review Progress

  • ❓ 1. The [description] looks good.
  • ❓ 2. There is [consensus] that the contribution should go into to Flink.
  • ❓ 3. Needs [attention] from.
  • ❓ 4. The change fits into the overall [architecture].
  • ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands
The @flinkbot bot supports the following commands:

  • @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
  • @flinkbot approve all to approve all aspects
  • @flinkbot approve-until architecture to approve everything until architecture
  • @flinkbot attention @username1 [@username2 ..] to require somebody's attention
  • @flinkbot disapprove architecture to remove an approval you gave earlier

Copy link
Member

@dmvk dmvk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @pltbkd 👍 Would it be possible to add a failing unit test verifying the change and preventing future regression?

@pltbkd
Copy link
Contributor Author

pltbkd commented Feb 10, 2022

@dmvk

Sure. I have added a test to verify this. The case will fail if I revert the change of this PR.

But I am not sure if it's at the right place. Since no translator has a test, I add it in the StreamGraphGeneratorTest. Do you think it's ok?

@dmvk
Copy link
Member

dmvk commented Feb 10, 2022

Sounds like a right place to me 👍

Copy link
Member

@dmvk dmvk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pltbkd for adding the test case 👍 This looks really good. Great job 🎉

@zentol zentol self-assigned this Feb 10, 2022
@zentol
Copy link
Contributor

zentol commented Feb 10, 2022

@pltbkd Could you rebase the PR?

Actually, I'll do that. I need to run CI manually anyway before the merge.

@pltbkd
Copy link
Contributor Author

pltbkd commented Feb 11, 2022

Noted.
Thanks @dmvk and @zentol for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants