Skip to content

Conversation

1996fanrui
Copy link
Member

@1996fanrui 1996fanrui commented Nov 25, 2022

What is the purpose of the change

When turn up the parallelism, the resources isn't enough for MiniCluster.

Brief change log

MiniCluster adapt the jobvertex-parallelism-overrides

Verifying this change

This change added tests and can be verified as follows:

  • PerJobMiniClusterFactoryTest#testTurnUpParallelismByOverwriteParallelism

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not documented

@flinkbot
Copy link
Collaborator

flinkbot commented Nov 25, 2022

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@1996fanrui 1996fanrui force-pushed the 30214/turn-up-parallelism branch 2 times, most recently from cac915b to e1dc2e4 Compare November 25, 2022 15:21
@1996fanrui 1996fanrui force-pushed the 30214/turn-up-parallelism branch from e1dc2e4 to 8af7df2 Compare November 25, 2022 15:31
@1996fanrui
Copy link
Member Author

Hi @gyfora @mxm

As we know, when the flink job run on IDEA, it will use the MiniCluster. And PerJobMiniClusterFactory#getMiniClusterConfig generate the numSlotsPerTaskManager by the JobGraph. However, the numSlotsPerTaskManager may be wrong after using jobvertex-parallelism-overrides. So the numSlotsPerTaskManager should be generated according to the JobGraph and jobvertex-parallelism-overrides.

Please help take a look this PR in your free time, thanks~

Copy link
Contributor

@gyfora gyfora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Copy link
Contributor

@mxm mxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gyfora gyfora merged commit ee4d5b8 into apache:master Nov 29, 2022
@1996fanrui 1996fanrui deleted the 30214/turn-up-parallelism branch November 29, 2022 16:50
@1996fanrui
Copy link
Member Author

@gyfora @mxm , thanks a lot for your review~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants