Skip to content

Conversation

@wanglijie95
Copy link
Contributor

What is the purpose of the change

Currently, the adaptive batch scheduler only respects the global maximum parallelism(which is configured by option parallelism.default or execution.batch.adaptive.auto-parallelism.max-parallelism, see FLINK-30686 for details) when deciding parallelism for job vertices, the maximum parallelism of vertices configured by the user through setMaxParallelism will not be respected.

In this ticket, we will change the behavior so that the user-configured max parallelism also be respected.

Verifying this change

Add tests in DefaultVertexParallelismAndInputInfosDeciderTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@wanglijie95 wanglijie95 requested a review from zhuzhurk February 6, 2023 08:49
@flinkbot
Copy link
Collaborator

flinkbot commented Feb 6, 2023

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Copy link
Contributor

@zhuzhurk zhuzhurk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

wanglijie95 added a commit that referenced this pull request Feb 14, 2023
…user-configured max parallelism when deciding parallelism

This closes #21861
mohsenrezaeithe pushed a commit to mohsenrezaeithe/flink that referenced this pull request Feb 21, 2023
…user-configured max parallelism when deciding parallelism

This closes apache#21861
@wanglijie95 wanglijie95 deleted the FLINK-30917 branch February 23, 2023 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants