Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-3089] Fix job parallelism resolution #4766

Closed
wants to merge 2 commits into from

Conversation

grzegorz8
Copy link

@grzegorz8 grzegorz8 commented Feb 28, 2018

DefaultParallelismFactory has been removed since it ignores parallelism value
which is specified at job submission. Instead, FlinkPipelineOptions#parallelism
defaults to -1, which means the execution environment's parallelism should
be used.

Now the precedence is as follows:

  1. the value specified directly in the source code
  2. the value passed as a Flink Runner parameter at job submission
  3. the value passed as a Flink CLI / Flink UI parameter at job submission
  4. the default value for Flink cluster

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand:
    • What the pull request does
    • Why it does it
    • How it does it
    • Why this approach
  • Each commit in the pull request should have a meaningful subject line and body.
  • Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

@grzegorz8
Copy link
Author

Some tests implicitly assume that parallelism=1, so I had to explicitly specify it. I can try to refactor them to work correctly regardless of the parallelism, if needed.

@aljoscha
Copy link
Contributor

aljoscha commented Mar 6, 2018

retest this please

@aljoscha
Copy link
Contributor

aljoscha commented Mar 6, 2018

Run Flink ValidatesRunner

@grzegorz8
Copy link
Author

retest this please

1 similar comment
@grzegorz8
Copy link
Author

retest this please

Grzegorz Kołakowski added 2 commits March 27, 2018 08:52
DefaultParallelismFactory has been removed since it ignores parallelism value
which is specified at job submission. Instead, FlinkPipelineOptions#parallelism
defaults to -1, which means the execution environment's parallelism should
be used.

Now the precedence is as follows:
 1. the value specified directly in the source code
 2. the value passed as a Flink Runner parameter at job submission
 3. the value passed as a Flink CLI / Flink UI parameter at job submission
 4. the default value for Flink cluster
@stale
Copy link

stale bot commented Jun 7, 2018

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

@stale stale bot added the wontfix label Jun 7, 2018
@stale
Copy link

stale bot commented Jun 14, 2018

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@stale stale bot closed this Jun 14, 2018
@kennknowles kennknowles added stale and removed wontfix labels Jun 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants