Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-8412] xlang test: set sdk worker parallelism to 1 #9817

Merged
merged 1 commit into from Oct 18, 2019

Conversation

@ibzib
Copy link
Contributor

ibzib commented Oct 17, 2019

I think cross language tests were relying on the fact that the default for this option was formerly 1, before changed to number of cores by #9785. (Multiple Docker SDK workers would have their own virtual filesystems.) We'll see if this fixes the test.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@ibzib

This comment has been minimized.

Copy link
Contributor Author

ibzib commented Oct 17, 2019

Run Python 2 PostCommit

@ibzib

This comment has been minimized.

Copy link
Contributor Author

ibzib commented Oct 17, 2019

Run CommunityMetrics PreCommit

@ibzib ibzib force-pushed the ibzib:xlang-test branch from 9a7dcfd to 766c1d2 Oct 17, 2019
@ibzib

This comment has been minimized.

Copy link
Contributor Author

ibzib commented Oct 17, 2019

Run Python 2 PostCommit

@ibzib

This comment has been minimized.

Copy link
Contributor Author

ibzib commented Oct 17, 2019

Py precommit flake unrelated (https://issues.apache.org/jira/browse/BEAM-8416)

@ibzib ibzib requested a review from chamikaramj Oct 17, 2019
@@ -128,6 +128,7 @@ task crossLanguagePortableWordCount {
"--shutdown_sources_on_final_watermark",
"--environment_cache_millis=10000",
"--expansion_service_jar=${testServiceExpansionJar}",
"--sdk_worker_parallelism=1"

This comment has been minimized.

Copy link
@tweise

tweise Oct 17, 2019

Contributor

The default for this option is 1. Is it set somewhere else to 0?

This comment has been minimized.

Copy link
@tweise

tweise Oct 17, 2019

Contributor

I just noticed that this option was repeated in the Python SDK with a different default. We should fix that: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L848

This comment has been minimized.

Copy link
@chamikaramj

chamikaramj Oct 17, 2019

Contributor

+1 for fixing Python SDK's default value instead of here. BTW does this mean that Flink's cross-language implementation is somehow broken when there is more than one worker process per worker node ?

cc: @mxm

This comment has been minimized.

Copy link
@tweise

tweise Oct 18, 2019

Contributor

@chamikaramj WriteToText relies on shared filesystem, so more than one worker can cause it to fail with the local FS.

If it takes longer to sort out the pipeline option default, then it's probably also OK to merge this first and unblock the CI (assuming no other pre-commit issues).

This comment has been minimized.

Copy link
@mxm

mxm Oct 18, 2019

Contributor

Let's add a comment here why this is necessary. Otherwise LGTM.

This comment has been minimized.

Copy link
@ibzib

ibzib Oct 18, 2019

Author Contributor
@mxm
mxm approved these changes Oct 18, 2019
@@ -128,6 +128,7 @@ task crossLanguagePortableWordCount {
"--shutdown_sources_on_final_watermark",
"--environment_cache_millis=10000",
"--expansion_service_jar=${testServiceExpansionJar}",
"--sdk_worker_parallelism=1"

This comment has been minimized.

Copy link
@mxm

mxm Oct 18, 2019

Contributor

Let's add a comment here why this is necessary. Otherwise LGTM.

@ibzib ibzib force-pushed the ibzib:xlang-test branch from 766c1d2 to 3ff698e Oct 18, 2019
@ibzib ibzib merged commit 16b4afa into apache:master Oct 18, 2019
3 of 5 checks passed
3 of 5 checks passed
Portable_Python ("Run Portable_Python PreCommit") Build started for merge commit.
Details
Python ("Run Python PreCommit") Build started for merge commit.
Details
Python2_PVR_Flink ("Run Python2_PVR_Flink PreCommit") SUCCESS
Details
PythonLint ("Run PythonLint PreCommit") SUCCESS
Details
RAT ("Run RAT PreCommit") SUCCESS
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.