[BEAM-8412] xlang test: set sdk worker parallelism to 1#9817
[BEAM-8412] xlang test: set sdk worker parallelism to 1#9817ibzib merged 1 commit intoapache:masterfrom
Conversation
|
Run Python 2 PostCommit |
|
Run CommunityMetrics PreCommit |
|
Run Python 2 PostCommit |
|
Py precommit flake unrelated (https://issues.apache.org/jira/browse/BEAM-8416) |
There was a problem hiding this comment.
The default for this option is 1. Is it set somewhere else to 0?
There was a problem hiding this comment.
I just noticed that this option was repeated in the Python SDK with a different default. We should fix that: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L848
There was a problem hiding this comment.
+1 for fixing Python SDK's default value instead of here. BTW does this mean that Flink's cross-language implementation is somehow broken when there is more than one worker process per worker node ?
cc: @mxm
There was a problem hiding this comment.
@chamikaramj WriteToText relies on shared filesystem, so more than one worker can cause it to fail with the local FS.
If it takes longer to sort out the pipeline option default, then it's probably also OK to merge this first and unblock the CI (assuming no other pre-commit issues).
There was a problem hiding this comment.
Let's add a comment here why this is necessary. Otherwise LGTM.
There was a problem hiding this comment.
Let's add a comment here why this is necessary. Otherwise LGTM.
4 participants
I think cross language tests were relying on the fact that the default for this option was formerly 1, before changed to number of cores by #9785. (Multiple Docker SDK workers would have their own virtual filesystems.) We'll see if this fixes the test.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.