Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-8157] Increase parallelism for Python PVR tests #9649

Merged
merged 1 commit into from Sep 27, 2019

Conversation

@mxm
Copy link
Contributor

mxm commented Sep 24, 2019

The Python PVR tests should be executing with a parallelism greater than one to
ensure that key and state partitioning work correctly. A recent change to the
key encoding for state requests uncovered that the non-parallel execution could
shadow a key encoding issue.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status --- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@mxm mxm force-pushed the mxm:pvr-parallelism branch 2 times, most recently from 16086e1 to 42cae0a Sep 24, 2019
The Python PVR tests should be executing with a parallelism greater than one to
ensure that key and state partitioning work correctly. A recent change to the
key encoding for state requests uncovered that the non-parallel execution could
shadow a key encoding issue.
@mxm mxm requested a review from tweise Sep 24, 2019
@tweise

This comment has been minimized.

Copy link
Contributor

tweise commented Sep 27, 2019

AFAIR the original reason to limit parallelism to 1 was resource consumption in Jenkins. So I wonder if we should only enable it for selected tests where the parallelism matters rather than doing it for all?

@mxm

This comment has been minimized.

Copy link
Contributor Author

mxm commented Sep 27, 2019

Yes, that is correct. The old parallelism was much higher than 2 though, typically 16-32 due to local execution defaulting to the number of available for cores by default. I think a parallelism of 2 won't hurt.

@tweise
tweise approved these changes Sep 27, 2019
@tweise tweise merged commit 469618c into apache:master Sep 27, 2019
4 checks passed
4 checks passed
Portable_Python ("Run Portable_Python PreCommit") SUCCESS
Details
Python ("Run Python PreCommit") SUCCESS
Details
Python_PVR_Flink ("Run Python_PVR_Flink PreCommit") SUCCESS
Details
RAT ("Run RAT PreCommit") SUCCESS
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.