[BEAM-6908] Refactor Python performance test groovy file for easy configuration #8518

markflyhigh · 2019-05-07T19:10:14Z

Combine Python performance test groovy files into one with structured configuration block. Adding a new python performance test job can be simply as adding a PerformanceTestConfigurations into testConfigurations list.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go	---	---	---	---	---	---
Java
Python	---			---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

markflyhigh · 2019-05-07T20:14:33Z

Run Seed Job

markflyhigh · 2019-05-07T20:37:18Z

Run Python Performance Test

markflyhigh · 2019-05-07T20:37:30Z

Run Python35 Performance Test

markflyhigh · 2019-05-07T20:46:48Z

+R: @tvalentyn @yifanzou

yifanzou

LGTM

.test-infra/jenkins/job_PerformanceTests_Python.groovy

markflyhigh · 2019-05-08T00:05:14Z

Run Seed Job

markflyhigh · 2019-05-08T00:23:17Z

Run Python35 Performance Test

markflyhigh · 2019-05-08T00:46:00Z

PTAL @yifanzou

.test-infra/jenkins/job_PerformanceTests_Python.groovy

tvalentyn · 2019-05-08T02:48:09Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+  String sdk = 'python'
+  String bigqueryTable
+  String itClass
+  String itModule


What is the meaning of this? I am confused by how this is actually used.

It's a benchmark flag defined in here. Basically it's the path of Gradle module.

How do we know how to set this correctly? It seems not intuitive...

we can rename it something like itGradleModule if helps.

Ok, but how do we know which gradle module to select? I see that you used a different value for Py2 and Py3 benchmark, how did you pick those specific ones? How does a person writing an new benchmark decides how to fill this value?

I think people should know how Perfkit beam_integration_benchmark works before configuring in Jenkins. Probably we need better document for that, and also happy to sync with you offline for more details.

For you question, beam_integration_benchmark uses Gradle task integrationTest which can be enabled through enablePythonPerformanceTest. So beam_it_module is the Gradle project where integrationTest located.

Per offline discussion, let's add a comment here:
Gradle project that defines 'runIntegrationTest' task. This task is executed by Perfkit Beam benchmark launcher.
This task can be added by enablePythonPerformanceTest() defined in BeamModulePlugin.

.test-infra/jenkins/job_PerformanceTests_Python.groovy

tvalentyn · 2019-05-08T02:51:25Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+        bigqueryTable     : 'beam_performance.wordcount_py35_pkb_results',
+        skipPrebuild      : true,
+        pythonSdkLocation : 'test-suites/dataflow/py35/build/apache-beam.tar.gz',
+        itClass           : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',


Is my understanding right that each performance test configuration can run only one IT? We might want to always include 'wordcount' in configuration flags or always omit if the suite will later include other benchmarks.

No, it can run multiple ITs if you want. itClass will be passed to classname of this function and eventually to -Dtests of the Gradle invocation.

Ok, but we will not get one test reading per each test, instead we will run both tests, and get a total runtime, right?

yes. This flag defines what tests run in a Gradle execution, and the benchmark will evaluate whole Gradle execution.

So in this case it's one test per configuration, so we may want to call it "WordCount" benchmark, instead of generic 'Performance test'.
Also, where do we specify the input for the WC pipeline?

tvalentyn · 2019-05-08T02:58:04Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+        jobTriggerPhrase  : 'Run Python35 Performance Test',
+        bigqueryTable     : 'beam_performance.wordcount_py35_pkb_results',
+        skipPrebuild      : true,
+        pythonSdkLocation : 'test-suites/dataflow/py35/build/apache-beam.tar.gz',


I wonder if we can always use the same location here. As I mentioned before, there is no difference which interpreter version to use to create tarball.
If we must specify location per suite, perhaps we could have one parameter, such as testRoot and evaluate sdkLocation and itModule value from that parameter. As mentioned above I also am not sure what itModule is for.

Currently, tar file is generated in build directory of the Gradle module where IT is located. We need to specify location per test. We can populate sdkLocation from itModule directly and in the future we could refactor Gradle build to generate tar file only once. markflyhigh#6 is a draft.

.test-infra/jenkins/job_PerformanceTests_Python.groovy

tvalentyn · 2019-05-08T03:02:32Z

Thanks, @markflyhigh !

markflyhigh · 2019-05-08T20:30:13Z

Updated comments to parameters in PerformanceTestConfigurations and addressed few questions.
PTAL @tvalentyn

markflyhigh · 2019-05-09T18:17:40Z

Run Seed Job

markflyhigh · 2019-05-09T18:38:03Z

Run Python35 Performance Test

markflyhigh · 2019-05-09T22:41:50Z

Run Seed Job

markflyhigh · 2019-05-09T22:58:20Z

Run Python35 Performance Test

tvalentyn · 2019-05-10T17:50:08Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+  String sdk = 'python'
+  String bigqueryTable
+  String itClass
+  String itModule


How do we know how to set this correctly? It seems not intuitive...

.test-infra/jenkins/job_PerformanceTests_Python.groovy

tvalentyn · 2019-05-10T18:35:59Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+        bigqueryTable     : 'beam_performance.wordcount_py35_pkb_results',
+        skipPrebuild      : true,
+        pythonSdkLocation : 'test-suites/dataflow/py35/build/apache-beam.tar.gz',
+        itClass           : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',


Ok, but we will not get one test reading per each test, instead we will run both tests, and get a total runtime, right?

markflyhigh · 2019-05-10T21:49:28Z

PTAL @tvalentyn

tvalentyn

Thanks, a few more comments.

tvalentyn · 2019-05-10T21:51:32Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

        benchmarks              : testConfig.benchmarkName,
        bigquery_table          : testConfig.resultTable,
        beam_it_class           : testConfig.itClass,
        beam_it_module          : testConfig.itModule,
-        beam_prebuilt           : testConfig.prebuilt.toString(),
+        beam_prebuilt           : 'true',


Could we add a comment above why this value needs to be true?

The comment says // always true for Python tests but does not explain why this needs to be true, so this configuration bit remains a little cryptic... Could we add a simple explanation?

tvalentyn · 2019-05-10T21:57:10Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+        bigqueryTable     : 'beam_performance.wordcount_py35_pkb_results',
+        skipPrebuild      : true,
+        pythonSdkLocation : 'test-suites/dataflow/py35/build/apache-beam.tar.gz',
+        itClass           : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',


So in this case it's one test per configuration, so we may want to call it "WordCount" benchmark, instead of generic 'Performance test'.
Also, where do we specify the input for the WC pipeline?

tvalentyn · 2019-05-10T21:58:25Z

.test-infra/jenkins/job_PerformanceTests_Python.groovy

+  String sdk = 'python'
+  String bigqueryTable
+  String itClass
+  String itModule


Ok, but how do we know which gradle module to select? I see that you used a different value for Py2 and Py3 benchmark, how did you pick those specific ones? How does a person writing an new benchmark decides how to fill this value?

.test-infra/jenkins/job_PerformanceTests_Python.groovy

markflyhigh · 2019-05-15T17:41:48Z

PTAL @tvalentyn

tvalentyn

Thanks, @markflyhigh, overall LGTM.

Left one comment inline, also, is there a link on somewhere on Beam website that points to results produced by these benchmarks?

If not, could we add it please?

tvalentyn · 2019-05-15T21:51:10Z

CC: @manisha252 who is working on performance test infrastructure and may have additional input here.

markflyhigh · 2019-05-15T23:09:59Z

Thank you @tvalentyn.

To your question, results are stored in Bigquery table which should be accessible from UI or gcloud commandline. Different job is likely to use own table and configured in testConfigurations so it's not possible to provide a static link in groovy file. I'll write up a document for Python performance test and I can include this info.

tvalentyn · 2019-05-15T23:46:35Z

I meant that we should have some pointers in Beam documentation that can point to existence of these benchmarks and dashboards they produce.

markflyhigh · 2019-05-16T22:36:19Z

Run Seed Job

markflyhigh · 2019-05-17T00:05:51Z

Run Python35 WordCountIT Performance Test

markflyhigh · 2019-05-17T00:30:35Z

@tvalentyn I can add link of Bigquery table and dashboard in Beam doc. Also fixed the comment for beam_prebuilt.

Synced with @manisha252 offline and got approved for this change.

…figuration (apache#8518) Refactor Python performance test groovy file for easy configuration.

Refactor Python performance test groovy file for easy configuration.

bfb0bbe

markflyhigh changed the title ~~Refactor Python performance test groovy file for easy configuration~~ [BEAM-6908] Refactor Python performance test groovy file for easy configuration May 7, 2019

yifanzou approved these changes May 7, 2019

View reviewed changes

.test-infra/jenkins/job_PerformanceTests_Python.groovy Outdated Show resolved Hide resolved

fixup! Group dataflowPipelineArgs for reuse

8cde119

tvalentyn reviewed May 8, 2019

View reviewed changes

markflyhigh force-pushed the py-perf-combine-groovy branch 2 times, most recently from 1a38f42 to 798ca0a Compare May 8, 2019 20:29

fixit! Address comments and renaming parameters

116ba25

markflyhigh force-pushed the py-perf-combine-groovy branch from 798ca0a to 116ba25 Compare May 9, 2019 22:41

tvalentyn reviewed May 10, 2019

View reviewed changes

markflyhigh force-pushed the py-perf-combine-groovy branch from 44b4f44 to cd4793a Compare May 15, 2019 17:40

markflyhigh force-pushed the py-perf-combine-groovy branch from cd4793a to bd49953 Compare May 15, 2019 18:31

tvalentyn approved these changes May 15, 2019

View reviewed changes

fixup! address comments

f16c2fc

markflyhigh force-pushed the py-perf-combine-groovy branch from bd49953 to f16c2fc Compare May 17, 2019 00:26

markflyhigh merged commit d77474d into apache:master May 17, 2019

markflyhigh mentioned this pull request May 17, 2019

Update .test-infra/jenkins/README #8609

Merged

3 tasks

ajamato pushed a commit to ajamato/beam that referenced this pull request May 18, 2019

[BEAM-6908] Refactor Python performance test groovy file for easy con…

4e7b208

…figuration (apache#8518) Refactor Python performance test groovy file for easy configuration.

[BEAM-6908] Refactor Python performance test groovy file for easy configuration #8518

[BEAM-6908] Refactor Python performance test groovy file for easy configuration #8518

Conversation

markflyhigh commented May 7, 2019 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

markflyhigh commented May 7, 2019

markflyhigh commented May 7, 2019

markflyhigh commented May 7, 2019

markflyhigh commented May 7, 2019 • edited

yifanzou left a comment

Choose a reason for hiding this comment

markflyhigh commented May 8, 2019

markflyhigh commented May 8, 2019

markflyhigh commented May 8, 2019

Choose a reason for hiding this comment

markflyhigh May 8, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markflyhigh May 10, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn commented May 8, 2019

markflyhigh commented May 8, 2019 • edited

markflyhigh commented May 9, 2019

markflyhigh commented May 9, 2019

markflyhigh commented May 9, 2019

markflyhigh commented May 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markflyhigh commented May 10, 2019

tvalentyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markflyhigh commented May 15, 2019

tvalentyn left a comment • edited

Choose a reason for hiding this comment

tvalentyn commented May 15, 2019 • edited

markflyhigh commented May 15, 2019 • edited

tvalentyn commented May 15, 2019

markflyhigh commented May 16, 2019

markflyhigh commented May 17, 2019

markflyhigh commented May 17, 2019

markflyhigh commented May 7, 2019 •

edited

markflyhigh commented May 7, 2019 •

edited

markflyhigh May 8, 2019 •

edited

markflyhigh May 10, 2019 •

edited

markflyhigh commented May 8, 2019 •

edited

tvalentyn left a comment •

edited

tvalentyn commented May 15, 2019 •

edited

markflyhigh commented May 15, 2019 •

edited