[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow #12612

kamilwu · 2020-08-18T14:11:01Z

Add Python GBK Load Tests for streaming on Dataflow. Cases 1, 2, 4 and 5 are temporarily excluded because their execution time is too long.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

kamilwu · 2020-08-18T14:11:18Z

Run Seed Job

kamilwu · 2020-08-18T14:27:01Z

Run Load Tests Python GBK Dataflow Batch

kamilwu · 2020-08-18T14:27:44Z

Run Load Tests Python GBK Dataflow Streaming

kamilwu · 2020-08-18T14:28:33Z

Run Load Tests Python GBK reiterate Dataflow Batch

kamilwu · 2020-08-18T14:28:47Z

Run Load Tests Python GBK reiterate Dataflow Streaming

kamilwu · 2020-08-20T10:13:20Z

Run Seed Job

kamilwu · 2020-08-20T10:20:02Z

Run Load Tests Python GBK Dataflow Batch

kamilwu · 2020-08-20T10:20:16Z

Run Load Tests Python GBK Dataflow Streaming

kamilwu · 2020-08-20T10:20:28Z

Run Load Tests Python GBK reiterate Dataflow Batch

kamilwu · 2020-08-20T10:20:34Z

Run Load Tests Python GBK reiterate Dataflow Streaming

kamilwu · 2020-08-20T10:58:34Z

Run Python PreCommit

kamilwu · 2020-08-20T11:42:24Z

R: @tysonjh Could you take a look?

tysonjh

Looks good! Some minor comments.

tysonjh · 2020-08-21T17:25:26Z

.test-infra/jenkins/job_LoadTests_GBK_Python.groovy

      ]
    ],
    [
      title          : 'GroupByKey Python Load test: 2GB of 100B records',
      test           : 'apache_beam.testing.load_tests.group_by_key_test',
      runner         : CommonTestProperties.Runner.DATAFLOW,
      pipelineOptions: [
-        job_name             : 'load-tests-python-dataflow-batch-gbk-2-' + now,
+        job_name             : 'load-tests-python-dataflow-${mode}-gbk-2-' + now,


Suggested change

job_name : 'load-tests-python-dataflow-${mode}-gbk-2-' + now,

job_name : "load-tests-python-dataflow-${mode}-gbk-2-${now}",

tysonjh · 2020-08-21T17:33:10Z

.test-infra/jenkins/job_LoadTests_GBK_Python.groovy

+def addStreamingOptions(test) {
+  test.pipelineOptions << [streaming: null, experiments: 'use_runner_v2',
+    enable_streaming_engine: null ]
+}


Can you add a comment to explain what these settings are? It's unexpected to see that 'streaming: null' or 'enable_streaming_engine: null' somehow enables streaming, or why 'use_runner_v2' is required.

I removed --enable_streaming_engine, since it is now being added automatically when using use_runner_v2: #12585

tysonjh · 2020-08-21T17:34:28Z

.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy

-def batchLoadTestJob = { scope, triggeringContext ->
-  scope.description('Runs Python GBK reiterate load tests on Dataflow runner in batch mode')
-  commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+def addStreamingOptions(test) {


Same here, please add a comment.

tysonjh · 2020-08-21T17:35:09Z

.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy

-  batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
-}
+CronJobBuilder.cronJob('beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch',
+    'H 14 * * *', this) {


What's the methodology for picking the time to trigger these? Is it documented anywhere?

Ideally, each test suite (GBK, ParDo, IO tests, etc.) should has its own, unique time in order no to flood Jenkins with many tests that are triggered at the same time. When adding a new test suite, a contributor has to take a look at what time slots are already occupied and avoid using them.

I think this is not documented. I'll add some information here: https://cwiki.apache.org/confluence/display/BEAM/Contribution+Testing+Guide#ContributionTestingGuide

tysonjh · 2020-08-21T17:47:18Z

.test-infra/jenkins/job_LoadTests_GBK_Python.groovy

@@ -22,119 +22,138 @@ import InfluxDBCredentialsHelper

 def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))

-def loadTestConfigurations = { datasetName ->
+// TODO(BEAM-10774): Skipping some cases because they are too slow.
+def STREAMING_TESTS_TO_SKIP = [1, 2, 4, 5]


@kkucharc made a good point here in PR#12435 about using indices for ignoring tests. I'm more inclined towards the approach @kkucharc is taking by excluding using the job_name.

Thanks, an argument that @kkucharc made in PR#12435 is convincing. I'll exclude those cases by using the job_name.

codecov · 2020-08-24T09:56:21Z

Codecov Report

Merging #12612 into master will decrease coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12612      +/-   ##
==========================================
- Coverage   34.47%   34.33%   -0.14%     
==========================================
  Files         684      692       +8     
  Lines       81483    81958     +475     
  Branches     9180     9264      +84     
==========================================
+ Hits        28090    28144      +54     
- Misses      52965    53391     +426     
+ Partials      428      423       -5

Impacted Files	Coverage Δ
typehints/typecheck_test_py3.py	`31.54% <0.00%> (-16.00%)`	⬇️
typehints/typecheck.py	`29.44% <0.00%> (-6.18%)`	⬇️
utils/interactive_utils.py	`30.95% <0.00%> (-2.39%)`	⬇️
testing/load_tests/load_test_metrics_utils.py	`34.98% <0.00%> (-1.39%)`	⬇️
runners/worker/opcounters.py	`33.81% <0.00%> (-0.87%)`	⬇️
pipeline.py	`22.04% <0.00%> (-0.28%)`	⬇️
options/pipeline_options.py	`52.99% <0.00%> (-0.16%)`	⬇️
transforms/ptransform_test.py	`18.37% <0.00%> (-0.09%)`	⬇️
transforms/core.py	`36.79% <0.00%> (-0.05%)`	⬇️
runners/common.py	`27.50% <0.00%> (ø)`
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 094d06c...11a1a3b. Read the comment docs.

kamilwu · 2020-08-24T09:58:56Z

Run Seed Job

kamilwu · 2020-08-24T10:07:57Z

Run Load Tests Python GBK Dataflow Streaming

kamilwu · 2020-08-24T10:08:04Z

Run Load Tests Python GBK Dataflow Batch

kamilwu · 2020-08-24T10:08:13Z

Run Load Tests Python GBK reiterate Dataflow Streaming

kamilwu · 2020-08-24T10:08:19Z

Run Load Tests Python GBK reiterate Dataflow Batch

kamilwu · 2020-08-24T11:09:22Z

@tysonjh Thanks! Could you verify my fixes before I merge?

…properly returned from Ungroup transform

enable_streaming_engine is now being added automatically when running with 'use_runner_v2'

tysonjh

LGTM, thanks!

kamilwu · 2020-08-26T08:39:29Z

Thanks!

…che#12612)

probot-autolabeler bot added infra python labels Aug 18, 2020

kamilwu force-pushed the python-gbk-streaming branch from e9c4510 to f67712b Compare August 20, 2020 10:12

kamilwu marked this pull request as ready for review August 20, 2020 10:13

tysonjh approved these changes Aug 21, 2020

View reviewed changes

tysonjh reviewed Aug 21, 2020

View reviewed changes

kamilwu added 4 commits August 24, 2020 15:51

[BEAM-10675] Protect against None-valued metrics

7909cab

[BEAM-10675] Python GBK load test: fix a bug where KV pairs were not …

88dd6b3

…properly returned from Ungroup transform

[BEAM-10675] Python GBK load test: add streaming job

0e6735b

fix: exclude test cases by job_name; add explanatory comments

686122d

kamilwu force-pushed the python-gbk-streaming branch from b1c9673 to e6bbfe2 Compare August 24, 2020 15:34

fix: remove enable_streaming_engine option

11a1a3b

enable_streaming_engine is now being added automatically when running with 'use_runner_v2'

kamilwu force-pushed the python-gbk-streaming branch from e6bbfe2 to 11a1a3b Compare August 24, 2020 15:35

tysonjh approved these changes Aug 25, 2020

View reviewed changes

kamilwu merged commit f675112 into apache:master Aug 26, 2020

kamilwu deleted the python-gbk-streaming branch August 26, 2020 09:46

ibzib pushed a commit to ibzib/beam that referenced this pull request Sep 30, 2020

[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow (apa…

270baf5

…che#12612)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow #12612

[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow #12612

kamilwu commented Aug 18, 2020 •

edited

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

tysonjh left a comment

tysonjh Aug 21, 2020

tysonjh Aug 21, 2020

kamilwu Aug 24, 2020

tysonjh Aug 21, 2020

tysonjh Aug 21, 2020

kamilwu Aug 24, 2020 •

edited

tysonjh Aug 21, 2020

kamilwu Aug 24, 2020

codecov bot commented Aug 24, 2020 •

edited

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

tysonjh left a comment

kamilwu commented Aug 26, 2020

	job_name : 'load-tests-python-dataflow-${mode}-gbk-2-' + now,
	job_name : "load-tests-python-dataflow-${mode}-gbk-2-${now}",

[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow #12612

[BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow #12612

Conversation

kamilwu commented Aug 18, 2020 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 18, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

kamilwu commented Aug 20, 2020

tysonjh left a comment

Choose a reason for hiding this comment

tysonjh Aug 21, 2020

Choose a reason for hiding this comment

tysonjh Aug 21, 2020

Choose a reason for hiding this comment

kamilwu Aug 24, 2020

Choose a reason for hiding this comment

tysonjh Aug 21, 2020

Choose a reason for hiding this comment

tysonjh Aug 21, 2020

Choose a reason for hiding this comment

kamilwu Aug 24, 2020 • edited

Choose a reason for hiding this comment

tysonjh Aug 21, 2020

Choose a reason for hiding this comment

kamilwu Aug 24, 2020

Choose a reason for hiding this comment

codecov bot commented Aug 24, 2020 • edited

Codecov Report

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

kamilwu commented Aug 24, 2020

tysonjh left a comment

Choose a reason for hiding this comment

kamilwu commented Aug 26, 2020

kamilwu commented Aug 18, 2020 •

edited

kamilwu Aug 24, 2020 •

edited

codecov bot commented Aug 24, 2020 •

edited