[BEAM-5985] Create jenkins jobs to run the load tests for Java SDK #7184

kkucharc · 2018-12-03T16:27:52Z

Created builder for preparing Jenkins load tests with DSL for group by key Java load test. It will be extended with other tests, suites and Python SDK in next PRs.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ x ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go	---	---	---	---	---	---
Java
Python	---			---	---	---

kkucharc · 2018-12-03T16:28:53Z

Run Seed Job

kkucharc · 2018-12-03T16:44:40Z

Run GroupByKey Java Load Test Direct

kkucharc · 2018-12-03T17:43:44Z

Run Seed Job

robertwb · 2018-12-07T11:43:33Z

Is this ready for review?

kkucharc · 2018-12-09T19:42:38Z

@robertwb I just want to run some tests to be sure if it works correctly on Jenkins. May I ask you for review when it's ready?

kkucharc · 2018-12-10T15:51:05Z

Run Seed Job

kkucharc · 2018-12-10T15:57:57Z

Run GroupByKey Small Java Load Test Direct

kkucharc · 2018-12-10T15:58:05Z

Run GroupByKey Small Java Load Test Dataflow

lgajowy · 2018-12-10T20:13:32Z

I think the "Java GroupByKey Small Load Test Dataflow" job didn't set max number of workers option and that caused a situation where a Load test took all available workers and starve other jobs for an hour.

See here (match by Job id): https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-10_08_16_32-12840883456333022524?project=apache-beam-testing

Full console logs: https://builds.apache.org/job/beam_Java_LoadTests_GroupByKey_Dataflow_Small/5/consoleText

Judging from the logs and the code we are missing maxNumWorkers option in the job. @kkucharc could you add that option?

@alanmyrvold : Sorry for this. How many workers at most can we use for such big job?

kkucharc · 2018-12-11T09:55:56Z

@lgajowy Thank you for letting me know. I got confused after gradle workers parameter. The "Small" test isn't that small after all.

@alanmyrvold I am really sorry for this - I wasn't aware of this consequences. Besides of @lgajowy question about recommended workers amount, is there any way to cancel my own job when I see that something bad is happening?

alanmyrvold · 2018-12-12T00:32:20Z

No problem. Glad you tracked it down. Something like 10 max workers would be reasonable.

kkucharc · 2018-12-13T16:13:38Z

Run Seed Job

kkucharc · 2018-12-13T16:21:58Z

Run GroupByKey Small Java Load Test Dataflow

kkucharc · 2018-12-13T16:47:51Z

Run GroupByKey Small Java Load Test Direct

kkucharc · 2018-12-13T17:19:33Z

This is ready for review. I tested it on smaller input data set and it was succeeded. @swegner and @lgajowy would you take a look at it, please? cc: @pabloem

swegner · 2018-12-13T18:48:12Z

I'm feeling a bit overloaded, and probably won't have time to look at this until next week; would you mind finding another reviewer? You can find additional reviewers in the OWNERS file, and use the code velocity dashboard to see who's overloaded.

kkucharc · 2018-12-14T10:52:34Z

Thank you @swegner for letting me know. Especially I didn't see code velocity dashboard yet.

lgajowy

Thanks @kkucharc! This almost looks good to me, I added some comments to be addressed before merging.

lgajowy · 2018-12-14T12:51:36Z

.test-infra/jenkins/LoadTestsBuilder.groovy

+                rootBuildScriptDir(commonJobProperties.checkoutDir)
+                tasks(':beam-sdks-java-load-tests:run')
+                commonJobProperties.setGradleSwitches(delegate)
+                switches("-Dorg.gradle.daemon=false")


Is skipping gradle daemon needed? If not please remove this line

Thanks a lot that you found this. Some 'leftovers' after testing.

lgajowy · 2018-12-14T12:58:00Z

.test-infra/jenkins/job_LoadTests_Java.groovy

+    create_load_test_job(testConfiguration)
+}
+
+private void create_load_test_job(testConfiguration) {


I think it's better to use "PhraseTriggeringPostCommitBuilder for this but in a loop. See here for reference: job_PostCommit_Java_Nexmark_Dataflow.groovy.

I changed it, thanks for suggestion!

.test-infra/jenkins/job_LoadTests_Java.groovy

lgajowy · 2018-12-14T13:12:24Z

.test-infra/jenkins/LoadTestsBuilder.groovy

+            tempLocation        : 'gs://temp-storage-for-perf-tests/loadtests',
+    ]
+
+    enum Runner {


This is common for Nexmark and load tests - can you move it to a common file and import here? Something like CommonTestProperties.groovy maybe?

Ok, I extracted common part for both suites. Thanks for suggestion.

lgajowy · 2018-12-14T15:28:51Z

@kkucharc please rebase before running the tests again - I provided fixes for BigQuery publishing code (#7283). Thanks!

kkucharc · 2019-01-02T16:27:32Z

Run seed job

kkucharc · 2019-01-02T16:49:34Z

Run GroupByKey Small Java Load Test DIrect

kkucharc · 2019-01-02T16:49:52Z

Run Dataflow Runner Nexmark Tests

kkucharc · 2019-01-03T14:32:05Z

@lgajowy This is ready for the second round of review.

All Jenkins jobs are triggered properly.

The Direct runner test is failing due to OutOfMemoryError described in detail here.
Test on Dataflow is actually failing but status is still success what was reported here and the reason why is failing is reported here.

lgajowy

LGTM. The failures have been addressed with appropriate JIRA issues so I think we can merge. Those are not periodic jobs so the failures can be fixed afterward.

@kkucharc please reorganize the history and add JIRA tags in the commit messages and I will merge

kkucharc · 2019-01-04T15:36:03Z

Run seed job

kkucharc · 2019-01-04T15:40:56Z

Run Dataflow Runner Nexmark Tests

kkucharc · 2019-01-04T15:41:06Z

Run Direct Runner Nexmark Tests

kkucharc · 2019-01-04T15:41:27Z

Run GroupByKey Small Java Load Test Dataflow

…s configuration. Some fixes.

lgajowy · 2019-01-07T11:48:30Z

Thanks! Merging.

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from e2bfc84 to 6dc237c Compare December 3, 2018 17:16

kkucharc changed the title ~~BEAM-5985 Create jenkins jobs to run the load tests for Java SDK~~ [BEAM-5985] Create jenkins jobs to run the load tests for Java SDK Dec 3, 2018

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch 2 times, most recently from 0aacba3 to 2675e26 Compare December 5, 2018 13:22

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch 3 times, most recently from f51e34f to 7d1fd3f Compare December 10, 2018 15:50

lgajowy mentioned this pull request Dec 10, 2018

[BEAM-6188] Unbouded synthetic source #7226

Merged

2 tasks

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from 6c18303 to e61a3a0 Compare December 13, 2018 17:15

lgajowy self-requested a review December 14, 2018 12:27

lgajowy reviewed Dec 14, 2018

View reviewed changes

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from ef72206 to c1beaa7 Compare January 2, 2019 11:17

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from c1beaa7 to ef74eee Compare January 2, 2019 11:21

lgajowy approved these changes Jan 3, 2019

View reviewed changes

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from 39127ee to 89e138a Compare January 4, 2019 15:35

kkucharc and others added 3 commits January 7, 2019 12:04

[BEAM-5985] Create jenkins jobs to run the load tests for Java SDK

0f892b5

[BEAM-5985] Introduced Small Load test suite.

cdf3f1d

[BEAM-5985] Extracted common part for Nexmark and LoadTests in Jenkin…

7604c4a

…s configuration. Some fixes.

kkucharc force-pushed the BEAM-5985-jenkins-java-load-test branch from 89e138a to 7604c4a Compare January 7, 2019 11:06

lgajowy merged commit e565087 into apache:master Jan 7, 2019

[BEAM-5985] Create jenkins jobs to run the load tests for Java SDK #7184

[BEAM-5985] Create jenkins jobs to run the load tests for Java SDK #7184

Conversation

kkucharc commented Dec 3, 2018

Post-Commit Tests Status (on master branch)

kkucharc commented Dec 3, 2018

kkucharc commented Dec 3, 2018

kkucharc commented Dec 3, 2018

robertwb commented Dec 7, 2018

kkucharc commented Dec 9, 2018

kkucharc commented Dec 10, 2018

kkucharc commented Dec 10, 2018

kkucharc commented Dec 10, 2018

lgajowy commented Dec 10, 2018

kkucharc commented Dec 11, 2018

alanmyrvold commented Dec 12, 2018

kkucharc commented Dec 13, 2018

kkucharc commented Dec 13, 2018

kkucharc commented Dec 13, 2018

kkucharc commented Dec 13, 2018

swegner commented Dec 13, 2018

kkucharc commented Dec 14, 2018

lgajowy left a comment

Choose a reason for hiding this comment

lgajowy Dec 14, 2018

Choose a reason for hiding this comment

kkucharc Dec 28, 2018 • edited

Choose a reason for hiding this comment

lgajowy Dec 14, 2018

Choose a reason for hiding this comment

kkucharc Dec 28, 2018

Choose a reason for hiding this comment

lgajowy Dec 14, 2018

Choose a reason for hiding this comment

kkucharc Dec 28, 2018

Choose a reason for hiding this comment

lgajowy commented Dec 14, 2018

kkucharc commented Jan 2, 2019

kkucharc commented Jan 2, 2019

kkucharc commented Jan 2, 2019

kkucharc commented Jan 3, 2019

lgajowy left a comment • edited

Choose a reason for hiding this comment

kkucharc commented Jan 4, 2019

kkucharc commented Jan 4, 2019

kkucharc commented Jan 4, 2019

kkucharc commented Jan 4, 2019

lgajowy commented Jan 7, 2019

kkucharc Dec 28, 2018 •

edited

lgajowy left a comment •

edited