[BEAM-7504] Created Combine Python Load Test Jenkins job #8813
Conversation
|
Run Seed Job |
|
Run Seed Job |
|
Run Seed Job |
|
Run Python Load Tests Combine Dataflow Batch |
|
Run Seed Job |
|
Run Python Load Tests Combine Dataflow Batch |
|
R: @kkucharc |
|
|
||
| self.top_count = self.pipeline.get_option('top_count') | ||
| if self.top_count is None: | ||
| self.fail('You should set \"--topCount\" option to use TOP combiners') |
There was a problem hiding this comment.
You have typo here topCount -> top_count
| self.top_count = self.pipeline.get_option('top_count') | ||
| if self.top_count is None: | ||
| self.fail('You should set \"--topCount\" option to use TOP combiners') | ||
| self.top_count = int(self.top_count) |
There was a problem hiding this comment.
What do you think about changing this and above construction to just try int() and then catch Type and Value exception? (for cases it's None or not a number value)
| input_options : '\'{"num_records": 20000000,' + | ||
| '"key_size": 10,' + | ||
| '"value_size": 90,' + | ||
| '"bundle_size_distribution_type": "const",' + |
There was a problem hiding this comment.
bundle_size_distribution_type, bundle_size_distribution_param and force_initial_num_bundles have default values -> we don't have to specify them here. WDYT about removing it from here?
| publish_to_big_query: true, | ||
| metrics_dataset : 'load_test', | ||
| metrics_table : 'python_dataflow_batch_combine_5', | ||
| input_options : '\'{"num_records": 20000000,' + |
There was a problem hiding this comment.
Here is also 20000000/8 = 2500000
| publish_to_big_query: true, | ||
| metrics_dataset : 'load_test', | ||
| metrics_table : 'python_dataflow_batch_combine_4', | ||
| input_options : '\'{"num_records": 20000000,' + |
There was a problem hiding this comment.
In proposal there is 20000000/4 = 5000000
There was a problem hiding this comment.
Oh, that's right. I didn't notice.
| '"bundle_size_distribution_type": "const",' + | ||
| '"bundle_size_distribution_param": 1,' + | ||
| '"force_initial_num_bundles": 1}\'', | ||
| max_num_workers : 5, |
There was a problem hiding this comment.
I know that max_num_workers and num_workers change autoscaling_algorithm to NONE, but maybe it would be good to be sure here and specify autoscaling_algorithm: "NONE" just in case, WDYT?
|
@kkucharc Thanks for review, I've just pushed my fixes. |
|
Run Python PreCommit |
|
R: @iemejia Could you take a look? |
|
@kamilwu sorry for the delayed review, it looks good, but can you please get the execution of the tests running. I will merge just afterwards. |
|
Run seed job |
|
Run Python Load Tests Combine Dataflow Batch |
|
Run Seed Job |
|
Run Portable_Python PreCommit |
|
Run Python Load Tests Combine Dataflow Batch |
|
Run Python PreCommit |
|
Run seed job |
|
Run Python Load Tests Combine Dataflow Batch |
|
@iemejia It's fine, thanks for your review. PR is now ready to be merged. |
|
Thanks @kamilwu it was quite hard to make jenkins happy but finally. |
3 participants
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.