[BEAM-5983] Create Combine load test#6985
Conversation
| applyStepIfPresent(input, format("Step: %d", i), syntheticStep) | ||
| .apply( | ||
| format("Convert to BigInteger: %d", i), MapElements.via(new ByteValueToBigInteger())) | ||
| .apply(format("Combine: %d", i), combiner); |
There was a problem hiding this comment.
I need some feedback about what Combine modes should we test. Are Top.perKeyLargest and Mean.perKey combine operations enough?
aaltay
left a comment
aaltay
left a comment
There was a problem hiding this comment.
Benchmark itself looks good, we could add more options.
| applyStepIfPresent(input, format("Step: %d", i), syntheticStep) | ||
| .apply( | ||
| format("Convert to BigInteger: %d", i), MapElements.via(new ByteValueToBigInteger())) | ||
| .apply(format("Combine: %d", i), combiner); |
| } | ||
|
|
||
| /** Pipeline options specific for this test. */ | ||
| interface Options extends LoadTestOptions { |
There was a problem hiding this comment.
Could we have some standard configs to run this load test? Perhaps a wrapper to run this with a set of parameters
I can think of:
Uniform
Normal
Onekey
Fanout 256 (could be uniform)
(+ maybe a few hot keys in a larger distribution.)
There was a problem hiding this comment.
I agree. This also applies to other tests (Gbk, CoGbk, ParDo etc). It's probably worth to add default parameter sets in a separate PR (I find it a cross-cutting concern). Wdyt?
There was a problem hiding this comment.
Created JIRA for this: https://issues.apache.org/jira/browse/BEAM-6065
|
@aaltay applied your suggestions. PTAL again. |
|
Run Java PreCommit |
|
Thanks! |
2 participants
@aaltay could you take a look at this too?
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.It will help us expedite review of your Pull Request if you tag someone (e.g.
@username) to look at it.Post-Commit Tests Status (on master branch)