[BEAM-5984] Enable publishing load test results to Big Query#7090
[BEAM-5984] Enable publishing load test results to Big Query#7090lgajowy merged 3 commits intoapache:masterfrom
Conversation
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestOptions.java
Show resolved
Hide resolved
| .build(); | ||
|
|
||
| bigQueryClient.insertRow(record, tableName); | ||
| bigQueryClient.insertRow(record, schema, tableName); |
There was a problem hiding this comment.
Should this test be changed to use the ResultPublisher? I see that the schemas are a tad different? runtimeSec vs runtime, etc?
There was a problem hiding this comment.
This is actually a good question. I refactored a little bit so that the BigQuery publisher part could be reused. Please let me know what do you think of it.
|
@pabloem I refactored a little bit more so that more code could be reused (please see the new commit description) What is more, current implementation allows us to easily make code responsible for publishing to console common too. I didn't want to do this now because the change would be huge. Could you take a look again? |
|
Run Direct Runner Nexmark Tests |
|
Run Spark Runner Nexmark Tests |
|
Run Java PreCommit |
|
@pabloem pinging this kindly :) |
|
Sorry about the long delay : / - Anyway, thanks, this looks great : ) |
|
Thanks, @pabloem! I rebased and will wait for the results to be green before merging. |
- Console and BigQuery publishers are now separated. - Additional tests were added for BigQueryPublisher - LoadTestResult POJO was created for result encapsulation - Common interface for NexmarkPerf and LoadTestResult was introduced
|
The results are green - merging. |
2 participants
This PR provides a way to publish metrics to BQ after specifying appropriate pipeline options.
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.It will help us expedite review of your Pull Request if you tag someone (e.g.
@username) to look at it.Post-Commit Tests Status (on master branch)