Handle Google Benchmark 'aggregate' results #276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, I assumed all results generated by Google Benchmark were of 'iteration' type. Now that I have more experience with Google Benchmark, I've started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms. Signed-off-by: Scott K Logan <logans@cottsay.net>
nuclearsandwich
approved these changes
Sep 3, 2020
There was a problem hiding this comment.
I spent some time convincing myself that the properties not specific to aggregation or iteration were handled by the generator in convert_extra_metrics. I got briefly hung up on a parameter name which I've commented on specifically but none of the comments are blocking.
ament_cmake_google_benchmark/ament_cmake_google_benchmark/__init__.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Scott K Logan <logans@cottsay.net>
ahcorde
pushed a commit
that referenced
this pull request
Oct 2, 2020
Previously, I assumed all results generated by Google Benchmark were of 'iteration' type. Now that I have more experience with Google Benchmark, I've started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms. Signed-off-by: Scott K Logan <logans@cottsay.net>
ahcorde
pushed a commit
that referenced
this pull request
Oct 6, 2020
Previously, I assumed all results generated by Google Benchmark were of 'iteration' type. Now that I have more experience with Google Benchmark, I've started generating aggregate results, which contain some different properties. This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms. Signed-off-by: Scott K Logan <logans@cottsay.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, I assumed all results generated by Google Benchmark were of 'iteration' type. Now that I have more experience with Google Benchmark, I've started generating aggregate results, which contain some different properties.
This change adds support for aggregate results and should make it easy to add any other result schemas we encounter in the future. For forward-compatibility, unsupported types will generate a warning message but will not fail the test. This makes the conversion tolerant to Google Benchmark adding new measures for existing mechanisms.
This change also uses the Jenkins Benchmark plugin's specific value types for numeric and boolean results. The reason all numbers are treated as floating point is that floating point numbers close enough to a round number seem to lose their decimal point and look like an integer. To make the data type consistent across runs, I'm opting to treat everything as a floating point number.
Follow-up to #275