Reducing the usage of Jmh Invocation level on setup #6459
Conversation
|
Nice catch! |
There was a problem hiding this comment.
I actually think this benchmark can just be deleted, it was added to check a performance bugfix, but isn't really testing anything specific to druid, rather just comparing the performance of removeAll vs looping remove. It was useful to prove the changes were the correct changes to make in that PR, but probably doesn't need to remain in the codebase.
Would you mind updating this PR to just delete this benchmark? Thanks for being on the lookout for these issues though, it is appreciated! 👍
|
@DiegoEliasCosta Thanks for taking the time to look into this! I'll second the suggestion from @clintropolis, I think this benchmark should really be removed. |
|
Okay then! I just removed LoadStatusBenchmark class from the codebase. |
Motivation:
We are conducting a scientific study to investigate bad practices/anti-patterns on creating micro-benchmarks using JMH, and we found an instance of harmfull usage of Invocation level in
LoadStatusBenchmark.The usage of Invocation level for JMH fixture methods (setup/teardown) inccurs in a significant impact in
in the benchmark time (see JMH Documentation).
When benchmark and setup/teardown is too small (less than a milisecond) the Invocation level might saturate the system with timestamp requests and iteration synchronizations which introduce artificial latency, throughput, and scalability bottlenecks.
Modification:
I moved the code that doesn't need to be created on every invocation into a setup with Trial level.
Results
In all our tests, benchmarks run faster now (from 30-50%), without the high JMH Overhead.
I am reporting the result of 5 full executions with default parameters (iterations, forks, warmups, etc..)
druid-fixed-antipatters-invo.zip
Environment:
Tests run on a Computational server with CPU: E5-1660-3.3GHZ (6 cores + HT), 64 GB RAM.