[KAFKA-14441] Benchmark performance impact of metrics library #13177
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clolov
reviewed
Feb 2, 2023
| Metrics metrics = new Metrics(); | ||
|
|
||
| List<Sensor> sensors = new ArrayList<>(); | ||
| @Param({"1", "5", "20"}) |
There was a problem hiding this comment.
In general I am more than happy with this benchmark, but where did these numbers come from? Are they roughly the number of sensors you have seen registered per Connector task?
There was a problem hiding this comment.
Yes, e.g Sink task iteration has around 5 sensor records. Adding a comment to clarify this. Thanks!
Hangleton
reviewed
Feb 2, 2023
| @State(Scope.Benchmark) | ||
| @Fork(value = 1) | ||
| @Warmup(iterations = 5) | ||
| @Measurement(iterations = 5) |
There was a problem hiding this comment.
Is the number of warm-up and iterations really enough? 5 may be a bit low for the code path exercised. Unsure how representative the benchmark is for high-throughput invocations in production environments.
There was a problem hiding this comment.
Sure, I reduced it for testing purpose. Increasing to 15, similar to other benchmarks — as I realize it's possible to override these via args.
There was a problem hiding this comment.
Unsure how representative the benchmark is for high-throughput invocations in production environments.
Yeah, I don't think these are highly representative, but a starting point to compare stats and how adding either sensors or stats affect latency (which is the concern from KIP-864).
Though, happy to hear thoughts on how to make these more solid. e.g. I have an idea to also add a benchmark of Yammer metrics to compare the impact of these two libraries.
Also, found there was some work around it:
- MetricsBench: simple average time execution of a sensor + a bunch of stats.
- Microbenchmarks which seems more elaborated, but haven't looked into details yet.
| @Fork(value = 1) | ||
| @Warmup(iterations = 5) | ||
| @Measurement(iterations = 5) | ||
| @BenchmarkMode(Mode.AverageTime) |
There was a problem hiding this comment.
What is the variance of the distribution? SampleTime may be useful to review the resulting percentiles.
There was a problem hiding this comment.
Sure! I have added SampleTime, and got some interesting stats:
Benchmark (numSensors) Mode Cnt Score Error Units
SensorBenchmark.recordBenchmark 1 avgt 4 122.285 ± 3.831 ns/op
SensorBenchmark.recordBenchmark 5 avgt 4 535.055 ± 10.002 ns/op
SensorBenchmark.recordBenchmark 20 avgt 4 2153.720 ± 562.511 ns/op
SensorBenchmark.recordWithAvgBenchmark 1 avgt 4 148.655 ± 7.245 ns/op
SensorBenchmark.recordWithAvgBenchmark 5 avgt 4 801.836 ± 25.243 ns/op
SensorBenchmark.recordWithAvgBenchmark 20 avgt 4 5934.782 ± 463.325 ns/op
SensorBenchmark.recordWithMaxAvgBenchmark 1 avgt 4 153.731 ± 8.982 ns/op
SensorBenchmark.recordWithMaxAvgBenchmark 5 avgt 4 1067.862 ± 68.656 ns/op
SensorBenchmark.recordWithMaxAvgBenchmark 20 avgt 4 10486.323 ± 209.097 ns/op
SensorBenchmark.recordWithMaxBenchmark 1 avgt 4 148.492 ± 20.097 ns/op
SensorBenchmark.recordWithMaxBenchmark 5 avgt 4 807.746 ± 300.785 ns/op
SensorBenchmark.recordWithMaxBenchmark 20 avgt 4 6361.096 ± 1160.907 ns/op
SensorBenchmark.recordWithMinMaxAvgBenchmark 1 avgt 4 178.676 ± 12.371 ns/op
SensorBenchmark.recordWithMinMaxAvgBenchmark 5 avgt 4 1749.455 ± 204.069 ns/op
SensorBenchmark.recordWithMinMaxAvgBenchmark 20 avgt 4 25027.613 ± 1407.740 ns/op
SensorBenchmark.recordWithMinMaxBenchmark 1 avgt 4 158.032 ± 9.249 ns/op
SensorBenchmark.recordWithMinMaxBenchmark 5 avgt 4 1001.310 ± 48.258 ns/op
SensorBenchmark.recordWithMinMaxBenchmark 20 avgt 4 8958.896 ± 264.169 ns/op
SensorBenchmark.recordWithPercentileBenchmark 1 avgt 4 133.637 ± 7.675 ns/op
SensorBenchmark.recordWithPercentileBenchmark 5 avgt 4 766.876 ± 14.212 ns/op
SensorBenchmark.recordWithPercentileBenchmark 20 avgt 4 5661.163 ± 463.749 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark 1 avgt 4 119.896 ± 13.392 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark 5 avgt 4 663.232 ± 9.704 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark 20 avgt 4 4976.913 ± 430.296 ns/op
SensorBenchmark.recordBenchmark 1 sample 1306062 176.717 ± 2.809 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.00 1 sample 92.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.50 1 sample 135.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.90 1 sample 150.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.95 1 sample 156.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.99 1 sample 181.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.999 1 sample 19325.984 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.9999 1 sample 32108.598 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p1.00 1 sample 195328.000 ns/op
SensorBenchmark.recordBenchmark 5 sample 1023159 694.906 ± 7.662 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.00 5 sample 455.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.50 5 sample 565.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.90 5 sample 780.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.95 5 sample 847.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.99 5 sample 1214.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.999 5 sample 24192.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.9999 5 sample 60338.432 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p1.00 5 sample 716800.000 ns/op
SensorBenchmark.recordBenchmark 20 sample 1271143 2011.945 ± 7.842 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.00 20 sample 1610.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.50 20 sample 1750.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.90 20 sample 2100.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.95 20 sample 2892.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.99 20 sample 3616.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.999 20 sample 28576.000 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p0.9999 20 sample 77425.357 ns/op
SensorBenchmark.recordBenchmark:recordBenchmark·p1.00 20 sample 1118208.000 ns/op
SensorBenchmark.recordWithAvgBenchmark 1 sample 1262271 179.238 ± 13.287 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.00 1 sample 102.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.50 1 sample 140.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.90 1 sample 149.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.95 1 sample 155.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.99 1 sample 174.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.999 1 sample 16272.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.9999 1 sample 29304.730 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p1.00 1 sample 4165632.000 ns/op
SensorBenchmark.recordWithAvgBenchmark 5 sample 933071 719.230 ± 4.662 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.00 5 sample 582.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.50 5 sample 633.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.90 5 sample 681.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.95 5 sample 718.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.99 5 sample 763.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.999 5 sample 24544.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.9999 5 sample 36588.339 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p1.00 5 sample 178176.000 ns/op
SensorBenchmark.recordWithAvgBenchmark 20 sample 1042420 4797.208 ± 8.407 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.00 20 sample 4052.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.50 20 sample 4496.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.90 20 sample 4712.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.95 20 sample 4976.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.99 20 sample 20864.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.999 20 sample 32480.000 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p0.9999 20 sample 64432.506 ns/op
SensorBenchmark.recordWithAvgBenchmark:recordWithAvgBenchmark·p1.00 20 sample 113280.000 ns/op
sample 204288.000 ns/op
...
SensorBenchmark.recordWithWindowedSumBenchmark 20 sample 1056588 4733.302 ± 8.413 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.00 20 sample 4052.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.50 20 sample 4416.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.90 20 sample 4656.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.95 20 sample 4976.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.99 20 sample 20672.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.999 20 sample 33088.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p0.9999 20 sample 64064.000 ns/op
SensorBenchmark.recordWithWindowedSumBenchmark:recordWithWindowedSumBenchmark·p1.00 20 sample 160000.000 ns/op
JMH benchmarks done
|
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/KAFKA-14441
Committer Checklist (excluded from commit message)