Repository navigation
proposal: testing: add a flag to report benchmark metrics for each run #51218
Description
I would like the package to be able to report metrics for each run, not just an aggregate.
I want to be able to analyze and graph the data using external tools, so I need the runtime etc. of each iteration of the benchmark. Using the built-in benchmarking is great but I need to see the distribution of individual iterations.
Activity
davecheney commentedon Feb 16, 2022
The reason the benchmark tool suppresses the intermediate benchmark runs is they are not statistically significant. Even the final benchmark number is not accurate in and of itself, it should be combined with flags like -count and tools like benchstat to account for warmup and cpu throttling issues.
[-]proposal: testing(benchmark): add a flag to report metrics for each run[/-][+]proposal: testing: add a flag to report benchmark metrics for each run[/+]rsc commentedon Feb 16, 2022
As @davecheney said, -count is the way to get multiple runs. You can also drop -benchtime to be lower than the default 2s if you want many short runs.
There is no way to get the distribution of every iteration in a particular run, because the benchmark functions themselves do that loop from 0 to b.N.
rsc commentedon Feb 16, 2022
No change in consensus, so declined.
— rsc for the proposal review group