proposal: testing: add a flag to report benchmark metrics for each run

[BrennaEpp]

I would like the package to be able to report metrics for each run, not just an aggregate.

I want to be able to analyze and graph the data using external tools, so I need the runtime etc. of each iteration of the benchmark. Using the built-in benchmarking is great but I need to see the distribution of individual iterations.

[davecheney]

The reason the benchmark tool suppresses the intermediate benchmark runs is they are not statistically significant. Even the final benchmark number is not accurate in and of itself, it should be combined with flags like -count and tools like benchstat to account for warmup and cpu throttling issues.

[rsc]

As @davecheney said, -count is the way to get multiple runs. You can also drop -benchtime to be lower than the default 2s if you want many short runs.

There is no way to get the distribution of every iteration in a particular run, because the benchmark functions themselves do that loop from 0 to b.N.

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group