Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
testing: add ratcheting variants #7465
For some testing and benchmark purposes, a ratchet is better suited than an average. https://golang.org/cl/67870053/ bumps up the number of AllocsPerRun runs of an http test to avoid flakiness. This test would be more reliable using a lower number of runs if it could measure the best run rather than the average. In addition, it could set an explicit (rather than comparative) goal for the number of allocs, which would allow it to catch other regressions. With care, MinAllocsPerRun could even use heuristics to avoid requiring the user to pass an explicit number of runs. For benchmarking tightly CPU-bound code with minimal scheduler/OS interactions, a ratcheting benchmark will often yield more stable, useful results than an averaging benchmark.
referenced this issue
Mar 6, 2017
For allocs, I agree that it would be nice to fix AllocsPerRun in some ideal world, although we're a bit stuck with it now. I'm also not sure we can build an API with no runs parameter: it seems like at the least you need a max count. If f is expensive then you might not want to run it very many times, and if f is unstable then you need to cut it off at some point. It might be nice to sketch out a
For CPU, I think the number of times when you actually want just a ratchet is pretty low. Modern systems are weird enough that even the lowest possible observed time can be misleading. Maybe 99% of the time the top takes 5ns but occasionally the stars align just right and it takes 3ns. I've seen craziness like this. Then the min of all the runs is noisier than the average. I do think we should expose the underlying distribution, as in #19128, which is much better than any one number.
Given #19128, can we trim this issue down to being just about allocation counting?