testing: add -benchsplit to get more data points

[bcmills]

For #18177, we need to make tradeoffs between worst-case latency and average throughput for the functions in question.

It's relatively easy to measure the average throughput today using benchmarks with the testing package.

It would be nice if I could use those same benchmarks to measure the distribution of timings across iterations. That isn't really feasible using (*B).N with caller-side iteration, but it might at least be possible to tap into (*PB).Next to get some idea of the latency between calls for parallel benchmarks.

[gopherbot]

CL https://golang.org/cl/37153 mentions this issue.

[gopherbot]

CL https://golang.org/cl/37342 mentions this issue.

[rsc]

/cc @aclements

[josharian]

Related: #7465

[aclements]

That isn't really feasible using (*B).N with caller-side iteration

One possibility is to crank up N only far enough to eliminate clock measurement overhead and then use caller-driven iteration to drive up the number of samples/total iterations to the desired measurement interval. However, this may not work well with benchmarks that perform non-trivial setup and use B.ResetTimer.

[josharian]

benchmarks that perform non-trivial setup and use B.ResetTimer

Also related: #10930. I bring it up because I learned there that you can in fact detect this case--both the case in which b.ResetTimer is called and the case in which b.ResetTimer is (erroneously) not called. Just look at how the total wall time to run the benchmark compares to the time as recorded by the timer.

[rsc]

We can also tell whether b.ResetTimer is in use because we control the implementation of b.ResetTimer.

One of the reasons we've moved from benchcmp to benchstat is that the latter allows multiple data points for a given benchmark. Today those data points are all for roughly the same large size N, but the eventual plan was to emit more lines about smaller N, so that benchstat can do more work with distributions. If anyone wants to experiment with that and let us know how it goes, please do.

I'd also like to have a check somewhere (in package testing or benchstat, probably the former) that b.N really is scaling linearly. If the time for b.N is 4X that of b.N/2 and 16X that of b.N/4, the benchmark function is using b.N incorrectly and shouldn't be reported at all. Doing this in package testing, without printing the smaller runs at all, would let benchstat continue to assume that all the reported runs are about the same size.

[josharian]

Adding a correlation coefficient test to benchmarks seems like a good idea. We could also use the estimated slope of a (weighted?) linear regression as the ns/op instead of straight division, which in theory might yield better results.

I tried hacking in a quick-and-dirty linear regression and got plausible slope numbers. (I didn't mail it because I used a naive rather than a numerically stable calculation.)

I don't plan to work on this further at the moment.

[rsc]

A few years ago I spent a while looking at what Haskell's benchmark tool does. I convinced myself that trying to be clever about slope vs N=0 cost was probably not worthwhile - better to brute force enough iterations that the slope is dominant.

The initial version of Haskell's tool (I forget the name) apparently did fancy things like boosting too, but that seems like overkill too, super subtle and easy to quietly get wrong. Later versions seemed to have backed off some of the complexity.

[josharian]

I convinced myself that trying to be clever about slope vs N=0 cost was probably not worthwhile - better to brute force enough iterations that the slope is dominant.

Fair enough. Looking at the N=0 cost may still be worth doing, if only to catch bugs like dbf533a. (Of course, one could brute force enough iterations for that, but it seems inadvisable.)

[rsc]

I'd like to move this along, assuming that we're not going to introduce a new API for writing benchmarks (so no equivalent of pb.Next) and that we can't emit extra output lines by default (preserving the current one benchmark = one output line). And when we do emit extra output lines, all the lines need to be about the same thing (no warmups to filter out from averaging calculations in benchstat or other tools).

That implies that we need to add some flag that causes a benchmark to be run multiple times for smaller counts and then with one line of output per each of those runs.

Assuming the default -benchtime=1s, if instead of printing output for one run of 1s we want to print output for 10 runs of 0.1s, this can be specified in two possible ways: by setting the target run duration with the count implicit (-benchtime2=0.1s), or by setting the target number of runs with the duration implicit (-benchsplit=10). We only want to add one of these to package testing.

I don't have a great name for the target run duration (obviously), but I think benchsplit could work if we decide to specify the number of runs instead.

The split factor is subject to change to respect the overall time limit. If, say, one iteration of a benchmark takes 0.6 seconds, then with -benchtime=1s -benchsplit=10, we'll still only run ~2 of them, not 10. This makes it sensible to run go test -benchsplit=10 in the go1 directory, as compared to go test -benchtime=0.1 -count=10. The latter runs even very expensive benchmarks 10 times, while the former only runs them once.

So my proposal for this proposal is to add -benchsplit=N as described above.

I don't know if this really helps @bcmills's original request. If not I apologize for hijacking. We've been talking about getting this data out for years, and it does provide more fine-grained detail about the distribution of runs, just not individual runs. If there are rare high-latency individual runs then hopefully you'd be able to split down to a small enough interval that some are lucky and some are not, and then that would come out in the data as a bimodal distribution.

[bcmills]

I don't think that really addresses my original request, unfortunately: for the kinds of latency distributions I'm interested in, one output line per run would be far more data than I want to sift through.

But it does at least seem like a step in the right direction.

[josharian]

Agreed: Seems like a step in the right direction, and I would definitely use this. The interaction between slow benchmarks and -count has always been painful.

To be clear on the (non-)interaction between -benchsplit and -count, if you did -benchtime=1s -benchsplit=10 -count=5, you'd get anywhere between 5 and 50 lines of output?

[rsc]

@bcmills, sifting through the output is for computers.

@josharian, yes.

for 0..count {
    total = 0
    for total < benchtime {
        t = runAndReport(benchtime/benchsplit)
        total += t
    }
}

[rsc]

Objections to adding -benchsplit?

/cc @aclements @mpvl

[rsc]

Sounds like there are no objections to -benchsplit. Anyone want to implement it?

[meirf]

(I've started working on this and plan on having a CL by early next week. Hopefully my recent testing contributions grant me such a runway.)

[gopherbot]

CL https://golang.org/cl/47411 mentions this issue.

[gopherbot]

Change https://golang.org/cl/80841 mentions this issue: testing: remove claim that b.Run is safe for concurrent use

[cespare]

Has progress on this stalled?

[cespare]

Answering my own question from a year ago: yes, it seems like CL 47411 has stalled out, but the proposal is approved and the details seem to have been mostly ironed out in the extensive discussion on that CL. I think anyone could pick up these changes and take them across the finish line.