Repository navigation
testing: only find the target b.N once when -count > 1 #23423
Description
go test runs benchmarks with increasing b.N values to find a large enough value depending on benchtime. If you run with -count > 1, it will do that every time rather than just once.
For instance, with this code:
func BenchmarkBenchmark(b *testing.B) {
start := time.Now()
for i := 0; i < b.N; i++ {
time.Sleep(10 * time.Millisecond)
}
b.Log("single call took", time.Since(start))
}
if I run
go1.10beta2 test -bench . -benchtime 5s -count 2 bench_test.go
I get
goos: linux
goarch: amd64
BenchmarkBenchmark-4 1000 10096302 ns/op
--- BENCH: BenchmarkBenchmark-4
bench_test.go:13: single call took 10.07786ms
bench_test.go:13: single call took 1.010803756s
bench_test.go:13: single call took 10.096243342s
BenchmarkBenchmark-4 1000 10092966 ns/op
--- BENCH: BenchmarkBenchmark-4
bench_test.go:13: single call took 10.08295ms
bench_test.go:13: single call took 1.008574479s
bench_test.go:13: single call took 10.092920926s
PASS
ok command-line-arguments 22.233s
In this case, if go test had remembered the target value (1000) the first time, it would've avoided two adjustment runs the second time around and would have saved about 1s. The effect is larger when -count is higher, as is typical for people gathering multiple runs for use with benchstat.
(Side note: it would've been nice if go test had picked a smaller value than 1000 here. It took 10s when I asked for 5s.)
It's possible that the fix for #19128 (-benchsplit) will also incidentally fix this. I haven't studied the CL yet.
/cc @rsc @josharian @bcmills
Activity
gopherbot commentedon May 2, 2018
Change https://golang.org/cl/110775 mentions this issue:
testing: only compute b.N once when passed -count > 1ALTree commentedon May 2, 2018
Moderately ugly fix I'm not 100% sure about in the change gopherbot linked above.
ALTree commentedon May 31, 2018
Re-opening this since the change linked above was reverted.
See #25622 for a description of the issues it was causing.
cespare commentedon Nov 23, 2024
In Go 1.24, benchmarks that use
testing.B.Loopdo ramp-up inside the test function and only call it once. Therefore, this problem goes away once you switch to usingb.Loop, so I'm going to close this issue.Thanks @aclements / @JunyangShao!
b.Loopis a really nice improvement.