New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: only find the target b.N once when -count > 1 #23423

Open
cespare opened this Issue Jan 12, 2018 · 3 comments

Comments

Projects
None yet
5 participants
@cespare
Contributor

cespare commented Jan 12, 2018

go test runs benchmarks with increasing b.N values to find a large enough value depending on benchtime. If you run with -count > 1, it will do that every time rather than just once.

For instance, with this code:

func BenchmarkBenchmark(b *testing.B) {
	start := time.Now()
	for i := 0; i < b.N; i++ {
		time.Sleep(10 * time.Millisecond)
	}
	b.Log("single call took", time.Since(start))
}

if I run

go1.10beta2 test -bench . -benchtime 5s -count 2 bench_test.go

I get

goos: linux
goarch: amd64
BenchmarkBenchmark-4        1000          10096302 ns/op
--- BENCH: BenchmarkBenchmark-4
        bench_test.go:13: single call took 10.07786ms
        bench_test.go:13: single call took 1.010803756s
        bench_test.go:13: single call took 10.096243342s
BenchmarkBenchmark-4        1000          10092966 ns/op
--- BENCH: BenchmarkBenchmark-4
        bench_test.go:13: single call took 10.08295ms
        bench_test.go:13: single call took 1.008574479s
        bench_test.go:13: single call took 10.092920926s
PASS
ok      command-line-arguments  22.233s

In this case, if go test had remembered the target value (1000) the first time, it would've avoided two adjustment runs the second time around and would have saved about 1s. The effect is larger when -count is higher, as is typical for people gathering multiple runs for use with benchstat.

(Side note: it would've been nice if go test had picked a smaller value than 1000 here. It took 10s when I asked for 5s.)

It's possible that the fix for #19128 (-benchsplit) will also incidentally fix this. I haven't studied the CL yet.

/cc @rsc @josharian @bcmills

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot May 2, 2018

Change https://golang.org/cl/110775 mentions this issue: testing: only compute b.N once when passed -count > 1

gopherbot commented May 2, 2018

Change https://golang.org/cl/110775 mentions this issue: testing: only compute b.N once when passed -count > 1

@ALTree

This comment has been minimized.

Show comment
Hide comment
@ALTree

ALTree May 2, 2018

Member

Moderately ugly fix I'm not 100% sure about in the change gopherbot linked above.

Member

ALTree commented May 2, 2018

Moderately ugly fix I'm not 100% sure about in the change gopherbot linked above.

@gopherbot gopherbot closed this in 05ca340 May 14, 2018

@ALTree

This comment has been minimized.

Show comment
Hide comment
@ALTree

ALTree May 31, 2018

Member

Re-opening this since the change linked above was reverted.

See #25622 for a description of the issues it was causing.

Member

ALTree commented May 31, 2018

Re-opening this since the change linked above was reverted.

See #25622 for a description of the issues it was causing.

@ALTree ALTree reopened this May 31, 2018

@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment