Repository navigation
testing: benchmarks timing issue when using -count > 1 #25622
Description
@laboger reported in #25520 that 05ca340 introduced timing issues in the results of some benchmarks.
For example, the ChanNonblocking benchmark in runtime is much slower after the 1st iteration on ppc64le
BenchmarkChanNonblocking-16 1000000000 3.15 ns/op
BenchmarkChanNonblocking-16 1000000000 32.7 ns/op
The benchmark uses RunParallel. It seems the the change linked above may have introduced timing issues on benchmarks that use RunParallel. For example, using this:
package p_test
import (
"bytes"
"testing"
"text/template"
)
func BenchmarkF(b *testing.B) {
templ := template.Must(template.New("test").Parse("Hello, {{.}}!"))
b.RunParallel(func(pb *testing.PB) {
var buf bytes.Buffer
for pb.Next() {
buf.Reset()
templ.Execute(&buf, "World")
}
})
}
I see on go1.10:
BenchmarkF-4 10000000 188 ns/op
BenchmarkF-4 10000000 190 ns/op
BenchmarkF-4 10000000 189 ns/op
BenchmarkF-4 10000000 188 ns/op
BenchmarkF-4 10000000 189 ns/op
tip:
BenchmarkF-4 10000000 187 ns/op
BenchmarkF-4 10000000 198 ns/op
BenchmarkF-4 10000000 195 ns/op
BenchmarkF-4 10000000 195 ns/op
BenchmarkF-4 10000000 194 ns/op
notice how on tip every run after the 1st consistently reports an higher ns/op.
Activity
[-]test: benchmarks timing issue when using -count > 1[/-][+]testing: benchmarks timing issue when using -count > 1[/+]erifan commentedon May 31, 2018
The same issue on arm64:
goos: linux
goarch: arm64
pkg: encoding/json
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 3.51 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 77.9 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 64.9 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 3.55 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 64.6 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 63.6 ns/op
ALTree commentedon May 31, 2018
I looked into this yesterday and I got nowhere. I'll send a revert for that change.
gopherbot commentedon May 31, 2018
Change https://golang.org/cl/115535 mentions this issue:
Revert "testing: only compute b.N once when passed -count > 1"ALTree commentedon May 31, 2018
The change was reverted. @laboger @erifan if you could verify that the bench regression is gone on ppc64le and arm64 that would be great.
laboger commentedon May 31, 2018
I noticed this went in and tried it right away -- looks like it is now working as expected for ppc64le.
erifan commentedon Jun 4, 2018
It is now working as expected for arm64.