testing: Benchmark reporting incorrect stats when running in parallel #28813
Description
What version of Go are you using (go version)?
$ go version go version go1.11 linux/amd64
Applies to earlier versions as well
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GOARCH="amd64" GOBIN="" GOCACHE="/root/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build896591388=/tmp/go-build -gno-record-gcc-switches"
What did you do?
Using the following test file:
package bechmarkperf
import (
"strconv"
"testing"
"time"
)
func BenchmarkSleepSerial(b *testing.B) {
for i := 0; i < b.N; i++ {
sleepFunc()
}
}
func BenchmarkSleepParallel(b *testing.B) {
for i := 1; i <= 8; i *= 2 {
b.Run(strconv.Itoa(i), func(b *testing.B) {
b.SetParallelism(i)
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
sleepFunc()
}
})
})
}
}
func sleepFunc() {
time.Sleep(1 * time.Millisecond)
}
Running in parallel, results were ~1ms / for ns/op. This mean that doubling threadcount would halve ns/op.
What did you expect to see?
ns/op hold the same and not go below 1ms.
What did you see instead?
When running with 128 goroutines, go the following output:
BenchmarkSleepParallel/8-16 100000 12024 ns/op 0 B/op 0 allocs/op
This is reporting that the benchmark process is taking well under 1ms (which is the sleep duration above).