New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Benchmark reporting incorrect stats when running in parallel #28813

Open
mnewswanger opened this Issue Nov 15, 2018 · 1 comment

Comments

Projects
None yet
3 participants
@mnewswanger

mnewswanger commented Nov 15, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11 linux/amd64

Applies to earlier versions as well

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build896591388=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Using the following test file:

package bechmarkperf

import (
	"strconv"
	"testing"
	"time"
)

func BenchmarkSleepSerial(b *testing.B) {
	for i := 0; i < b.N; i++ {
		sleepFunc()
	}
}

func BenchmarkSleepParallel(b *testing.B) {
	for i := 1; i <= 8; i *= 2 {
		b.Run(strconv.Itoa(i), func(b *testing.B) {
			b.SetParallelism(i)
			b.RunParallel(func(pb *testing.PB) {
				for pb.Next() {
					sleepFunc()
				}
			})
		})
	}
}

func sleepFunc() {
	time.Sleep(1 * time.Millisecond)
}

Running in parallel, results were ~1ms / for ns/op. This mean that doubling threadcount would halve ns/op.

What did you expect to see?

ns/op hold the same and not go below 1ms.

What did you see instead?

When running with 128 goroutines, go the following output:

BenchmarkSleepParallel/8-16  	  100000	     12024 ns/op	       0 B/op	       0 allocs/op

This is reporting that the benchmark process is taking well under 1ms (which is the sleep duration above).

@gopherbot

This comment has been minimized.

gopherbot commented Nov 15, 2018

Change https://golang.org/cl/149720 mentions this issue: testing: correct benchmark output when running in parallel

@ianlancetaylor ianlancetaylor changed the title from Benchmark reporting incorrect stats to testing: Benchmark reporting incorrect stats when running in parallel Nov 16, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.13 milestone Nov 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment