Skip to content

testing: early break in a testing.B.Loop benchmark loop should be reported. #72933

Description

@JunyangShao

Go version

go version go1.24.1 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/shaojunyang/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/shaojunyang/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/6t/d5y6mrc90rzd1gm2q2k99lb400zg0c/T/go-build3077995788=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/Users/shaojunyang/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/shaojunyang/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/shaojunyang/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

For benchmark that breaks with its own logic, -benchtime won't work and the users should be notified.

What did you see happen?

If we run the following benchmark with go test -bench=. -benchtime=1s:

func BenchmarkFoo(b *testing.B) {
        i := 0
        for b.Loop() {
                fmt.Printf("iter %d\n", i)
                i += 1
		if i > 5 {
			break
		}
                time.Sleep(100 * time.Millisecond)
        }
}

The output would be

iter 0
iter 1
iter 2
iter 3
iter 4
iter 5
goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkFoo-10    	      10	  50555750 ns/op
PASS
ok  	command-line-arguments	0.732s

What did you expect to see?

Instead of allowing the benchmark to pass with bogus numbers, we should let the user know that the benchmark failed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FixPendingIssues that have a fix which has not yet been reviewed or submitted.FrozenDueToAge

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions