testing: TestBenchmarkBLoopIterationCorrect fails under GOMAXPROCS=1

[dmitshur]

At tip (commit c5adb82), TestBenchmarkBLoopIterationCorrect succeeds under most conditions.

Details

$ go test -run=^TestBenchmarkBLoopIterationCorrect$ -count=1 -v testing 
=== RUN   TestBenchmarkBLoopIterationCorrect
    testing_test.go:978: /var/folders/yn/rz8r25f10z5fmwdg34tz6z1c0000gn/T/go-build514388636/b001/testing.test -test.run=^BenchmarkBLoopPrint$ -test.bench=BenchmarkBLoopPrint -test.v -test.parallel=2 -test.benchtime=2x: <nil>
        goos: darwin
        goarch: arm64
        pkg: testing
        cpu: Apple M4 Pro
        BenchmarkBLoopPrint
            testing_test.go:1020: Ramping up from BenchmarkBLoopPrint
            testing_test.go:1022: Printing from BenchmarkBLoopPrint
            testing_test.go:1022: Printing from BenchmarkBLoopPrint
        BenchmarkBLoopPrint-14    	       2	      4146 ns/op
        PASS
--- PASS: TestBenchmarkBLoopIterationCorrect (0.01s)
PASS
ok  	testing	0.161s

But it fails consistently when run in an environment where GOMAXPROCS is 1:

$ GOMAXPROCS=1 go test -run=^TestBenchmarkBLoopIterationCorrect$ -count=1 -v testing
=== RUN   TestBenchmarkBLoopIterationCorrect
    testing_test.go:978: /var/folders/yn/rz8r25f10z5fmwdg34tz6z1c0000gn/T/go-build2068115649/b001/testing.test -test.run=^BenchmarkBLoopPrint$ -test.bench=BenchmarkBLoopPrint -test.v -test.parallel=2 -test.benchtime=2x: <nil>
        goos: darwin
        goarch: arm64
        pkg: testing
        cpu: Apple M4 Pro
        BenchmarkBLoopPrint
            testing_test.go:1020: Ramping up from BenchmarkBLoopPrint
            testing_test.go:1022: Printing from BenchmarkBLoopPrint
            testing_test.go:1022: Printing from BenchmarkBLoopPrint
        BenchmarkBLoopPrint 	       2	      4729 ns/op
        PASS
    testing_test.go:995: Missing benchmark output
--- FAIL: TestBenchmarkBLoopIterationCorrect (0.01s)
FAIL
FAIL	testing	0.164s
FAIL

Also see https://ci.chromium.org/b/8729873930854023953/test-results as an example where a builder is running into this. (CC @golang/plan9.)

---

The regexp added in CL 630455 (CC @JunyangShao) seems to require a -n suffix after the benchmark name, but that suffix isn't included when n is 1.

[gabyhelp]

Related Issues

• testing: benchmark iteration reports incorrectly #41637 (closed)
• testing: run first benchmark with last value from -cpu list GOMAXPROCS value #49980
• go test -bench #24948 (closed)
• testing: no tests to run warning when running benchmarks #17603 (closed)
• Go bench test never ends #46493 (closed)

Related Code Changes

• cmd/go: make Script/test_regexps less flaky under load
• testing: harmonize handling of prefix-matched benchmarks
• testing: parallelize tests over count
• cmd/compile,testing: implement one-time rampup logic for testing.B.Loop
• testing: fail benchmarks that don't loop over b.N

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[gopherbot]

Change https://go.dev/cl/632655 mentions this issue: testing: consider -N suffix after benchmark name optional