testing: Creating sub-benchmark with b.Run means that parent will not show anything from b.ReportMetric

[HaraldNordgren]

Go version

go version go1.21.6 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/Harald/Library/Caches/go-build'
GOENV='/Users/Harald/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/Harald/go/pkg/mod'
GONOPROXY='github.com/dietdoctor/*'
GONOSUMDB='github.com/dietdoctor/*'
GOOS='darwin'
GOPATH='/Users/Harald/go'
GOPRIVATE='github.com/dietdoctor/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.6/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.6/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.6'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/Harald/dd/hive/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/j2/0w9hqz1x01g3j4_yz0w0sw780000gp/T/go-build4150210059=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

func BenchmarkTest(b *testing.B) {
	for _, name := range []string{"Test"} {
		b.Run(name, func(b *testing.B) {
			for _, subName := range []string{"Subtest"} {
				b.Run(subName, func(bb *testing.B) {
					bb.ReportMetric(100, "sub_test_metric")
				})
			}
			b.ReportMetric(200, "some_metric")
		})
	}
}

What did you see happen?

BenchmarkTest
BenchmarkTest/Test
BenchmarkTest/Test/Subtest
BenchmarkTest/Test/Subtest-8 	1000000000	         0.0000002 ns/op	       100.0 sub_test_metric
PASS

What did you expect to see?

I want to see some_metric be reported on the BenchmarkTest/Test benchmark.

[dr2chase]

CC @neild @adonovan

[adonovan]

@aclements Was this effect an unexpected consequence of ResetTimer's clearing of the extra mapping, or was it intended, perhaps on the groups that b.Run has all kinds of performance effects that render the outer benchmark invalid as anything other than a container for subbenchmarks?

Should the extra mapping be inherited by subbenchmarks?

[HaraldNordgren]

@adonovan Any updates on this? 🤗

[seankhliao]

When I looked at the benchmark code, I was under the impression that this was expected, since if a benchmark has subbenchmarks, then it itself is not a benchmark and thus reports no metrics at all.

Ref: https://go.googlesource.com/go/+/8397de2241ca71ba5df2fa60564a8559b9a2d25d/src/testing/benchmark.go#689

[HaraldNordgren]

@seankhliao It could be by design in the code. I got around it by not using sub-benchmarks at all.

But in that case, maybe it should be documented?

Or even better give an error when calling Run to create a sub-benchmark or ReportMetric on a sub-benchmark?

Currently it happens without any notice.

[seankhliao]

b.Run says:

A subbenchmark is like any other benchmark. A benchmark that calls Run at least once will not be measured itself and will be called once with N=1.