proposal: testing: custom benchmark labels

[mmcloughlin]

Currently go benchmark output reports some labels:

$ go version
go version go1.11.1 linux/amd64
$ go test -run NONE -bench . crypto/aes
goos: linux
goarch: amd64
pkg: crypto/aes
BenchmarkEncrypt-8   	100000000	        10.5 ns/op	1516.95 MB/s
BenchmarkDecrypt-8   	100000000	        10.2 ns/op	1566.61 MB/s
BenchmarkExpand-8    	20000000	        57.3 ns/op
PASS
ok  	crypto/aes	3.310s

These are added by

go/src/testing/benchmark.go

Lines 277 to 283 in dd78955

	labelsOnce.Do(func() {
	fmt.Fprintf(b.w, "goos: %s\n", runtime.GOOS)
	fmt.Fprintf(b.w, "goarch: %s\n", runtime.GOARCH)
	if b.importPath != "" {
	fmt.Fprintf(b.w, "pkg: %s\n", b.importPath)
	}
	})

It would be great if the user could add custom labels relevant to the performance of their code. For example:

• CPU type/Microarchitecture/CPUID bits
• Software version (perhaps git tag)
• Presumably many others

I have noticed that the perf.golang.org service appears to support this, but I haven't looked into exactly how. Apologies if I've overlooked some existing feature.

[josharian]

Dup of #26037? I’d love to see that go in for 1.12 but I don’t think either @aclements or I have the bandwidth to push it through.

[josharian]

Oh, reading closer, not a dup. Sorry. At least I’ve now cc’d Austin. :)

[mmcloughlin]

Seemed this should have the proposal: prefix. Regarding bandwidth, I would be happy to put together a CL if the proposal is accepted.

[ianlancetaylor]

What exactly are you proposing? Adding an option to go test which prints labels? It seems that you could print those yourself just as easily.

[mmcloughlin]

An option to go test seems less useful to me. This feature is more important when it comes to labels that require code to derive (such as hardware parameters).

Proposal

My concrete proposal is an additional function in the testing package.
Perhaps something like:

// SetBenchmarkLabel records a value relevant to benchmark performance. This
// will be included in benchmark output ahead of individual benchmark results.
// Labels "goos", "goarch" and "pkg" are set by default.
SetBenchmarkLabel(key, value string)

Not sure about naming, but hopefully this gives the idea.

"Case Study"

I recently implmented Meow hash for Golang. This hash is designed to take advantage of hardware AES instructions, therefore the implementation dispatches to one of three backends based on CPUID flags at runtime (pure Go, AES-NI and VAES-256/512).

To provide useful context in the test/benchmark output, I have two test functions that exist
purely to dump internal parameters:

https://github.com/mmcloughlin/meow/blob/7358a9c1d22772fbddc8c47fd74cf501c0136ec1/dispatch_amd64_test.go#L10-L17
https://github.com/mmcloughlin/meow/blob/7358a9c1d22772fbddc8c47fd74cf501c0136ec1/meow_test.go#L23-L25

I would prefer to include these as "official" labels:

testing.SetBenchmarkLabel("backend", implementation)
testing.SetBenchmarkLabel("hasaesni", cpu.HasAES)
testing.SetBenchmarkLabel("hasavx", cpu.HasAVX)
...

Comparison: Google Benchmark

Note that Google benchmark also provides context with benchmark runs. For
example:

$ ./benchmark
2018-10-31 12:31:20
Running ./benchmark
Run on (4 X 3500 MHz CPU s)
CPU Caches:
  L1 Data 32K (x2)
  L1 Instruction 32K (x2)
  L2 Unified 262K (x2)
  L3 Unified 4194K (x1)
--------------------------------------------------------------------
Benchmark                             Time           CPU Iterations
--------------------------------------------------------------------
meow_hash_benchmark/1                14 ns         14 ns   45173239   66.5225MB/s
meow_hash_benchmark/2                15 ns         15 ns   46804897   125.632MB/s
meow_hash_benchmark/4                16 ns         16 ns   44735009   240.535MB/s
...

It would be great it was also possible to add CPU/cache information in Go
benchmark output.

Benchmark Format

Note that this output has been formalized in the following document:

https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md#configuration-lines

In particular these "labels" are called "Configuration Lines".

Moreover, this document makes go bench output the official format for
representing benchmark results. While it is always possible to output this
kind of metadata in a separate program, that makes it harder to integrate with
automated tooling for running and storing benchmarks.

[gopherbot]

Change https://golang.org/cl/146897 mentions this issue: testing: add SetBenchmarkLabel

[mmcloughlin]

Added a CL to concretely demonstrate the proposal (and get something on the record before the freeze).

[seebs]

I'd use it if it existed, which I like to think is an argument in favor.

[rsc]

Does SetBenchmarkLabel keep track of what's already been printed and not repeat prints? In theory you can change the labels between runs.

[mmcloughlin]

The existing implementation (and prototype CL) print the labels inside a sync.Once. Therefore any labels added or modified after the first print would be ignored. My expectation was that this would be used in init or other setup functions. This behavior seems reasonable to me but should be documented for clarity, perhaps:

// SetBenchmarkLabel records a value relevant to benchmark performance. This
// will be included in benchmark output ahead of individual benchmark results.
// Any SetBenchmarkLabel calls after the first benchmark is executed will be
// ignored.  Labels "goos", "goarch" and "pkg" are set by default.
SetBenchmarkLabel(key, value string)

Did you have some other behavior in mind?

[rsc]

SetBenchmarkLabel should track what it has printed (a map[string]string) and silence duplicate prints but not throw away prints that change the value associated with a given key.

Otherwise this seems OK to me. @aclements?

[rsc]

Ping @aclements