cmd/go: decide, document standard benchmark data format

We propose to make the current output of go test -bench the defined format for recording all Go benchmark data. Having a defined format allows benchmark measurement programs and benchmark analysis programs to interoperate while evolving independently.

See golang.org/design/14313-benchmark-format for details.

design: add 14313-benchmark-format.md

Why have the key start with a lowercase letter rather than an uppercase one? Wouldn't having it start with an uppercase letter make it easier to write go code that unmarshals the configuration (for example, not needing struct tags like in json)?

A line from the example output:

BenchmarkDecode/text:digits/level:speed/size:1e4/gomaxprocs:8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op

is it possible to change "100" to "100 iters" or "100 runs" so that every number also has a unit? Or will that be backwards incompatible?

@nigeltao

@nigeltao, unfortunately that would be backwards incompatible.

What's the reasoning for this restriction?

Tooling can expect that the unit strings are identical for all runs to be compared; for example, a result reporting “ns/op” need not be considered comparable to one reporting “µs/op.”

It doesn't seem like it makes the job of a parser author harder to just drop this restriction (and generation code can pick an appropriate unit per line, if desired).

@cespare

@cespare, in the general case dropping that restriction would make the job of the parser much harder, if not impossible. Since we're proposing that benchmarks can emit custom metrics, that would require the parser to know how to conform arbitrary scalings of arbitrary units. It would also require that we specify much more precisely the syntax of units. Currently we've specified the unit syntax enough to make "best effort" rescaling possible in benchmark processors, but even if a processor doesn't understand a unit, it can at least expect it to be spelled consistently so it can match up results.

@aclements

@aclements got it, thanks.

This proposal adds one new kind of line: configuration. What do you think about also standardizing a column header line format? Right now the format doesn't provide any mechanism for carrying the context around with it, so a line like this

BenchmarkDecodeDigitsSpeed1e4-8          100        154125 ns/op      64.88 MB/s       40418 B/op          7 allocs/op

only makes sense given the current benchmark conventions.

Now, given that lines other than configuration lines and benchmark result lines are ignored, it's possible for someone to make up their own format for column headers. But might it be better to settle on a single convention from the start?

(A slightly different convention would be to decide on a well-known configuration key (columns?) which specifies the column names.)

By the way, I commented on a few typos in the CL (https://go-review.googlesource.com/#/c/19490/) after it was already submitted.

I wonder whether the specification should call out that the file is UTF-8 encoded and that lines are separated by line feeds.

It may also be helpful to specify whether or not a tool can assume that value units are always ratios, so must include a slash character. There is also the question whether a value unit with multiple slashes is undefined, forbidden or something else.

I suggest to require all lines beginning with a # character to be ignored.

Currently

#foo: bar

is a valid configuration line. It cannot be ignored because of the word other in

All other lines in the data file, including but not limited to blank lines and lines beginning with a # character, are ignored.

LGTM overall. Seems very useful.

Some nits and questions:

space characters separate “key:” from “value.”

0 or more space characters, or 1 or more space characters?

Value can be omitted entirely but the colon must still be present.

Presumably if the answer to the above is "1 or more", then the spaces may also be omitted in this case.

The first field is the benchmark name, which must begin with Benchmark and is typically followed by a capital letter, as in BenchmarkReverseString. Tools displaying benchmark data conventionally omit the Benchmark prefix.

It'd be better to require a following capital letter. If the name is just Benchmark and the tool omits the prefix, the name will be blank, which is likely to confuse the user or look like a bug. The Go tool already requires this of benchmark function names.

The same benchmark name can appear on multiple result lines, indicating that the benchmark was run multiple times.

This is a mismatch with running benchmarks over multiple packages right now. Using tip:

$ go test -bench=. encoding/base32 encoding/base64
PASS
BenchmarkEncodeToString-8      30000         42033 ns/op     194.89 MB/s
BenchmarkDecodeString-8        10000        182128 ns/op      71.99 MB/s
ok      encoding/base32 3.593s
PASS
BenchmarkEncodeToString-8     100000         22681 ns/op     361.17 MB/s
BenchmarkDecodeString-8        20000         99789 ns/op     109.47 MB/s
ok      encoding/base64 5.487s

According to the proposal, this would be interpreted incorrectly as two benchmarks run twice, rather than four benchmarks run once.

If this proposal were adopted, however, we could add to the output a config line to disambiguate, perhaps like:

$ go test -bench=. encoding/base32 encoding/base64
PASS
package: encoding/base32
BenchmarkEncodeToString-8      30000         42033 ns/op     194.89 MB/s
BenchmarkDecodeString-8        10000        182128 ns/op      71.99 MB/s
ok      encoding/base32 3.593s
PASS
package: encoding/base64
BenchmarkEncodeToString-8     100000         22681 ns/op     361.17 MB/s
BenchmarkDecodeString-8        20000         99789 ns/op     109.47 MB/s
ok      encoding/base64 5.487s

The main known issue with the current go test -bench is that we'd like to emit finer-grained detail about runs, for linearity testing and more robust statistics. This proposal allows that by simply printing more result lines.

Link to issue 10669.

We anticipate that the new x/perf subrepo will include a library for loading benchmark data from files

Will https://godoc.org/golang.org/x/tools/benchmark/parse be updated? Deprecated? Replaced?

@ulikunitz

Thanks for the comments.

@ulikunitz:

I wonder whether the specification should call out that the file is UTF-8 encoded and that lines are separated by line feeds.

Sure.

It may also be helpful to specify whether or not a tool can assume that value units are always ratios, so must include a slash character. There is also the question whether a value unit with multiple slashes is undefined, forbidden or something else.

It's a fascinating question, but one that maybe we should leave open, at least at first. I can't think of any units that wouldn't have exactly one slash, but maybe (probably) I am just not thinking hard enough.

I suggest to require all lines beginning with a # character to be ignored.

Nice catch. Thank you.

Is there anything left to be done for this proposal? (Does the documentation need to be moved to some more prominent or searchable location?)

Hi,

I didn't find one, so I wrote an API for encoding/decoding/streaming the benchmark output as defined by the spec readme at https://github.com/cep21/benchparse. I'm happy to contribute it if desired.

@rsc

Ping @rsc @aclements .

We are unaware of any standard formats for recording raw benchmark data, and we've been unable to find any using web searches. One might expect that a standard benchmark suite such as SPEC CPU2006 would have defined a format for raw results, but that appears not to be the case.

Looks like you want to reinvent a wheel. I know about at least these formats: JMeter, Skia, Chrome, BenchmarkAnything, Google Benchmark, dstat.

P.S. I have overlooked that it has been already merged.

So… What’s left in this issue?

I think we can close this.
The testing docs link to https://go.googlesource.com/proposal/+/master/design/14313-benchmark-format.md

Navigation Menu

Repository navigation

cmd/go: decide, document standard benchmark data format #14313

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cmd/go: decide, document standard benchmark data format #14313

Description

Activity

btracey commented on Feb 12, 2016

nigeltao commented on Feb 12, 2016

aclements commented on Feb 12, 2016

cespare commented on Feb 12, 2016

aclements commented on Feb 12, 2016

cespare commented on Feb 13, 2016

cespare commented on Feb 13, 2016

cespare commented on Feb 13, 2016

ulikunitz commented on Feb 13, 2016

ulikunitz commented on Feb 13, 2016

josharian commented on Feb 15, 2016

rsc commented on Feb 16, 2016

25 remaining items

bcmills commented on Jan 23, 2019

cep21 commented on Sep 8, 2019

mdempsky commented on Oct 3, 2019

ligurio commented on Oct 23, 2019

AlekSi commented on Sep 28, 2024

seankhliao commented on Mar 22, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions