proposal: cmd/go: -benchformat option in go test

[nodirt]

Background

Currently go test -bench . prints out benchmark results in a certain format, but there is no guarantee that this format will not change. Thus a tool that parses go test output may break if an incompatible change to the output format is made.

Also currently there is no good way to distinguish benchmark results from the rest of go test output, other than checking that a line starts with "Benchmark".

Proposal

1. Add -benchformat flag to go test. Its value is a text/template template with BenchmarkResult input, similar to -f flag in go list.
2. Add Proc to BenchmarkResult. It will have the value of runtime.GOMAXPROCS at the beginning of B.launch call.

Example:

$ go test -run=^$  -bench . \
  -benchformat "BENCHMARK STARTS HERE:{{.Name}}:{{.Proc}}:{{.T}}:{{.N}}:{{.Bytes}}"
PASS
BENCHMARK STARTS HERE:BenchmarkFoo:8:1s:10000:0
BENCHMARK STARTS HERE:BenchmarkBar:8:1s:10000:0

[nodirt]

One may be concerned that adding -benchformat for benchmarks without adding -format for tests would be inconsistent. We can add -format for tests too.

Currently there is no struct for a test result template. It would need to be added. I don't see a strong reason to add testing.TestResult to that API, though, as we can just document the test result struct in go test -help.

[bradfitz]

Considering that go list accepts a text/template format, this seems consistent. I think it's probably worth exploring and writing up a more formal proposal, unless somebody disagrees.

/cc @adg

[minux]

See also #2981. I think having a -json flag is enough to solve this problem once and for all.

[nodirt]

Thanks for mentioning #2981.

Assuming implementation would use encoding/json package without manual concatenation of json chunks, one advantage of -format over -json is that go test can stream results to stdout, rather than waiting for completion of all tests/benchmarks and then dump a big json blob. This is important for tools that present test results to the user. In my case, I need to process test output and print it out (annotated) as soon as a benchmark completes. Without streaming UX would be worse.

Even if we concatenate json chunks, parsing them with encoding/json would not be convenient either.

Also from reading #2981 it seems that adding -json is hard, whereas adding -format is trivial, more consistent with go list and provides same, if not more, functionality.

[minux]

With -json it can still stream the result out as a single json object for each test/benchmark. Actually, I think that's the only way to go, because we don't want to lose progress even if the test binary panics in midway. Even if adding -json is hard, if we intended to add -json eventually, I don't think we should add more redundant mechanisms now. This is different from the go list case because go list are normally used by shell scripts so having a -f flag in addition to -json makes sense.

[nodirt]

Let's continue the discussion about -json in #2981.

[rsc]

Currently go test -bench . prints out benchmark results in a certain format, but there is no guarantee that this format will not change.

OK. Counter-proposal: let's guarantee that this format will not change.

Specifically, I propose that the output format is exactly a sequence of lines containing space-separated fields:

<name> <iterations> (<value> <unit>)+

The number of on a particular line is not specified, nor is the space of possible s.

This describes all the current output and blesses (instead of forcing rewrites of) tools that parse the output already, like benchcmp and benchstat.

/cc @aclements

[aclements]

I don't have a strong opinion about this, but I'm inclined to think it's unnecessary as long as we simply bless the current format as suggested by @rsc and especially unnecessary if we add -json (and if I had to choose between the two, I would definitely choose -json).

Suppose we were to simply bless the current format. What future changes to go test -bench might that cause friction with? I have a few things on my list. I'd like an external tool (e.g., benchstat) to be able to do linearity testing. We could support that by outputting lines for each run-up step. I'd like go test -bench to be able to output many more instances of shorter runs so external tools can compute more robust statistics. Again, go test -bench can just output more instances of the same benchmark line. I've thought about adding custom outputs (# of GCs, max pause, etc). We could support that by adding more (value unit) pairs.

[rsc]

Formal counter-proposal: #14313.

[adg]

Either #14313 or #2981 will go ahead in favor of this proposal. Marking as declined. Thanks for your time.