x/perf/benchstat: geomean includes benchmarks that are in only one file

[josharian]

$ cat b
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkB 10 10000 ns/op
BenchmarkB 10 10000 ns/op
BenchmarkB 10 10000 ns/op
BenchmarkB 10 10000 ns/op
BenchmarkB 10 10000 ns/op
BenchmarkB 10 10000 ns/op
$ cat a
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
BenchmarkA 10 100 ns/op
$ benchstat -geomean b a
name        old time/op  new time/op  delta
A            100ns ± 0%   100ns ± 0%     ~     (all equal)
[Geo mean]  1.00µs       0.10µs       -90.00%

The geomean should be 0.

cc @aclements @rsc @quentinmit

[aclements]

FWIW, this just confused another person: https://groups.google.com/d/topic/golang-dev/nHlZ5scH5nY/discussion

[aclements]

I'm not positive what the "right" behavior here is, though the current behavior is clearly not it. :)

Should we show benchmarks that don't match or not? It's kind of annoying that we silently drop that information, and it makes benchstat somewhat less predictable. That suggests we should show such benchmarks in the appropriate column. But that makes the geomeans definitely less useful, and makes the geomean delta completely meaningless.

Perhaps we should show non-matching benchmarks in their own group after the geomean row, probably separated by a blank line?

[josharian]

This just bit another person (private correspondence).

Your solution seems fine, Austin. (I assume that you would not include the non-matching benchmarks in the geomean.)

[gopherbot]

Change https://golang.org/cl/309969 mentions this issue: cmd/benchstat: new version of benchstat