Skip to content

proposal: testing: add fields to BenchmarkResult to report errors, skips, and logs #61050

@bcmills

Description

@bcmills

In reviewing the testing package, I noticed that it currently contains some tests that call testing.Benchmark, where the benchmark function calls Error, Fatal, or SkipNow.
(https://cs.opensource.google/go/go/+/master:src/testing/benchmark_test.go;l=94-127;drc=cecaa25bd31622f5ae0df7fd5c8c7f02afde8715)

Those tests do not currently emit any output when run, and as far as I can tell the testing.BenchmarkResult type does not provide a way for them to do so. As a result, I believe that failures in calls to testing.Benchmark in general tend to go undetected.

~/go/src$ go1.21rc2 test testing -run='TestRunParallel.*' -v -count=1
=== RUN   TestRunParallel
--- PASS: TestRunParallel (0.16s)
=== RUN   TestRunParallelFail
--- PASS: TestRunParallelFail (0.00s)
=== RUN   TestRunParallelFatal
--- PASS: TestRunParallelFatal (0.00s)
=== RUN   TestRunParallelSkipNow
--- PASS: TestRunParallelSkipNow (0.00s)
PASS
ok      testing 0.178s

To allow users of testing.Benchmark to write more robust benchmarks, I propose the addition of the following fields to the BenchmarkResult struct:

type BenchmarkResult struct {
	…
	Log []byte  // The output produced by calls to b.Log, b.Error, b.Skip, and similar during the benchmark run.
	Failed bool  // The value returned by b.Failed at the end of the run.
	Skipped bool // The value returned by b.Skipped at the end of the run.
}

Since these are new fields, I believe this change is strictly backward-compatible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Incoming

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions