Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: add fields to BenchmarkResult to report errors, skips, and logs #61050

Open
bcmills opened this issue Jun 28, 2023 · 0 comments
Labels
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 28, 2023

In reviewing the testing package, I noticed that it currently contains some tests that call testing.Benchmark, where the benchmark function calls Error, Fatal, or SkipNow.
(https://cs.opensource.google/go/go/+/master:src/testing/benchmark_test.go;l=94-127;drc=cecaa25bd31622f5ae0df7fd5c8c7f02afde8715)

Those tests do not currently emit any output when run, and as far as I can tell the testing.BenchmarkResult type does not provide a way for them to do so. As a result, I believe that failures in calls to testing.Benchmark in general tend to go undetected.

~/go/src$ go1.21rc2 test testing -run='TestRunParallel.*' -v -count=1
=== RUN   TestRunParallel
--- PASS: TestRunParallel (0.16s)
=== RUN   TestRunParallelFail
--- PASS: TestRunParallelFail (0.00s)
=== RUN   TestRunParallelFatal
--- PASS: TestRunParallelFatal (0.00s)
=== RUN   TestRunParallelSkipNow
--- PASS: TestRunParallelSkipNow (0.00s)
PASS
ok      testing 0.178s

To allow users of testing.Benchmark to write more robust benchmarks, I propose the addition of the following fields to the BenchmarkResult struct:

type BenchmarkResult struct {
	…
	Log []byte  // The output produced by calls to b.Log, b.Error, b.Skip, and similar during the benchmark run.
	Failed bool  // The value returned by b.Failed at the end of the run.
	Skipped bool // The value returned by b.Skipped at the end of the run.
}

Since these are new fields, I believe this change is strictly backward-compatible.

@gopherbot gopherbot added this to the Proposal milestone Jun 28, 2023
@bcmills bcmills changed the title proposal: testing: proposal: testing: add fields to BenchmarkResult to report errors, skips, and logs Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

2 participants