Repository navigation
Description
Propose to add an Elapsed method to testing.B, as follows:
// Elapsed returns the measured elapsed time of the benchmark. If the timer is
// running, it will be stopped.
func (b *B) Elapsed() time.Duration {
b.StopTimer()
return b.duration
}Motivation
Custom benchmark metrics are a powerful feature. Go introduced partial support with the testing.B.ReportMetric() API following #26037. However, this API seems to leave a gap in supporting rate metrics. Specifically, rate metrics are those that would be divided by the total benchmark time and reported with the /s suffix, such as the built-in MB/s. These cannot be reported using the existing API since there is no way to access the measured time.
Rate metrics have many use cases. The existence of MB/s proves at least one. You could also imagine reporting packets/s for a network benchmark, points/s for a geometry benchmark. I was just working on a benchmark for the avo assembler and wanted to report instructions/s.
Rate metrics are supported by other benchmark libraries, such as Google Benchmark, which has become somewhat of a de-facto standard for C++ benchmarking. Google Benchmark's user counters are configurable to support rate metrics with the kIsRate flag, along with many others. Google Benchmark also supports the special-case SetItemsProcessed to support examples like packets/s or points/s mentioned previously. As an example, the abseil library uses SetItemsProcessed to report the rate items are pushed to a vector.
I don't think there are satisfactory workarounds for this at the moment. You can measure the time yourself and use it to report a rate metric with ReportMetric. Alternatively, you can abuse SetBytes and post-process the output or inform your users to interpret it differently.
Propose extending the testing.B API such that this is possible. Ideas:
- Add a
ReportRatemethod, similar toReportMetricbut the value will be divided by time. - Add a
SetItemsProcessedmethod similar to Google Benchmark. - Expose the measured time so users can compute the rate themselves and report it with the existing API.
Following discussion below, we propose the Elapsed() method given above.
Activity
ianlancetaylor commentedon Jan 11, 2021
CC @aclements
josharian commentedon Jan 11, 2021
This isn't too onerous a workaround; measuring elapsed time is both short and simple.
An alternative fix here is to add methods on
testing.Bthat let you query the accumulated metrics, e.g.func (b *B) Elapsed() time.Duration). Sample usage:mmcloughlin commentedon Jan 11, 2021
Sure, not onerous in terms of lines of code. However, it seems to me that timing code execution is a core responsibility of a benchmark framework, so it seems odd to cede that responsibility to your users.
I haven't even looked at the code under the hood, but you could imagine that the
testingpackage wanted to use a higher precision timer than thetimepackages uses by default. It's nice that this is hidden from users and can be changed in the future if necessary.Yes, this would be great. This is what I had in mind with the "Expose the measured time so users can compute the rate themselves and report it with the existing API" idea in the description. It does seem slightly awkward that you'd need some contract about calling
StopTimerfirst.aclements commentedon Jan 11, 2021
I wouldn't be opposed to exposing the measured elapsed time of the benchmark. While it's true that often this isn't hard to measure for yourself, it becomes much harder in benchmarks that use
StopTimerandStartTimer, and having a way to retrieve the framework's own measured time would ensure that any custom time-rate measurements had exactly the same denominator as the built-in ns/op measurement. Having a consistent denominator does depend on callingStopTimerbefore retrieving the elapsed time, as Josh did.mmcloughlin commentedon Jan 12, 2021
Peeking at the internals, this could be as simple as adding a
Duration() time.Durationmethod that returns theb.durationfield. The nuance is just what it should do if it's called when the timer is running. Should it:timerOnis truepaniciftimerOnis trueaclements commentedon Jan 12, 2021
Returning 0 seems like it would invite bugs, but I'm not sure between the other two options.
josharian commentedon Jan 12, 2021
A fourth option: stop the timer. Easy to document, yields the correct outcome in most situations with a minimum of fuss.
I would advocate for
ElapsedoverDuration, because duration could also mean the target duration of the benchmark run.mmcloughlin commentedon Apr 5, 2021
How do we progress this proposal through the process to get a decision? Based on the discussion I think the simplest concrete proposal is the addition of the following method to
*testing.B:Perhaps an additional note in the documentation for
ReportMetricwould be in order?[-]proposal: testing: support benchmark rate metrics[/-][+]proposal: testing: add Elapsed() method to testing.B[/+]rsc commentedon Jun 22, 2022
Leaving the timer running seems much less surprising than silently stopping the timer.
Otherwise this seems fine to me.
rsc commentedon Jun 22, 2022
This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group
mmcloughlin commentedon Jun 22, 2022
Great to see this taken up!
Yes, I can see the argument for that.
rsc commentedon Jun 29, 2022
Assuming Elapsed reports elapsed time but does not change the running state of the timer, does anyone object to adding Elapsed?
3 remaining items
[-]proposal: testing: add Elapsed() method to testing.B[/-][+]testing: add Elapsed() method to testing.B[/+]gopherbot commentedon Jul 30, 2022
Change https://go.dev/cl/420254 mentions this issue:
testing: add Elapsed() method to testing.Btesting: add Elapsed method to testing.B
gopherbot commentedon Nov 9, 2022
Change https://go.dev/cl/449077 mentions this issue:
doc/go1.20: add a release note for (*testing.B).Elapseddoc/go1.20: add a release note for (*testing.B).Elapsed
testing: add Elapsed method to testing.B