Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing (*B).Lap(string name, b *testing.B) for sub-benchmarks of a single process #33194

Open
seebs opened this issue Jul 19, 2019 · 0 comments

Comments

@seebs
Copy link
Contributor

commented Jul 19, 2019

What version of Go are you using (go version)?

N/A (but 1.12.x)

What did you do?

I'm benchmarking some stuff. I have a longish and complicated process with multiple parts which are hard to separate or run independently. (Approximately in the vicinity of setup/teardown code, where the setup has to be matched by paired teardowns). Was looking at benchmarking options.

Thought of a thing that I'm not sure is all that useful, but the more I think about it, the more I think it would have desireable consequences.

What did you expect to see?

expect is a strong term.

What I'd like is a way to handle sub-benchmarks where it's not possible to just run each one 1..N times separately, but where the entire procedure has to be run 1..N times, but I'd like to track different parts of it.

What did you see instead?

This is sort of possible to emulate-ish using StartTimer/StopTimer, but there's a performance problem here; if one of the sections takes significantly less time than the others, running the entire procedure enough times that that section takes over a second to run takes a very long time, and is highly redundant. This is the sort of thing for which stopwatches have lap timers -- sub timers that can be captured while the main timer continues, and then reviewed separately.

So the idea I have is something like:

func BenchFoo(b *testing.B) {
	for i := 0; i < b.N; i++ {
		b.Lap("foo_setup", func(b *testing.B) {
			// setup code
		})
		b.Lap("foo_process", func(b *testing.B) {
			// functional code
		})
		b.Lap("foo_teardown", func(b *testing.B) {
			// cleanup code
		}
	}
}

And now we get to the kicker: The reason I think this is actually a sort of good idea.

Right now, the above would probably be written as:

func BenchFoo(b *testing.B) {
	b.StopTimer()
	for i := 0; i < b.N; i++ {
		// setup code
		b.StartTimer()
		// functional code
		b.StopTimer()
		// cleanup code
	}
}

Often, when I've had problems figuring out why benchmarks are taking so long to run even though the performance reported is good, the reason is issues in the performance of the setup/cleanup code.

If this feature existed, it would be close to free to report on the performance of those sections too -- and in several cases, if I'd been tracking that performance, I'd have noticed a serious performance regression in something that I wasn't consciously thinking about, but which was impacted by a change I was making.

Basically, I want to encourage people to collect the data, since for some of the benchmarks I do, the setup or cleanup code can easily be 80% of the runtime of a benchmark, and if I'm gonna be spending the CPU cycles to run it, it'd be nice to be able to track it without preventing me from getting good data on the performance of the inner bit.

(I haven't thought very carefully about the API, but I think it probably does want to get an inner b, because it might want to record custom stats, etc.)

@gopherbot gopherbot added this to the Proposal milestone Jul 19, 2019

@gopherbot gopherbot added the Proposal label Jul 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.