`state.PauseTiming()` and state.ResumeTiming() take a long time

[izaid]

I'm trying to benchmark a custom map type. I basically want to see how long it takes to look up random keys.

To do so, I'm generating random keys in the state.KeepRunning() loop. I obviously don't want the random key generation to be added to the overall time of the benchmark. To prevent this, I tried to use state.PauseTiming() and state.ResumeTiming(), but I found using those functions added 100s of nanoseconds to my benchmark -- which is much more than the map lookup itself.

Is this a problem with these functions? Or should I be generating my keys a different way?

[dmah42]

This might be a good use for fixtures. You can generate your keys in SetUp and store them in the fixture object as state.

see test/fixture_test.cc for an example.

[izaid]

Thanks for the suggestion! I switched to fixtures here.

I guess I need to also modify the normal state.KeepRunning() loop to terminate when I run out of keys. I can add a simple check to see if there are no more keys, but what's the right way to signal to state to stop the benchmark?

[dmah42]

does your benchmark do one lookup per 'keep running' iteration? do you want to look up every key?

one thing that comes to mind is that if we passed the state to 'SetUp' you could generate a number of keys based on state::range_x() and then loop through them in the keeprunning loop (like BM_SetInsert in the benchmark_test.cc file). Then you just need to set the number of processed items to state::iterations() * state::range_x().

Alternatively, you could pick a random key from your set of keys every time through the loop, but you might get more variation between benchmark runs.

[izaid]

I was doing one lookup per 'keep running' iteration. I really just want to benchmark the lookup cost, and this seems like the right way to do it. But, if it's not, let me know.

Your example with the SetUp seems reasonable to me. That's what I'm attempting to do now -- e.g., N lookups per 'keep running' iteration. I've got this working as per the set example. I guess it's slightly annoying that I have to divide the benchmark time by the number of lookups to get the actual cost per lookup, but I can live with that. Is it possible to get access to state::range_x() in the benchmark fixture? (e.g., how should I tell my fixture the size of the random set it should generate?)

[pleroy]

I seem to remember that state.PauseTiming() and state.ResumeTiming() are expensive because they call CHECK and CHECK is expensive. It does:

# define CHECK(b) ::benchmark::internal::GetNullLogInstance()

But it should really do:

# define CHECK(b) while (false) ::benchmark::internal::GetNullLogInstance()

[izaid]

Even if I comment out all the CHECK(...) I can find, it still seems PauseTiming() and ResumeTiming() takes in the 100s of nanoseconds.

In my particular situation, using the fixture (or otherwise precomputing the keys) avoids that problem. Whether this is a problem in general, I don't know.

[dmah42]

I did some testing (see #181) which required passing a const version of the state to SetUp. I'm ok with this change, by the way.

It's clear from the results that there is a significant impact for small maps. So much so that I'm thinking of deprecating PauseTiming and ResumeTiming now we have the fixtures. I think they cover every use-case for pausing the timer. At least the ones I expected.

Is my test similar to what you had in mind?

[dmah42]

@pleroy I think the cost is likely in the mutex and barrier to handle multiple threads pausing/resuming. Though yes, the CHECK doesn't help.

[izaid]

@dominichamon Thanks, your map test is exactly what I have in mind. Would love to see that merged.

As you also found out, PauseTiming and ResumeTiming basically drown out the map lookup. The fixture is a totally acceptable solution here, so I'd agree we should probably deprecate PauseTiming and ResumeTiming -- I can't imagine they will ever be fast enough to be negligible.

Here's another idea that may be worth considering: What about if we allow the user to specify the number of iterations in state. That way, in something like BM_MapLookup, we will have N iterations that correspond to N lookups from the random key set prepared in the fixture. The advantage of this is that the time reported will be exactly the map lookup time, not the total time for N map lookups.

[dmah42]

Merged. I think we'll remove PauseTiming and RemoveTiming.

Regarding the iterations -- there's an old version that allowed number of iterations to be specified but trying to create an API that supports time-based and iteration-based constraints is complicated. Given they're benchmarks, in general we're interested in doing as many iterations as possible to get good statistics. As such, setting time-based constraints is a more reasonable API.

I understand your specific requirement, but I hope that you can get the results you need with the existing set up.

[EricWF]

I haven't looked into this in depth, but you can do the initialization you need before entering the "keepRunning" loop.

IMO If you need to pause a benchmark in the middle you should try to redesign your benchmark.

[didenko]

(reposting under the proper account, sorry for the confusion)

Given that the #183 PR was dropped, any ideas on how to deal with the Pause/Resume issues? I think there are couple more illustrations of the issue at the discussion group: https://groups.google.com/forum/#!topic/benchmark-discuss/o5BvqFZIs3Y