Explanation of the benchmark results

[zack-snyder]

Is there an explanation of the standard benchmark table results?

Time: ?
CPU: ?
Iterations: ?

What is this exactly, and didn't find a clear explanation of this in the documentation.
2)
I put a sleep(100) into a while loop with some code.
For unknown reason we got 0ns at the CPU column.
Although there were some code which does some calculation.

 while (state.KeepRunning())
 {
     sleep(100);
     benchmark::DoNotOptimize(some_func());
 }

How big is the time out of all iterations?

static int result = 0;
 while (state.KeepRunning())
 {
     sleep(100);
     std::cout << ++result << std::endl;
 }

He printed till 111 and said iterations: 100
Why?

[EricWF]

It printed till 111 and said iterations: 100. Why?

Because the benchmark is tested a couple of times with different iteration counts to determine the correct count to use. After it finds an iteration count that causes the benchmark to run for a significant enough period of time it reports only that run, and not the test runs before it.

Resetting result to 0 at the start of the benchmark will show you this.

Is there an explanation of the standard benchmark table results?

Time: The average wall time per iteration.

CPU: The average CPU time per iteration. By default this clock is used when determining the amount of iterations to run. When you sleep your process it is no longer accumulating CPU time. I'm assuming some_func() is almost trivial, and that the compiler optimized almost all of it away. (DoNotOptimize is not magical and needs to be used very carefully. See https://github.com/google/benchmark#preventing-optimisation)

Iterations: The number of iterations the benchmark ran. (See https://github.com/google/benchmark#controlling-number-of-iterations)

Perhaps somebody should add basic documentation about the output information.

[zack-snyder]

I do some initialization before the loop.
Will this also be included in the measurement?
Something like this:

static void BM_somefunc(benchmark::State& state)
{
    auto foo = do_some_init();
    while (state.KeepRunning())
    {
        do_some_calculation(foo);
    }
}

I get sometimes following results

---------------------------------------------------------
Benchmark                  Time           CPU Iterations
---------------------------------------------------------
BM_fastFuncReturn      16327 ns      16392 ns      44800
BM_slowFuncReturn      17499 ns      16881 ns      40727

You see that the CPU time is in one case higher. How can this be?

[dmah42]

1. no it won't. and if you want to do some reinitialization in the loop you can use PauseTiming and ResumeTiming. 2. because slowFuncReturn is slower? Dominic Hamon | Google *There are no bad ideas; only good ideas that go horribly wrong.*

…

On Wed, Jun 7, 2017 at 6:09 AM, zack-snyder ***@***.***> wrote: 1. I do some initialization before the loop. Will this also be included in the measurement? Something like this: static void BM_somefunc(benchmark::State& state) { auto foo = do_some_init(); while (state.KeepRunning()) { do_some_calculation(foo); } } 1. I get sometimes following results --------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------- BM_fastFuncReturn 16327 ns 16392 ns 44800 BM_slowFuncReturn 17499 ns 16881 ns 40727 You see that the CPU time is in one case higher. How can this be? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#397 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfIMmr0LoZi4vo2Bw93pmRdoXesTgiIks5sBqEjgaJpZM4NuTrF> .

[zack-snyder]

How can the CPU timer higher, then the wall clock time?
How is the CPU time measured, with ticks?

[werdna87]

I believe there might be a few ways. The most obvious is if your test uses multiple threads

[zack-snyder]

no, there are no multiple threads involved

[mattreecebentley]

I feel the basic documentation is incredibly sparse and not very detailed about how any of the functions and macros work, their methodology or reasons why they are doing what they're doing. Coming to it afresh having not studied it before, I find it almost impenetrable. I would recommend a full review of the documentation as it stands.

[HaoQChen]

It printed till 111 and said iterations: 100. Why?

Because the benchmark is tested a couple of times with different iteration counts to determine the correct count to use. After it finds an iteration count that causes the benchmark to run for a significant enough period of time it reports only that run, and not the test runs before it.

Resetting result to 0 at the start of the benchmark will show you this.

Is there an explanation of the standard benchmark table results?

Time: The average wall time per iteration.

CPU: The average CPU time per iteration. By default this clock is used when determining the amount of iterations to run. When you sleep your process it is no longer accumulating CPU time. I'm assuming some_func() is almost trivial, and that the compiler optimized almost all of it away. (DoNotOptimize is not magical and needs to be used very carefully. See https://github.com/google/benchmark#preventing-optimisation)

Iterations: The number of iterations the benchmark ran. (See https://github.com/google/benchmark#controlling-number-of-iterations)

Perhaps somebody should add basic documentation about the output information.

are you sure time is The average wall time per iteration.?

[HaoQChen]

In linux, it seems to use clock_gettime function to get thread time and process time

see src/timers.cc line 130 and 174

[LebedevRI]

Regarding documentation, it's a problem or knowledge.
Those who know the answers, know the answers, and don't airlessness think
that some particular thing is missing from the docs/is hard to understand.
Which can of course not be so for new users.

So i think it would be useful if someone could actually comply the list of topics
that seem to be not widely represented in docs, and then someone who can,
would document them.

[GACLove]

https://pythonspeed.com/articles/blocking-cpu-or-io/ time and cpu time. May be help you

[dmah42]

closing this out as a specific issue, but i remain open to anyone who wants to enhance the documentation further.