Simple benchmarking framework

[gavinandresen]

Benchmarking framework, loosely based on google's micro-benchmarking
library (https://github.com/google/benchmark)

Wny not use the Google Benchmark framework? Because adding Even More Dependencies
isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate
timings of threaded code then switching to the full-blown Google Benchmark library
should be considered.

The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout. It is left as an
exercise for later (or maybe never) to add command-line arguments to specify which
benchmark(s) to run, how long to run them for, how to format results, etc etc etc.
Again, see the Google Benchmark framework for where that might end up.

See src/bench/Examples.cpp for a sanity-test benchmark that just benchmarks
'sleep 100 milliseconds' and a benchmark for math.h's sin() function.

To compile and run benchmarks:

  configure --enable-bench  ...other configure options
  cd src; make bench

Sample output:

Benchmark,count,min,max,average
Sleep100ms,10,0.10048,0.104789,0.103251
Trig,37748735,0,4.76837e-07,2.71684e-08

[laanwj]

This works well as long as 'stuff you want to time' does not run too quickly - otherwise the state.KeepRunning(), which does a gettimeofday syscall, with the implied latency of that, will dominate.
(an alternative to calling the time function in the benchmark loop that is commonly used, is to first time a fixed # of runs of the loop, then from the result compute a number of iterations so that it runs for approximately the allotted time)

[laanwj]

Concept ACK

[jonasschnelli]

rm $(CURL_LIBS)? This seems to have be sneaked in from XT.

[jonasschnelli]

Concept ACK

[gavinandresen]

Removed the $(CURL_LIBS) noticed by @jonasschnelli.

And added the new libevent and libzmq dependencies.

I also decided to make it disabled by default, so you have to configure --enable-bench, so it doesn't slow down compile times for everybody.

@laanwj : RE: gettimeofday overhead and timing things that are very fast: timing how fast gettimeofday() is and then warning the developer if their benchmarking code runs into that overhead (maybe suggest that they loop 1,000 times inside the KeepRunning loop) would be another way of preventing that problem. Feel free to improve, I'm planning on using this to benchmark things that take microseconds to run, not nanoseconds.

[jgarzik]

Disagree on default-disabled.

Principle: Code that is not built by default bitrots rapidly, is not tested as much, receives less maintenance in general by random developers updating the code.

[gavinandresen]

@jgarzik : I KNEW you were going to say that...

Would a Travis configuration that runs benchmarks be good enough to prevent code rot? Compiling and linking Yet Another Binary that approximately nobody will run (unless you're actively working on optimizing something) tickles my "features shouldn't cost anything unless you're using them" sensibility.

[paveljanik]

This looks a bit strange in the benchmarking tool:

SelectParams(CBaseChainParams::MAIN);

[paveljanik]

Spaces around =?

[paveljanik]

I think it should stay disabled by default as it is now, because benchmarking will be done on purpose by developers only.

Concept ACK. I like it.

This is a way to benchmark parts of code. Now we should identify parts that need some love and prepare benchmarks for them specifically. With this framework, people can report speed fixes with proper benchmark code which can be tested before and after applying the fix.

@laanwj @gavinandresen I think that both ways to benchmark are useful and maybe we should add the second example benchmark there to illustrate.

[sipa]

Compile by default but not run, seems fine to me.

[gavinandresen]

Fixed @paveljanik 's spaces nits, and removed the unneeded SelectParams inherited from the unit test code.

Data on the "compile by default or not" decision:

It takes 8 seconds on my machine to compile and link the benchmarking code, single-processor, ccache cleared.
5 seconds doing parallel make.
Under 2 seconds with a 'hot' ccache.

[gavinandresen]

Damn you @laanwj, you inspired me to spend another hour implementing support for really fast benchmarks....

Picked @paveljanik 's CreateNewBlock nit (good catch, I copied this from a CreateNewBlock benchmark I haven't finished). I decided to rename MilliSleep.cpp to Examples.cpp, and added a "see how fast sin() runs" benchmark to test the really-fast-benchmark support (works nicely).

And switched back to compile-by-default, which seems to be the consensus.

I'm really and truly done tweaking this now, assuming Travis is happy.

[btcdrak]

Concept ACK

[jgarzik]

Yes, to be specific, I meant "compile by default but not run"

[dcousens]

concept ACK

[0xC2]

Tested.

[paveljanik]

You do not need chainparams.h now.

[paveljanik]

ACK

[paveljanik]

If enabled by default, please make it --disable-bench (see #6739).

[laanwj]

ACK