Simple benchmarking framework #6733

Merged
merged 2 commits into from Oct 6, 2015

Conversation

Projects
None yet
9 participants
@gavinandresen
Contributor

gavinandresen commented Sep 28, 2015

Benchmarking framework, loosely based on google's micro-benchmarking
library (https://github.com/google/benchmark)

Wny not use the Google Benchmark framework? Because adding Even More Dependencies
isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate
timings of threaded code then switching to the full-blown Google Benchmark library
should be considered.

The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout. It is left as an
exercise for later (or maybe never) to add command-line arguments to specify which
benchmark(s) to run, how long to run them for, how to format results, etc etc etc.
Again, see the Google Benchmark framework for where that might end up.

See src/bench/Examples.cpp for a sanity-test benchmark that just benchmarks
'sleep 100 milliseconds' and a benchmark for math.h's sin() function.

To compile and run benchmarks:

  configure --enable-bench  ...other configure options
  cd src; make bench

Sample output:

Benchmark,count,min,max,average
Sleep100ms,10,0.10048,0.104789,0.103251
Trig,37748735,0,4.76837e-07,2.71684e-08
+static void CODE_TO_TIME(benchmark::State& state)
+{
+ ... do any setup needed...
+ while (state.KeepRunning()) {

This comment has been minimized.

@laanwj

laanwj Sep 28, 2015

Member

This works well as long as 'stuff you want to time' does not run too quickly - otherwise the state.KeepRunning(), which does a gettimeofday syscall, with the implied latency of that, will dominate.
(an alternative to calling the time function in the benchmark loop that is commonly used, is to first time a fixed # of runs of the loop, then from the result compute a number of iterations so that it runs for approximately the allotted time)

@laanwj

laanwj Sep 28, 2015

Member

This works well as long as 'stuff you want to time' does not run too quickly - otherwise the state.KeepRunning(), which does a gettimeofday syscall, with the implied latency of that, will dominate.
(an alternative to calling the time function in the benchmark loop that is commonly used, is to first time a fixed # of runs of the loop, then from the result compute a number of iterations so that it runs for approximately the allotted time)

@laanwj laanwj added the Tests label Sep 28, 2015

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Sep 28, 2015

Member

Concept ACK

Member

laanwj commented Sep 28, 2015

Concept ACK

@jonasschnelli

View changes

src/Makefile.bench.include
+if ENABLE_WALLET
+bench_bench_bitcoin_LDADD += $(LIBBITCOIN_WALLET)
+endif
+bench_bench_bitcoin_LDADD += $(LIBBITCOIN_CONSENSUS) $(BDB_LIBS) $(SSL_LIBS) $(CRYPTO_LIBS) $(MINIUPNPC_LIBS) $(CURL_LIBS)

This comment has been minimized.

@jonasschnelli

jonasschnelli Sep 28, 2015

Member

rm $(CURL_LIBS)? This seems to have be sneaked in from XT.

@jonasschnelli

jonasschnelli Sep 28, 2015

Member

rm $(CURL_LIBS)? This seems to have be sneaked in from XT.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 28, 2015

Member

Concept ACK

Member

jonasschnelli commented Sep 28, 2015

Concept ACK

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Sep 29, 2015

Contributor

Removed the $(CURL_LIBS) noticed by @jonasschnelli.

And added the new libevent and libzmq dependencies.

I also decided to make it disabled by default, so you have to configure --enable-bench, so it doesn't slow down compile times for everybody.

@laanwj : RE: gettimeofday overhead and timing things that are very fast: timing how fast gettimeofday() is and then warning the developer if their benchmarking code runs into that overhead (maybe suggest that they loop 1,000 times inside the KeepRunning loop) would be another way of preventing that problem. Feel free to improve, I'm planning on using this to benchmark things that take microseconds to run, not nanoseconds.

Contributor

gavinandresen commented Sep 29, 2015

Removed the $(CURL_LIBS) noticed by @jonasschnelli.

And added the new libevent and libzmq dependencies.

I also decided to make it disabled by default, so you have to configure --enable-bench, so it doesn't slow down compile times for everybody.

@laanwj : RE: gettimeofday overhead and timing things that are very fast: timing how fast gettimeofday() is and then warning the developer if their benchmarking code runs into that overhead (maybe suggest that they loop 1,000 times inside the KeepRunning loop) would be another way of preventing that problem. Feel free to improve, I'm planning on using this to benchmark things that take microseconds to run, not nanoseconds.

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 29, 2015

Contributor

Disagree on default-disabled.

Principle: Code that is not built by default bitrots rapidly, is not tested as much, receives less maintenance in general by random developers updating the code.

Contributor

jgarzik commented Sep 29, 2015

Disagree on default-disabled.

Principle: Code that is not built by default bitrots rapidly, is not tested as much, receives less maintenance in general by random developers updating the code.

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Sep 29, 2015

Contributor

@jgarzik : I KNEW you were going to say that...

Would a Travis configuration that runs benchmarks be good enough to prevent code rot? Compiling and linking Yet Another Binary that approximately nobody will run (unless you're actively working on optimizing something) tickles my "features shouldn't cost anything unless you're using them" sensibility.

Contributor

gavinandresen commented Sep 29, 2015

@jgarzik : I KNEW you were going to say that...

Would a Travis configuration that runs benchmarks be good enough to prevent code rot? Compiling and linking Yet Another Binary that approximately nobody will run (unless you're actively working on optimizing something) tickles my "features shouldn't cost anything unless you're using them" sensibility.

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Sep 29, 2015

Contributor

This looks a bit strange in the benchmarking tool:

SelectParams(CBaseChainParams::MAIN);
Contributor

paveljanik commented Sep 29, 2015

This looks a bit strange in the benchmarking tool:

SelectParams(CBaseChainParams::MAIN);
@paveljanik

View changes

src/Makefile.bench.include
@@ -0,0 +1,45 @@
+bin_PROGRAMS += bench/bench_bitcoin
+BENCH_SRCDIR = bench
+BENCH_BINARY=bench/bench_bitcoin$(EXEEXT)

This comment has been minimized.

@paveljanik

paveljanik Sep 29, 2015

Contributor

Spaces around =?

@paveljanik

paveljanik Sep 29, 2015

Contributor

Spaces around =?

@paveljanik

View changes

src/Makefile.bench.include
+BENCH_BINARY=bench/bench_bitcoin$(EXEEXT)
+
+
+bench_bench_bitcoin_SOURCES =\

This comment has been minimized.

@paveljanik

paveljanik Sep 29, 2015

Contributor

Space after =

@paveljanik

paveljanik Sep 29, 2015

Contributor

Space after =

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Sep 29, 2015

Contributor

I think it should stay disabled by default as it is now, because benchmarking will be done on purpose by developers only.

Concept ACK. I like it.

This is a way to benchmark parts of code. Now we should identify parts that need some love and prepare benchmarks for them specifically. With this framework, people can report speed fixes with proper benchmark code which can be tested before and after applying the fix.

@laanwj @gavinandresen I think that both ways to benchmark are useful and maybe we should add the second example benchmark there to illustrate.

Contributor

paveljanik commented Sep 29, 2015

I think it should stay disabled by default as it is now, because benchmarking will be done on purpose by developers only.

Concept ACK. I like it.

This is a way to benchmark parts of code. Now we should identify parts that need some love and prepare benchmarks for them specifically. With this framework, people can report speed fixes with proper benchmark code which can be tested before and after applying the fix.

@laanwj @gavinandresen I think that both ways to benchmark are useful and maybe we should add the second example benchmark there to illustrate.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 29, 2015

Member

Compile by default but not run, seems fine to me.

Member

sipa commented Sep 29, 2015

Compile by default but not run, seems fine to me.

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Sep 29, 2015

Contributor

Fixed @paveljanik 's spaces nits, and removed the unneeded SelectParams inherited from the unit test code.

Data on the "compile by default or not" decision:

It takes 8 seconds on my machine to compile and link the benchmarking code, single-processor, ccache cleared.
5 seconds doing parallel make.
Under 2 seconds with a 'hot' ccache.

Contributor

gavinandresen commented Sep 29, 2015

Fixed @paveljanik 's spaces nits, and removed the unneeded SelectParams inherited from the unit test code.

Data on the "compile by default or not" decision:

It takes 8 seconds on my machine to compile and link the benchmarking code, single-processor, ccache cleared.
5 seconds doing parallel make.
Under 2 seconds with a 'hot' ccache.

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Sep 29, 2015

Contributor

Damn you @laanwj, you inspired me to spend another hour implementing support for really fast benchmarks....

Picked @paveljanik 's CreateNewBlock nit (good catch, I copied this from a CreateNewBlock benchmark I haven't finished). I decided to rename MilliSleep.cpp to Examples.cpp, and added a "see how fast sin() runs" benchmark to test the really-fast-benchmark support (works nicely).

And switched back to compile-by-default, which seems to be the consensus.

I'm really and truly done tweaking this now, assuming Travis is happy.

Contributor

gavinandresen commented Sep 29, 2015

Damn you @laanwj, you inspired me to spend another hour implementing support for really fast benchmarks....

Picked @paveljanik 's CreateNewBlock nit (good catch, I copied this from a CreateNewBlock benchmark I haven't finished). I decided to rename MilliSleep.cpp to Examples.cpp, and added a "see how fast sin() runs" benchmark to test the really-fast-benchmark support (works nicely).

And switched back to compile-by-default, which seems to be the consensus.

I'm really and truly done tweaking this now, assuming Travis is happy.

@btcdrak

This comment has been minimized.

Show comment
Hide comment
@btcdrak

btcdrak Sep 29, 2015

Member

Concept ACK

Member

btcdrak commented Sep 29, 2015

Concept ACK

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 29, 2015

Contributor

Yes, to be specific, I meant "compile by default but not run"

Contributor

jgarzik commented Sep 29, 2015

Yes, to be specific, I meant "compile by default but not run"

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Sep 29, 2015

Contributor

concept ACK

Contributor

dcousens commented Sep 29, 2015

concept ACK

@0xC2

This comment has been minimized.

Show comment
Hide comment
@0xC2

0xC2 Sep 30, 2015

Tested.

0xC2 commented Sep 30, 2015

Tested.

@paveljanik

View changes

src/bench/bench_bitcoin.cpp
+
+#include "bench.h"
+
+#include "chainparams.h"

This comment has been minimized.

@paveljanik

paveljanik Sep 30, 2015

Contributor

You do not need chainparams.h now.

@paveljanik

paveljanik Sep 30, 2015

Contributor

You do not need chainparams.h now.

+int
+main(int argc, char** argv)
+{
+ ECC_Start();

This comment has been minimized.

@paveljanik

paveljanik Sep 30, 2015

Contributor

ECC_*() are here probably because you are testing ConnectBlock()? This should be in the specific benchmark init section IMO, not here.

@paveljanik

paveljanik Sep 30, 2015

Contributor

ECC_*() are here probably because you are testing ConnectBlock()? This should be in the specific benchmark init section IMO, not here.

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Sep 30, 2015

Contributor

ACK

Contributor

paveljanik commented Sep 30, 2015

ACK

gavinandresen added some commits Sep 24, 2015

Simple benchmarking framework
Benchmarking framework, loosely based on google's micro-benchmarking
library (https://github.com/google/benchmark)

Wny not use the Google Benchmark framework? Because adding Even More Dependencies
isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate
timings of threaded code then switching to the full-blown Google Benchmark library
should be considered.

The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout. It is left as an
exercise for later (or maybe never) to add command-line arguments to specify which
benchmark(s) to run, how long to run them for, how to format results, etc etc etc.
Again, see the Google Benchmark framework for where that might end up.

See src/bench/MilliSleep.cpp for a sanity-test benchmark that just benchmarks
'sleep 100 milliseconds.'

To compile and run benchmarks:
  cd src; make bench

Sample output:

Benchmark,count,min,max,average
Sleep100ms,10,0.101854,0.105059,0.103881
Support very-fast-running benchmarks
Avoid calling gettimeofday every time through the benchmarking loop, by keeping
track of how long each loop takes and doubling the number of iterations done
between time checks when they take less than 1/16'th of the total elapsed time.
@@ -91,6 +91,11 @@ AC_ARG_ENABLE(tests,
[use_tests=$enableval],
[use_tests=yes])
+AC_ARG_ENABLE(bench,
+ AS_HELP_STRING([--enable-bench],[compile benchmarks (default is yes)]),

This comment has been minimized.

@paveljanik

paveljanik Oct 2, 2015

Contributor

If enabled by default, please make it --disable-bench (see #6739).

@paveljanik

paveljanik Oct 2, 2015

Contributor

If enabled by default, please make it --disable-bench (see #6739).

@@ -61,6 +61,7 @@ endif
bin_PROGRAMS =
TESTS =
+BENCHMARKS =

This comment has been minimized.

@paveljanik

paveljanik Oct 2, 2015

Contributor

BTW - why do you have this variable here? Can it be deleted?

@paveljanik

paveljanik Oct 2, 2015

Contributor

BTW - why do you have this variable here? Can it be deleted?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 6, 2015

Member

ACK

Member

laanwj commented Oct 6, 2015

ACK

@laanwj laanwj merged commit 7072c54 into bitcoin:master Oct 6, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Oct 6, 2015

Merge pull request #6733
7072c54 Support very-fast-running benchmarks (Gavin Andresen)
535ed92 Simple benchmarking framework (Gavin Andresen)

@laanwj laanwj referenced this pull request Apr 15, 2016

Closed

Add benchmarks to `bench_bitcoin` #7883

6 of 8 tasks complete

@str4d str4d referenced this pull request in zcash/zcash Oct 15, 2016

Merged

Upstream gitian updates #1541

@str4d str4d referenced this pull request in zcash/zcash May 9, 2017

Open

Merge upstream's benchmarking framework. #2357

@dagurval dagurval referenced this pull request in bitcoinxt/bitcoinxt Jan 16, 2018

Merged

Simple benchmarking framework #298

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment