[Tests] Benchmarking Framework #1650
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
random-zebra
added
Upstream
Tests
Needs Release Notes
random-zebra
force-pushed
the
202005_microbench
branch
from
6ba1f51
to
77cbd62
Compare
- backports bitcoin/bitcoin@535ed92 Benchmarking framework, loosely based on google's micro-benchmarking library (https://github.com/google/benchmark) Wny not use the Google Benchmark framework? Because adding Even More Dependencies isn't worth it. If we get a dozen or three benchmarks and need nanosecond-accurate timings of threaded code then switching to the full-blown Google Benchmark library should be considered. The benchmark framework is hard-coded to run each benchmark for one wall-clock second, and then spits out .csv-format timing information to stdout. It is left as an exercise for later (or maybe never) to add command-line arguments to specify which benchmark(s) to run, how long to run them for, how to format results, etc etc etc. Again, see the Google Benchmark framework for where that might end up. See src/bench/MilliSleep.cpp for a sanity-test benchmark that just benchmarks 'sleep 100 milliseconds.' To compile and run benchmarks: cd src; make bench Sample output: Benchmark,count,min,max,average Sleep100ms,10,0.101854,0.105059,0.103881
- backports bitcoin/bitcoin@7072c54 Avoid calling gettimeofday every time through the benchmarking loop, by keeping track of how long each loop takes and doubling the number of iterations done between time checks when they take less than 1/16'th of the total elapsed time.
backports bitcoin/bitcoin@214de7e - ensure header namespaces and end comments are correct - add missing header end comments - ensure minimal formatting (add newlines etc.)
- backports bitcoin/bitcoin@32114dd Add benchmarks for the cryptographic hash algorithms: - RIPEMD160 - SHA1 - SHA256 - SHA512 Continues work on bitcoin#7883.
- backports bitcoin/bitcoin@63ff57d Previously the benchmark code used an integer division (%) with a non-constant in the inner-loop. This is quite slow on many processors, especially ones like ARM that lack a hardware divide. Even on fairly recent x86_64 like haswell an integer division can take something like 100 cycles-- making it comparable to the runtime of siphash. This change avoids the division by using bitmasking instead. This was especially easy since the count was only increased by doubling. This change also restarts the timing when the execution time was very low this avoids mintimes of zero in cases where one execution ends up below the timer resolution. It also reduces the impact of the overhead on the final result. The formatting of the prints is changed to not use scientific notation make it more machine readable (in particular, gnuplot croaks on the non-fixedpoint, and it doesn't sort correctly). This also hoists out all the floating point divisions out of the semi-hot path because it was easy to do so. It might be prudent to break out the critical test into a macro just to guarantee that it gets inlined. It might also make sense to just save out the intermediate counts and times and get the floating point completely out of the timing loop (because e.g. on hardware without a fast hardware FPU like some ARM it will still be slow enough to distort the results). I haven't done either of these in this commit.
- backports bitcoin/bitcoin@e0a9cb2 Make sure that the count is a zero modulo the new mask before scaling, otherwise the next time until a measure triggers will take only 1/2 as long as accounted for. This caused the 'min time' to be potentially off by as much as 100%.
- backports bitcoin/bitcoin@3532818 This adds cycle min/max/avg to the statistics. Supported on x86 and x86_64 (natively through rdtsc), as well as Linux (perf syscall).
- backports bitcoin/bitcoin@4716267 To determine the default for `-par`, the number of script verification threads, use [boost:thread:physical_concurrency()](http://www.boost.org/doc/libs/1_58_0/doc/html/thread/thread_management.html#thread.thread_management.thread.physical_concurrency) which counts only physical cores, not virtual cores. Virtual cores are roughly a set of cached registers to avoid context switches while threading, they cannot actually perform work, so spawning a verification thread for them could even reduce efficiency and will put undue load on the system. Should fix issue dashpay#6358, as well as some other reported system overload issues, especially on Intel processors. The function was only introduced in boost 1.56, so provide a utility function `GetNumCores` to fall back for older Boost versions.
available cores - backports bitcoin/bitcoin@937bf43
- backports bitcoin/bitcoin@9f03110 - backports bitcoin/bitcoin@aad4cb5
- backports bitcoin/bitcoin@29c5328 The initialization order of global data structures in different implementation units is undefined. Making use of this is essentially gambling on what the linker does, the so-called [Static initialization order fiasco](https://isocpp.org/wiki/faq/ctors#static-init-order). In this case it apparently worked on Linux but failed on OpenBSD and FreeBSD. To create it on first use, make the registration structure local to a function. Fixes bitcoin#8910.
unreachable - backports bitcoin/bitcoin@db07f91
- backports bitcoin/bitcoin@663fbae
- backports bitcoin/bitcoin@6835cb0 Avoid static analyzer warnings regarding "Function call argument is a pointer to uninitialized value" in cases where we are intentionally using such arguments. This is achieved by using ... `f(b.begin(), b.end())` (`std::array<char, N>`) ... instead of ... `f(b, b + N)` (`char b[N]`) Rationale: * Reduce false positives by guiding static analyzers regarding our intentions. Before this commit: ``` $ clang-tidy-3.5 -checks=* src/bench/base58.cpp bench/base58.cpp:23:9: warning: Function call argument is a pointer to uninitialized value [clang-analyzer-core.CallAndMessage] EncodeBase58(b, b + 32); ^ $ clang-tidy-3.5 -checks=* src/bench/verify_script.cpp bench/verify_script.cpp:59:5: warning: Function call argument is a pointer to uninitialized value [clang-analyzer-core.CallAndMessage] key.Set(vchKey, vchKey + 32, false); ^ $ ``` After this commit: ``` $ clang-tidy-3.5 -checks=* src/bench/base58.cpp $ clang-tidy-3.5 -checks=* src/bench/verify_script.cpp $ ```
std::fixed/setprecision - backports bitcoin/bitcoin@fd05132
- backports bitcoin/bitcoin@0b1b914 We were saving a div by caching the inverse as a float, but this ended up requiring a int -> float -> int conversion, which takes almost as much time as the difference between float mul and div. There are lots of other more pressing issues with the bench framework which probably require simply removing the adaptive iteration count stuff anyway.
- backports bitcoin/bitcoin@c515d26 std::chrono removes portability issues. Rather than storing doubles, store the untouched time_points. Then convert to nanoseconds for display. This allows for maximum precision, while keeping results comparable between differing hardware/operating systems. Also, display full nanosecond counts rather than sub-second floats.
zero in constructor - backports bitcoin/bitcoin@069215e lastCycles was introduced in 3532818 which was merged into master yesterday. Also initialize beginCycles to zero for consistency and completeness.
random-zebra
force-pushed
the
202005_microbench
branch
from
77cbd62
to
5a1be90
Compare
random-zebra
added a commit
to random-zebra/PIVX
that referenced
this pull request
Jun 24, 2020
Document the changes introduced in: - PIVX-Project#1549: Nuke zPIV from the GUI - PIVX-Project#1586: Minimum value for stake split threshold - PIVX-Project#1633: Bitcoin 0.12-0.14 serialization improvements - PIVX-Project#1645: Implement accurate memory accounting for mempool - PIVX-Project#1647: MemPool package tracking and limits - PIVX-Project#1650: Benchmarking Framework - PIVX-Project#1688: TopBar navigation (sync/peers)
random-zebra
added a commit
to random-zebra/PIVX
that referenced
this pull request
Jun 24, 2020
Document the changes introduced in: - PIVX-Project#1549: Nuke zPIV from the GUI - PIVX-Project#1586: Minimum value for stake split threshold - PIVX-Project#1633: Bitcoin 0.12-0.14 serialization improvements - PIVX-Project#1645: Implement accurate memory accounting for mempool - PIVX-Project#1647: MemPool package tracking and limits - PIVX-Project#1650: Benchmarking Framework - PIVX-Project#1688: TopBar navigation (sync/peers)
random-zebra
added a commit
that referenced
this pull request
Jun 29, 2020
4819ee7 [Doc] Add/Update some release notes for 4.2 (random-zebra) Pull request description: Document the changes introduced in: - #1549: Nuke zPIV from the GUI - #1586: Minimum value for stake split threshold - #1633: Bitcoin 0.12-0.14 serialization improvements - #1645: Implement accurate memory accounting for mempool - #1647: MemPool package tracking and limits - #1650: Benchmarking Framework - #1688: TopBar navigation (sync/peers) ACKs for top commit: furszy: utACK 4819ee7 Fuzzbawls: ACK 4819ee7 Tree-SHA512: 62ad949ea26a2f877ef0b40ec86616cc8105f81e1fcd380c8162cd93af04a46f1093f878c0668408654f198a0059b240798b83af3bf1d5e6c1c1d8611276a325
random-zebra
removed
the
Needs Release Notes
Liquid369
added a commit
to Liquid369/PIVX
that referenced
this pull request
Jun 30, 2020
[Doc] Add/Update some release notes for 4.2 Document the changes introduced in: - PIVX-Project#1549: Nuke zPIV from the GUI - PIVX-Project#1586: Minimum value for stake split threshold - PIVX-Project#1633: Bitcoin 0.12-0.14 serialization improvements - PIVX-Project#1645: Implement accurate memory accounting for mempool - PIVX-Project#1647: MemPool package tracking and limits - PIVX-Project#1650: Benchmarking Framework - PIVX-Project#1688: TopBar navigation (sync/peers) Update settingsfaqwidget.ui Fix Linter
Liquid369
added a commit
to Liquid369/PIVX
that referenced
this pull request
Jun 30, 2020
[Doc] Add/Update some release notes for 4.2 Document the changes introduced in: - PIVX-Project#1549: Nuke zPIV from the GUI - PIVX-Project#1586: Minimum value for stake split threshold - PIVX-Project#1633: Bitcoin 0.12-0.14 serialization improvements - PIVX-Project#1645: Implement accurate memory accounting for mempool - PIVX-Project#1647: MemPool package tracking and limits - PIVX-Project#1650: Benchmarking Framework - PIVX-Project#1688: TopBar navigation (sync/peers) Update settingsfaqwidget.ui Fix Linter [RPC] Register calls where they are defined
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces the benchmarking framework, loosely based on google's micro-benchmarking library (https://github.com/google/benchmark), ported from Bitcoin, up to 0.16.
The benchmark framework is hard-coded to run each benchmark for one wall-clock second,
and then spits out .csv-format timing information to stdout.
Backported PR:
Current output of
src/bench/bench_pivx:NOTE: Not all the tests have been pulled yet (as we might not have the code being tested, or it would require rewrites to work with our different code base), but the framework is updated to December 2017.