benchmark cpu: throughput column, buffer sizes, section selection - #10052
Merged
ThomasWaldmann merged 11 commits intoAug 7, 2026
Conversation
The output only gave the time for a fixed amount of data, so comparing
rows meant dividing in your head. Every row now ends in MB/s - always
MB/s, never scaled to kB/s or GB/s, so the numbers stay comparable
between a chunker at 3500 and lzma at 5.
fastcdc,19,23,21,2 1.00 GB 0.283s 3537.6 MB/s
lzma,6 100.00 MB 18.815s 5.3 MB/s
msgpack packs items rather than bytes, so it reports kItems/s instead;
that row already had "100k Items" where the others have a byte count.
The reported sizes are now derived from the work actually done
(data_size * iterations) instead of being hardcoded. In the normal run
that is the same 1 GB and 100 MB as before, but under
_BORG_BENCHMARK_CPU_TEST - one iteration over 100 kB, used to keep CI
fast - the old constants claimed 1 GB for 100 kB of work, which would
have made the new column wrong by four orders of magnitude. The msgpack
item count had the same problem and is now counted the same way.
JSON output is unchanged apart from those honest sizes: throughput is
size/time, so a consumer can compute it and does not need a new field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
…ackup#10050 blake3 and zstd only use multiple threads above a size threshold, so a single buffer size hid most of what there is to see. The hashes are now measured at 100kiB (below the threshold), 2MiB (a typical borg chunk) and 50MiB (a borg pack), the compressors at 100kiB and 2MiB. Every row still processes about the same total number of bytes, so the throughput column stays comparable across sizes. That difference is the whole point of the change: blake3-mt (100kiB) 999.94 MB 0.851s 1175.3 MB/s blake3-mt (2MiB) 998.24 MB 0.078s 12867.8 MB/s blake3-mt (50MiB) 996.15 MB 0.058s 17294.7 MB/s Algorithms whose behaviour does not depend on the buffer size get one row at 2MiB rather than three identical ones. Adds --chunking, --hashing, --encryption, --compression and --msgpack; giving none runs everything, as before. The issue asked for the first four - msgpack got one too, so that every section can be selected and none is stuck always running. The full run does not get slower despite having 46 rows instead of 28: the compressors now do half as many bytes per size, so their total is unchanged, and they dominate the runtime. JSON entries for hashes and compression gain a buffer_size field, without which rows for the same algorithm could no longer be told apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
The compression section fed os.urandom() to every codec. Nothing
compresses, so they all took their incompressible fast path: the levels
barely differed, and zstd's multi-threading looked like a 2-5x loss when
on real data it is a 1.7-3x win. The section measured a case no backup
ever hits.
It now compresses a deterministic word soup instead - built from a fixed
seed with a hand-written xorshift, so the bytes do not depend on the
Python version's PRNG internals and every machine and run sees the same
input. It compresses about 4.3x at zstd,3, and the section prints that
ratio so the numbers can be read.
The multi-threading threshold now shows up with the right sign:
zstd,1 (100kiB) 9.93 MB 0.018s 562.7 MB/s
zstd,1 (2MiB) 8.39 MB 0.006s 1417.6 MB/s
Only the compressors change. The chunkers keep random data, which they
need for realistic cut points, and hash/encryption throughput does not
depend on the content.
Bytes per row drop from 50 to 10 MB: with real work to do the slow codecs
are much slower, and the section went from ~30 to 91 seconds. At 10 MB it
takes 18, with a floor of 3 repetitions so the fast codecs still get a
few.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
There were two rows, blake3 and blake3-mt, but borg has no such choice:
AESKeyBase.id_hash passes max_threads=AUTO only when the chunk reaches
get_blake3_mt_threshold() (256 KiB by default), and 1 below it. So one of
the two rows was always measuring something borg would not do at that
size.
Now there is a single blake3 row applying that same rule, so each buffer
size shows what borg would really do with a chunk of that size:
blake3 (100kiB) 999.94 MB 0.442s 2260.1 MB/s
blake3 (2MiB) 998.24 MB 0.077s 12968.2 MB/s
blake3 (50MiB) 996.15 MB 0.057s 17623.1 MB/s
The 100kiB row is single-threaded and the faster for it - which is the
threshold doing its job, not a penalty.
It calls get_blake3_mt_threshold() rather than hardcoding the size, so
BORG_BLAKE3_MT_THRESHOLD moves the benchmark and borg together and the
benchmark can be used to tune it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
The data volumes were decimal (10^9 and 10^7 bytes) and the buffer sizes
binary, so the totals landed on awkward values and the size column read
"998.24 MB" or "9.93 MB" depending on which buffer a row used.
Every row now processes exactly 1 GiB, or 10 MiB for the compressors,
and says so:
blake3 (128kiB) 1.00 GiB 0.474s 2266.5 MB/s
blake3 (2MiB) 1.00 GiB 0.085s 12590.2 MB/s
lz4 (128kiB) 10.00 MiB 0.011s 988.4 MB/s
Exactness needs buffers that divide those totals, so the two that did not
became powers of two: 100kiB -> 128kiB and 50MiB -> 64MiB for the hashes,
and the chunker/encryption buffer 10 MB -> 8 MiB. They serve the same
purpose as before - 128kiB is still below blake3's 256 KiB threshold and
zstd's 768 KiB one, 64MiB is still pack sized.
The size column now prints IEC units directly rather than going through
format_file_size. These volumes are constants of the benchmark, not user
data: following BORG_UNITS would make two people's output incomparable,
and would render 1 GiB as "1.07 GB".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
The order was historical rather than meaningful. Now the two AEAD modes come first - aes-256-ocb and chacha20-poly1305, which is what borg actually uses - followed by the older encrypt-then-MAC ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
They were ordered codec-first, so each codec's two sizes sat together and comparing codecs meant reading every other line. Now the buffer size is the outer loop: all codecs at 2MiB, then all at 128kiB. 2MiB comes first because that is the size borg actually compresses - a typical chunk - and it is the one above zstd's multi-threading threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
zlib level 0 stores rather than compresses - it returns the input unchanged, ratio 1.00x - so its row measured memcpy speed dressed up as a compressor. zlib,1 is the fastest level that actually compresses, which is what the other codecs' lowest rows measure. Also drops the "(test data is compressible, Nx at zstd,3)" line: it cost a compression run just to print, and the ratio belongs in the commit that introduced the test data rather than in every run's output. lzma,0 stays. Unlike zlib it is not a store mode - it compresses, and on repetitive input to the same size as lzma,6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
blake3 leads, at pack size then chunk size, followed by hmac-sha256 and
blake2b-256 at chunk size:
blake3 (64MiB) 1.00 GiB 0.063s 17147.6 MB/s
blake3 (2MiB) 1.00 GiB 0.088s 12252.2 MB/s
hmac-sha256 (2MiB) 1.00 GiB 0.352s 3050.6 MB/s
blake2b-256 (2MiB) 1.00 GiB 1.003s 1070.3 MB/s
Drops the 128kiB blake3 row. It sat below blake3's multi-threading
threshold, so it measured the single-threaded path - true to what borg
does with a chunk that small, but not what the section is for now that
the remaining rows are all multi-threaded and directly comparable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
The epilog still described earlier revisions of this branch: 100kiB and
50MiB rather than the 128kiB and 64MiB the code uses, hashes measured
below the multi-threading threshold when that row is gone (they are now
measured at pack and chunk size, both above it), and "every row processes
roughly the same total number of bytes" when the compressors do 10 MiB
and everything else 1 GiB. It also no longer said anywhere that the
compression section runs on compressible data, without which its numbers
cannot be read. This text ends up in the man page, so it should be right.
The spec column was 26 characters wide in three sections and 20 in the
other two, and the msgpack row padded its size field to 10 where the rest
use 11, so nothing lined up between sections:
blake3 (64MiB) 1.00 GiB 0.062s 17253.6 MB/s
msgpack 128k Items 0.051s 2515.6 kItems/s
One WIDTH constant now covers all of them - 26 fits the longest spec of
any section, "buzhash64,19,23,21,4095,2".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10052 +/- ##
==========================================
+ Coverage 86.63% 86.75% +0.12%
==========================================
Files 97 97
Lines 16912 17061 +149
Branches 2550 2582 +32
==========================================
+ Hits 14651 14802 +151
+ Misses 1570 1569 -1
+ Partials 691 690 -1 ☔ View full report in Codecov by Harness. |
--encryption / --compression / --msgpack become --encrypting / --compressing / --msgpacking, so all five section flags are gerunds like the already existing --chunking and --hashing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #10049 (throughput column) and #10050 (buffer sizes, section selection), plus some follow-up fixes to make the numbers mean something.
Output
Whole run takes about 29 s, down from ~50 s before, despite having more rows.
#10049 - throughput column
Every row ends in MB/s - always MB/s, never rescaled, so a chunker at 3452 and lzma at 4.0 stay comparable. msgpack reports kItems/s, since it packs items rather than bytes.
The reported sizes are now derived from the work actually done rather than hardcoded. Under
_BORG_BENCHMARK_CPU_TEST(one iteration over a small buffer, used to keep CI fast) the old constants claimed 1 GB for 100 kB of work, which would have made the new column wrong by four orders of magnitude.#10050 - buffer sizes and section selection
blake3 and zstd only use multiple threads above a size threshold, so a single buffer size hid most of what there is to see. Hashes are measured at 64MiB (a borg pack) and 2MiB (a borg chunk), compressors at 2MiB and 128kiB (below zstd's 768 KiB threshold). Every row processes exactly 1 GiB, or 10 MiB for the compressors, so the throughput column is comparable across sizes.
--chunking,--hashing,--encrypting,--compressing,--msgpackingrun only those sections; none given runs everything.Follow-ups that changed what is measured
Compressible test data. The compression section fed
os.urandom()to every codec. Nothing compresses, so they all took their incompressible fast path: levels barely differed, and zstd's multi-threading looked like a 2-5x loss when on real data it is a 1.7-3x win. It now compresses a deterministic word soup (fixed seed, hand-written xorshift so the bytes do not depend on the Python version's PRNG), which compresses about 4.3x at zstd,3. The threshold now shows with the right sign -zstd,1544 MB/s at 128kiB vs 1312 at 2MiB.blake3 as borg uses it. There were two rows,
blake3andblake3-mt, but borg has no such choice:AESKeyBase.id_hashpassesmax_threads=AUTOonly aboveget_blake3_mt_threshold(). One row now applies that same rule, calling the helper rather than hardcoding the size, soBORG_BLAKE3_MT_THRESHOLDmoves borg and the benchmark together.zlib,1 instead of zlib,0. zlib level 0 stores rather than compresses (ratio 1.00x), so that row measured memcpy speed dressed up as a compressor - it reported ~7900 MB/s. (
lzma,0is kept: unlike zlib it is not a store mode, it compresses to the same size aslzma,6on repetitive input.)Binary data sizes. 1 GiB and 10 MiB rather than decimal 10^9/10^7, with buffer sizes as powers of two so every row lands on exactly those figures instead of "998.24 MB". The size column prints IEC units directly rather than via
format_file_size: these are constants of the benchmark, not user data, and followingBORG_UNITSwould make two people's output incomparable and render 1 GiB as "1.07 GB".Ordering. AEAD encryption modes first; hashes fastest first; compression grouped by buffer size (all codecs at 2MiB, then all at 128kiB) so codecs can be compared down a column.
Notes
buffer_sizeon the hashes and compression entries, without which rows for the same algorithm could not be told apart. Throughput is not added - it issize/time, so consumers can compute it.🤖 Generated with Claude Code