use multi-threaded blake3 for big chunks, #9958 - #9959
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9959 +/- ##
==========================================
+ Coverage 85.68% 85.70% +0.01%
==========================================
Files 95 95
Lines 16799 16815 +16
Branches 2574 2577 +3
==========================================
+ Hits 14394 14411 +17
+ Misses 1669 1668 -1
Partials 736 736 ☔ View full report in Codecov by Harness. |
blake3 can hash a single input in parallel, but borg always used the single-threaded path. Let id_hash use blake3's internal thread pool once the input is big enough. Multi-threading is not a win at every size: blake3 parallelises over its binary hash tree, so the speedup peaks at powers of two and decays in between, and for small inputs the thread dispatch costs much more than it saves. Measured on a 12-core machine (Apple M3 Pro): 0.16x at 8 KiB, still below 1x at 240 KiB, then 2.0x at 256 KiB, 3.2x at 512 KiB, 5.6x at 2 MiB and 7.3x at 8 MiB. 256 KiB was the smallest size that never lost there, so that is the default and smaller chunks keep hashing single-threaded. Chunk sizes are content-defined, so we can not rely on hitting the sizes that happen to parallelise well and have to be conservative. The optimal value depends on the core count and therefore differs per machine, so BORG_BLAKE3_MT_THRESHOLD (in KiB) can override the default. It is evaluated on first use and then cached: get_blake3_mt_threshold() runs for every chunk and os.environ.get() costs ~425ns (~30% of id_hash for a 512B chunk), while the cached call costs ~45ns. Evaluating it lazily rather than at import time also means an invalid value is reported by borg's normal error handling instead of a traceback while importing. scripts/blake3-optimize-mt-threshold.py measures the best value for a machine: it sweeps input sizes, verifies a candidate by probing the sizes that are usually worst, re-measures the point that determines the threshold to keep scheduling noise out, and optionally writes an HTML or SVG chart. This only affects repos using --id-hash blake3. The resulting chunk ids are unchanged - that single- and multi-threaded blake3 agree is the blake3 project's invariant, so there is no test for it here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
508e7f9 to
198c751
Compare
|
End-to-end benchmark of Workload: one 10 GiB file of incompressible random data, All three rows are the same binary from this branch — the single-threaded variant is produced with fastcdc,19,23,21,2
buzhash64,19,23,21,4095,2
Reading it
The cost: CPU timeblake3-mt buys wallclock by spending ~55% more CPU seconds on create (45.3s vs 29.3s) and ~2x on extract (30.8s vs 15.0s). It is trading otherwise-idle cores for latency, not doing less work - rayon's tree splitting has real overhead. On a 12-core desktop with 11 cores idle that is close to free. On a busy server, a low-core NAS, or anything where borg competes for CPU with other work, it is not obviously a good trade. Two things follow:
Happy to add that note to Scope reminderNone of this affects default repositories, which use sha256 chunk ids. It only applies to Reproduce with 🤖 Generated with Claude Code |
As I followed this discussion, I was going to make the point that I now see that Claude has made, quoted above. Looking at the table shows that create's wall clock goes from 32.8s to 30.3s while the cpu time goes from 31.2s to 47.2s, and this does not seem worth it to me. That will interfere with other processes, cause more heat and more power usage, and may make fans run more, all for the time taken by what is usually a background job. On some systems, this might be worth it, but I don't think it should be the default. So I would agree with what Claude said, namely that the threshold should be conservative by default, with multithreading only kicking in when the time savings is more significant than 32.8s -> 30.3s. (I know that this can be tweaked by the user by setting an environment variable, so I'm just arguing that the default should be different.) I don't fully understand how the numbers above would change with different thresholds, so I don't have a specific suggestion for a value. Maybe it's worth some additional testing for overall I think similar arguments apply to the |
|
@jdchristensen Thanks for your feedback. If one values CPU cycles as high as saving overall runtime, you're totally right. My goal was to optimise for overall runtime though, considering there are lots of cores nowadays, backups are often run in less busy times (e.g. at night or weekend), so most of these cores are sitting around idle. You say "who cares for runtime, it is a cron job", but some people have so much data to back up that timing becomes an issue. Also, if one interactively runs backups, one usually also prefers higher speed over saving cpu cycles. Last, but not least, people like to compare tools with benchmarks. :-) It is also NOT the case that borg would hog all the cores all the time, it is just using MT for blake3 (and zstd, see the other PR), everything else is still very sequential and using less than 1 full core). For extract, the overall runtime is even much more important: if you lost a disk, you will run a full extract and until that is done, you maybe can't use that system. So you definitely will prefer using more cores over waiting longer. Same thing, even if it is just a few, but big files, e.g. a VM or big database dump. So I'ld say, we'll just test it as is and wait for feedback from testers. About "being worth it": the roughly 10% overall savings is quite a lot, considering that hashing is just one of many steps. I did some own measurements last night and had a speed up of borg create of x1.75 (it was just 1 big file) when comparing blake3+zstd MT to non-MT. If one wants to rather safe energy etc., guess hw accelerated (hmac-)sha256 (and lz4) is the way to go. |
|
@ThomasWaldmann You make some good points, but I wonder if a higher threshold would get most of the wall clock savings with lower cpu time? The wall clock savings for sizes just above the current threshold was much lower, so that's the range in which you might be spending lots of cpu on little gain. Doing a comparison of a full create with the current threshold and double the current threshold would be pretty informative, I think. |
|
A higher threshold would be better for efficiency, but worse for speed, because less chunks then would qualify. It could be also set e.g. at 512kiB, but then a lot of files between 256 and 512kiB would not get a (small) speedup by MT. Guess we'll need some experiment with real world datasets to see how much difference that makes. |
2 participants
Fixes #9958.
borg only ever used single-threaded blake3, even though the library can hash a single input in parallel. This lets
ID_BLAKE3_256.id_hash()use blake3's internal thread pool once the input is big enough.Why a threshold, and why 256 KiB
Multi-threading is not a win at every size. blake3 parallelises over its binary hash tree, so the speedup peaks at powers of two and decays in between, and for small inputs the thread dispatch costs far more than it saves. Sweeping 71 input sizes on a 12-core machine (Apple M3 Pro, 6P+6E, blake3 1.0.8):
The sub-256 KiB region is erratic as well as bad: 128 KiB wins at 1.34x but 144 KiB loses at 0.87x, 192 KiB wins at 1.03x but 208 KiB loses at 0.83x. Chunk sizes are content-defined, so we cannot rely on landing on the sizes that happen to parallelise well.
256 KiB is the smallest size that never lost across the measured range (worst case above it: 1.07x at 368 KiB), so that is the default and anything smaller keeps hashing single-threaded.
Measured end-to-end through
id_hash()on this branch:Full measurement data and charts are in the issue comment.
Tuning
The optimal threshold depends on the core count, so
BORG_BLAKE3_MT_THRESHOLDoverrides the default. The unit is KiB, which is also what the measurement script prints, so its output can be pasted verbatim:0 means "always multi-threaded", a very large value effectively disables multi-threading. Documented in
docs/usage/general/environment.rst.inc, which points at the measurement script below.Scope and compatibility
--id-hash blake3. sha256 repos (the default) are untouched.max_threadsonly affects how the hash is computed, not the result — that invariant is the blake3 project's to test, so there is no test for it here.scripts/blake3-optimize-mt-threshold.pyThe optimal threshold depends on the core count, so it differs per machine — a 4-core NAS CPU will not look like the numbers above. This adds a small end-user script to measure it:
It only needs the
blake3package, which borg already depends on. Two details matter for getting a trustworthy answer:It can also write a stand-alone SVG (
--svg) and re-render a chart from an earlier run's JSON without re-measuring (--from-json).Notes for review
max_threads=1is already the blake3 default, so theelse 1branch is technically redundant. It is spelled out on purpose: it keeps a single call site, and the blake3 docs note that API-compatible reimplementations may ignoremax_threadsentirely, so being explicit keeps the small-chunk path single-threaded regardless of what a future version defaults to.Error: BORG_BLAKE3_MT_THRESHOLD must be an integer (KiB), but is: '64k'and exits 2. (For comparison,BORG_PACK_MAX_SIZE=abccurrently produces an unhandled crash dump - could be worth fixing the same way.)get_blake3_mt_threshold()runs for every chunk andos.environ.get()is not free: ~425ns per call (the env-unset path is the slower one, it raises and catchesKeyErrorinternally), which is ~30% ofid_hash()for a 512B chunk. Cached, the call costs ~45ns.blake3-mtentry inborg benchmark cpuinto a size sweep, so this is checkable per platform without a separate script.Tests: full
src/borg/testsuite/crypto/suite passes (86, incl. a new one covering the env var parsing), pluscreate_cmd_test.pyandcheck_cmd_test.py. black and ruff clean.🤖 Generated with Claude Code