borg2: use blake3-mt

[ThomasWaldmann]

Even if borg is configured to use blake3 as id-hash, it currently uses only the single-threaded blake3.

[ThomasWaldmann]

I measured max_threads=AUTO vs single-threaded blake3 across input sizes to find out where the MT path is actually worth taking. Short version: it is not a simple threshold — the speedup is a sawtooth that spikes at powers of two, and below 256 KiB blake3-mt is mostly slower, down to 0.16x.

Test machine

CPU	Apple M3 Pro, 12 cores (6 performance + 6 efficiency), 16 MB L2
RAM	18 GiB
OS	macOS 15
Python	3.11.15
blake3	1.0.8

Method: keyed blake3, 32-byte digest, exactly as ID_BLAKE3_256.id_hash() calls it. Median of 7 timed runs per point, ~200 MiB hashed per run, single- and AUTO-threaded measured back-to-back per size so thermal drift cancels in the ratio. Each size was also measured at a non-1-KiB-multiple length (partial final leaf chunk); those track the aligned numbers within a few percent, so leaf alignment is not what drives the shape.

1 – 256 KiB: the loss zone

Below 8 KiB blake3 never dispatches to the thread pool, so the ratio is a flat 1.00x and tiny inputs are harmless. From 8 KiB the rayon pool engages and immediately loses — 0.16x at 8 KiB, a 6x slowdown — and stays underwater almost everywhere up to 240 KiB. The two points above the line in that stretch (128 KiB at 1.34x, 192 KiB at 1.03x) are exactly why no gate below 256 KiB is safe: their neighbours at 144 and 208 KiB fall straight back to 0.87x and 0.83x, and chunk sizes are content-defined so you cannot rely on landing on the good ones.

256 – 1040 KiB: always a win, spiking at powers of two

Every point is a win here; the floor is 1.07x at 368 KiB. Peaks land precisely on powers of two — 256 → 2.15x, 512 → 3.32x, 1024 → 4.71x — with secondary peaks at 384, 640, 768 and 896 KiB (2^k + 2^(k-1) forms). Between spikes the ratio decays as the tree gets more lopsided, bottoming just below the next power of two: 1008 KiB manages only 1.81x against 1024 KiB's 4.71x.

Beyond this range it keeps improving: 2 MiB → 5.60x, 8 MiB → 7.00x, 50 MB → ~6.7x (plateau ~14–15 GiB/s vs ~2.15 GiB/s single-threaded).

Recommended threshold, prioritising speed

Gate at 256 KiB.

def id_hash(self, data):
    mt = blake3.AUTO if len(data) >= 256 * 1024 else 1
    return blake3(data, key=self.id_key, max_threads=mt).digest(length=32)

256 KiB is the smallest size at which AUTO never loses across the whole measured range (worst case 1.07x), so it extracts the maximum available speed without ever regressing. Going lower is not a speed/safety trade-off — it is simply slower on average, because the sub-256 KiB region is dominated by losses.

For comparison, a more conservative 512 KiB gate (which has the tidy property of coinciding with CHUNK_MIN_EXP) leaves the 256–512 KiB band on the table — that band runs 1.07–1.77x, median about 1.3x. If the goal is speed, take it.

Practical impact with default CHUNKER_PARAMS (min chunk 512 KiB): normal content chunks are all ≥512 KiB and get the fast path under either gate (~5.6x at the typical ~2 MiB). The 256 KiB gate additionally covers sub-minimum chunks — whole files smaller than 512 KiB, file tails, and the items-metadata stream (ITEMS_CHUNKER_PARAMS = 15,19,17, i.e. avg ~128 KiB / max 512 KiB). Those matter most in many-small-files backups, which is also where id-hashing is the largest single CPU consumer.

Caveat

This sawtooth comes from rayon splitting work across 12 cores. Both the shape and the exact cliff position will differ on other core counts — a 4-core NAS CPU will not look like this. The threshold should be treated as hardware-tuned rather than universal, and it would be worth confirming on a low-core-count machine before settling on a constant. borg benchmark cpu already has a blake3-mt entry; extending it to sweep sizes would make this checkable per platform.

Full data (71 points)

ratio = aligned size, unaligned = same size plus a random 1–1023 byte tail. Bold = power of two.

KiB	single MiB/s	AUTO MiB/s	ratio	unaligned
1	742	748	1.01x	0.99x
2	846	846	1.00x	0.99x
4	1592	1589	1.00x	1.06x
8	1873	305	0.16x	0.17x
12	1840	419	0.23x	0.23x
16	1898	482	0.25x	0.26x
24	1908	417	0.22x	0.23x
32	1954	810	0.41x	0.47x
48	1964	659	0.34x	0.34x
64	1983	1429	0.72x	0.72x
80	1983	1019	0.51x	0.52x
96	1986	1224	0.62x	0.56x
112	1980	954	0.48x	0.49x
128	1996	2676	1.34x	1.32x
144	1995	1745	0.87x	0.88x
160	1957	1780	0.91x	0.89x
176	1992	1427	0.72x	0.72x
192	1999	2067	1.03x	1.03x
208	1994	1663	0.83x	0.82x
224	1998	1824	0.91x	0.97x
240	1998	1479	0.74x	0.74x
256	2002	4309	2.15x	2.09x
272	2003	2829	1.41x	1.41x
288	2004	2676	1.34x	1.34x
304	1997	2276	1.14x	1.12x
320	2003	2953	1.47x	1.51x
336	2003	2463	1.23x	1.32x
352	2003	2453	1.23x	1.22x
368	2009	2148	1.07x	1.06x
384	2003	3537	1.77x	1.73x
400	2007	2798	1.39x	1.41x
416	2005	2745	1.37x	1.36x
432	2004	2370	1.18x	1.19x
448	2006	2911	1.45x	1.46x
464	2005	2636	1.31x	1.25x
480	2007	2509	1.25x	1.21x
496	1917	2098	1.09x	1.15x
512	1974	6546	3.32x	3.25x
528	2006	4454	2.22x	2.37x
544	2008	4288	2.14x	2.10x
560	2008	3649	1.82x	1.79x
576	2009	4627	2.30x	2.32x
592	2007	3831	1.91x	1.91x
608	2012	4347	2.16x	1.87x
624	2009	3228	1.61x	1.63x
640	2008	4934	2.46x	2.42x
656	2015	3999	1.99x	1.97x
672	1928	4148	2.15x	2.09x
688	2015	3443	1.71x	1.70x
704	2005	4032	2.01x	2.04x
720	2003	3491	1.74x	1.77x
736	2010	3466	1.72x	1.71x
752	2013	3067	1.52x	1.51x
768	2015	5397	2.68x	2.75x
784	2019	4972	2.46x	2.18x
800	2011	4368	2.17x	2.17x
816	2010	3752	1.87x	1.88x
832	2011	4481	2.23x	2.24x
848	2010	3830	1.91x	1.92x
864	2011	3887	1.93x	1.91x
880	2009	3399	1.69x	1.68x
896	2013	4752	2.36x	2.35x
912	1971	4030	2.05x	2.00x
928	2009	3932	1.96x	1.97x
944	2008	3588	1.79x	1.77x
960	2009	4090	2.04x	2.02x
976	2010	3659	1.82x	1.82x
992	2008	3735	1.86x	1.79x
1008	2009	3626	1.81x	1.64x
1024	2011	9467	4.71x	4.59x
1040	2010	7006	3.49x	3.44x

The 656, 752 and 768 KiB points were disturbed by background load in the initial sweep and are 15-run re-measurements.

[ThomasWaldmann]

@PhrozenByte Please use that new script from PR #9959 on your old intel machine, would be interesting what threshold it recommends.

[PhrozenByte]

With pleasure! Here we go:

x86_64, 8 cores available (of 8)
Linux-7.1.5-arch1-1-x86_64-with-glibc2.44, python 3.14.6, blake3 1.0.8
46 sizes, 5 runs each, ~96 MiB per run

    KiB     single       AUTO    ratio
--------------------------------------
      4       1399       1340    0.96x  slower
      8       2321       2300    0.99x  slower
     12       2219       2120    0.96x  slower
     16       2505        579    0.23x  slower
     24       2410        667    0.28x  slower
     32       3008       1070    0.36x  slower
     48       3211        924    0.29x  slower
     64       2847       2044    0.72x  slower
     80       3024       1514    0.50x  slower
     96       3132       1750    0.56x  slower
    112       2933       1473    0.50x  slower
    128       3137       2783    0.89x  slower
    144       3353       2353    0.70x  slower
    160       3262       2483    0.76x  slower
    176       3182       2023    0.64x  slower
    192       3091       2717    0.88x  slower
    208       3164       2326    0.74x  slower
    224       3079       2502    0.81x  slower
    240       3144       2131    0.68x  slower
    256       3286       4068    1.24x
    288       3473       3460    1.00x  slower
    320       3205       3527    1.10x
    352       3188       3190    1.00x
    368       3285       2851    0.87x  slower
    384       3181       3813    1.20x
    416       3213       3403    1.06x
    448       3335       3581    1.07x
    480       3118       3114    1.00x  slower
    496       3079       2880    0.94x  slower
    512       3138       5688    1.81x
    576       3107       4602    1.48x
    640       3014       4868    1.62x
    704       2825       4821    1.71x
    768       2983       5222    1.75x
    832       2899       4602    1.59x
    896       2758       4040    1.46x
    960       2678       4523    1.69x
   1008       2838       3904    1.38x
   1024       2616       6144    2.35x
   1280       2842       5854    2.06x
   1536       2521       5531    2.19x
   1792       2737       5820    2.13x
   2032       2665       4705    1.77x
   2048       2453       5671    2.31x
   3072       2671       7113    2.66x
   4096       2553       7754    3.04x

verifying 512 KiB: probing 9 sizes that are usually worst ...

re-measuring 496 KiB (0.94x) with 15 runs ...
  0.94x -> 1.46x

re-measuring 480 KiB (1.00x) with 15 runs ...
  1.00x -> 1.04x

verifying 384 KiB: probing 2 sizes that are usually worst ...
    464 KiB 0.89x  <- disproves 384 KiB

re-measuring 464 KiB (0.89x) with 15 runs ...
  0.89x -> 1.02x

re-measuring 368 KiB (0.87x) with 15 runs ...
  0.87x -> 0.90x

============================================================
Recommended threshold: 384 KiB  (393216 bytes)
  at the threshold      : 1.20x
  worst case above it   : 1.02x
  at 4096 KiB            : 3.04x

Tell borg about it (its default is 256 KiB):

    export BORG_BLAKE3_MT_THRESHOLD=384

Only repositories using --id-hash blake3 are affected.
============================================================

measured in 28s. Note the speedup is not monotonic: it peaks at powers of two,
so a size just below a power of two is a local worst case.

Here's borg2 benchmark cpu:

Chunkers =======================================================
buzhash,19,23,21,4095      1.00 GB    2.981s
buzhash64,19,23,21,4095,2  1.00 GB    2.696s
fastcdc,19,23,21,2         1.00 GB    1.987s
fixed,1048576              1.00 GB    0.219s
Cryptographic hashes / MACs ====================================
hmac-sha256                1.00 GB    2.590s
blake2b-256                1.00 GB    1.577s
blake3                     1.00 GB    0.334s
blake3-mt                  1.00 GB    0.137s
Encryption =====================================================
aes-256-ctr-hmac-sha256    1.00 GB    3.286s
aes-256-ctr-blake2b        1.00 GB    2.562s
aes-256-ocb                1.00 GB    0.499s
chacha20-poly1305          1.00 GB    0.841s
Compression ====================================================
lz4          100.00 MB  0.014s
zstd,1       100.00 MB  0.054s
zstd,3       100.00 MB  0.067s
zstd,5       100.00 MB  0.126s
zstd,10      100.00 MB  0.149s
zstd,16      100.00 MB  14.589s
zstd,22      100.00 MB  14.415s
zlib,0       100.00 MB  0.073s
zlib,6       100.00 MB  3.516s
zlib,9       100.00 MB  3.548s
lzma,0       100.00 MB  18.847s
lzma,6       100.00 MB  34.483s
lzma,9       100.00 MB  26.399s
msgpack ========================================================
msgpack      100k Items 0.222s

Note that I ran this on a system in use and didn't boot up a clean live system. I did run borg2 benchmark cpu three times and the script five times to verify that the results are sound. For borg2 benchmark cpu they are, the script's measured rates are quite sporadic though (the ratios to a way lesser extend). But the script's recommendation is always the same. Let me know if you want me to run it again.

[PhrozenByte]

Playing with --thorough and --runs a bit: With a higher --runs (e.g. --runs 20, I even tried --runs 60 once, see the generated SVG below) it recommends the default 256 KiB most of the time (a few times it recommended 384 KiB instead, once even 240 KiB). With --thorough the script basically always recommends 512 KiB instead (it recommended 448 KiB once).

[ThomasWaldmann]

Great, looks like the default 256kiB threshold even holds for very different CPUs.

It's not a problem if for some chunks it gets slower, if for most chunks it gets significantly faster.