Sneak peek to borg 2.0.0b23

[ThomasWaldmann]

I installed a new warp core and this is the result:

Compression

Recently, I enabled zstd's internal multithreading when compressing big chunks - it can get faster than lz4 then (measured with compressible data):

Apple M3 Pro: borg benchmark cpu --compressing, 1 run
--------------------------------------------------------------
lz4 (2MiB)                 10.00 MiB   0.014s     740.7 MB/s
zstd,1 (2MiB)              10.00 MiB   0.008s    1317.5 MB/s
zstd,3 (2MiB)              10.00 MiB   0.008s    1322.7 MB/s
zstd,5 (2MiB)              10.00 MiB   0.019s     549.7 MB/s

AMD Ryzen 5 8500GE: borg benchmark cpu --compressing, 1 run
--------------------------------------------------------------
lz4 (2MiB)                 10.00 MiB   0.010s    1055.0 MB/s
zstd,1 (2MiB)              10.00 MiB   0.013s     838.3 MB/s
zstd,3 (2MiB)              10.00 MiB   0.014s     731.7 MB/s
zstd,5 (2MiB)              10.00 MiB   0.030s     346.7 MB/s

Chunking

I added some new chunkers, optimized some we already had:

Apple M3 Pro: borg benchmark cpu --chunking, median of 3 runs, MB/s
                    |  scalar | blockwise |    neon |     evp | aes-arm64 |   n/a
--------------------+---------+-----------+---------+---------+-----------+---------
 fixed,1048576      |       - |         - |       - |       - |         - | 17968.8
 buzhash            |       - |         - |       - |       - |         - |  1505.9
 fastcdc            |  2152.7 |    2247.7 |  3677.3 |       - |         - |       -
 buzhash64          |  1418.6 |    2438.1 |  2407.0 |       - |         - |       -
 toeplitz-aes       |       - |         - |       - |   569.4 |     982.4 |       -
 rabin-aes          |       - |         - |       - |   547.3 |     975.4 |       -
 goldilocks-aes     |       - |         - |       - |   367.9 |     545.0 |       -

AMD Ryzen 5 8500GE: median of 3, MB/s
                    |  scalar | blockwise |    avx2 |  avx512 |     evp |  aes-ni |    vaes |   n/a
--------------------+---------+-----------+---------+---------+---------+---------+---------+---------
 fixed,1048576      |       - |         - |       - |       - |       - |       - |       - | 18918.2
 buzhash            |       - |         - |       - |       - |       - |       - |       - |  1895.4
 fastcdc            |  3776.0 |     826.1 |  2600.0 |  2681.4 |       - |       - |       - |       -
 buzhash64          |  2320.5 |     730.8 |  1433.1 |  1466.8 |       - |       - |       - |       -
 toeplitz-aes       |       - |         - |       - |       - |   229.0 |   790.6 |   915.4 |       -
 rabin-aes          |       - |         - |       - |       - |   224.6 |   779.8 |   895.7 |       -
 goldilocks-aes     |       - |         - |       - |       - |   176.2 |   434.8 |   460.4 |       -

Notable:

• the simple scalar sequential loop wins for fastcdc and buzhash64. for the *-aes chunkers vector AES is a win.
• avx512 on that zen4 is double-pumped (hw is only 256bit, so needs 2 cycles), zen5 is 512bit and should be faster.
• fastcdc will be the new default for borg2
• buzhash is a compatible reimplementation of what we have in borg 1.x (32bit).
• buzhash64 is: buzhash as 64bit + normalized chunking + improved table init

Hashing

I recently added blake3 which will replace blake2b in borg 2 - it is much faster and has internal multithreading (enabled for big chunks).

Apple M3 Pro: borg benchmark cpu --hashing, 1 run
--------------------------------------------------------------
blake3 (64MiB)             1.00 GiB    0.059s   18141.3 MB/s
blake3 (2MiB)              1.00 GiB    0.084s   12852.9 MB/s
hmac-sha256 (2MiB)         1.00 GiB    0.347s    3097.6 MB/s
blake2b-256 (2MiB)         1.00 GiB    1.000s    1073.5 MB/s

AMD Ryzen 5 8500GE: borg benchmark cpu --hashing, 1 run
--------------------------------------------------------------
blake3 (64MiB)             1.00 GiB    0.033s   32601.1 MB/s
blake3 (2MiB)              1.00 GiB    0.032s   33848.1 MB/s
hmac-sha256 (2MiB)         1.00 GiB    0.433s    2480.2 MB/s
blake2b-256 (2MiB)         1.00 GiB    0.915s    1173.2 MB/s

Warning

Don't expect too much after reading these impressive throughput numbers:

• borg also has to read and write the data, limiting the throughput and adding latency
• for small files, there is considerable overhead and only single-threaded hashing/compressing (multi-threaded would be slower for that case)
• borg is mostly sequential internally, except the store thread, that overlaps writing a pack with preparing the next pack.