Why not BLAKE3?

[mernen]

Hi, I've recently learned about this project, and took interest in its architecture. One thing that surprised me, though, is its extensive use of BLAKE2b. Why was BLAKE3 not chosen for such a new tool starting fresh?

In fact, the included architecture review skill even mentions BLAKE3, with this excerpt that summarizes its main advantage quite well:

BLAKE3 is the recommended hash for new backup tools: 6.4 GB/s single-threaded (AVX2), up to 92 GB/s multithreaded on 16 cores. Its built-in Merkle tree structure enables parallelism at the algorithm level. Use the rayon feature for update_rayon() and the mmap feature for memory-mapped file hashing.

[m3nu]

Fair question, and a fair catch on our own docs — the architecture rationale we wrote only compares BLAKE2b to SHA-256, never to BLAKE3, so it does not actually answer you.

The honest version: the one load-bearing use of the hash is the chunk ID, which is a keyed MAC (keyed BLAKE2b-256 with a per-repo secret chunk_id_key). The keying is the part that matters to us — it is what prevents a content-confirmation attack, i.e. an adversary with access to the repo cannot test whether a known file is in your backup without the key. That is the same dedup-privacy model borg and restic use, and BLAKE2b's native keyed mode is the conservative, well-trodden way to get it. That lineage is really why BLAKE2b is there — not a technical constraint.

And to be clear, it is not a constraint. BLAKE3 has a first-class keyed mode (blake3::keyed_hash), and we do not rely on any BLAKE2b-specific feature anywhere — no personalization, no salt, no parameter-block tricks (we do domain separation via AEAD AAD instead). So BLAKE3 would be a drop-in for every place we hash.

Why we have not reached for it:

• Hashing is not our bottleneck. Each chunk is hashed once and then zstd-compressed and AEAD-encrypted; compression and encryption dominate the per-chunk cost. We also already parallelize hashing across chunks with rayon, so BLAKE3's headline multithreaded numbers (the 92 GB/s tree-hashing figure) largely describe something we already get at the chunk level. The relevant comparison is single-threaded per-chunk, where BLAKE3 is faster but the end-to-end backup win is modest.
• Switching is a repo-format break: chunk IDs, pack IDs, the upload-integrity header, the TOFU fingerprint, and the on-disk dedup cache all change. That is much cheaper to do now (pre-1.0) than later, so it is a legitimate thing to reconsider before we stabilize the format — just not something with a payoff big enough that we did it reflexively.

Short version: BLAKE2b was the safe, familiar keyed-MAC choice, not a requirement. BLAKE3 is a reasonable alternative and we are open to revisiting it while the format is still fluid.

[mernen]

Hashing is not our bottleneck. […]

That's a good point; I can see how BLAKE3 would give minimal gains when uploading. I do wonder, though, if it still couldn't provide a meaningful advantage when re-checking existing files? I.e. when a large file had minimal or no changes, so you're for the most part only doing chunking and hashing. Chunking is certainly the bottleneck, but if you have a map of predicted chunk sizes and they all hit, all you have left is reading files and hashing. (This is just armchair reasoning, in practice it could still be barely noticeable, for example if most of the time you have enough files to fill all workers)

Switching is a repo-format break

That's totally fair, I wouldn't even suggest switching to an established solution like, say, Borg. But as Vykar began this year with a different repository format, it seemed like the perfect opportunity to start right off with BLAKE3.

[m3nu]

You're right on the re-check path — and it's worth spelling out what vykar actually does there, because it cuts both ways.

For files that are fully unchanged, the file cache already does what you describe: each entry stores the file's metadata (device, inode, mtime, ctime, size) plus its chunk-ID list, and on an exact match the file is never opened — no read, no chunking, no hashing, just refcount bumps. So the common incremental case would see no benefit from a faster hash.

But for files whose metadata changed while the content mostly didn't — VM images, databases, mail files, anything touched — the cache misses and every byte is re-read, chunked, and keyed-hashed. Chunks that dedup out skip compression, encryption, and upload entirely (just an in-memory index lookup), so on that path the per-byte CPU cost really is chunking + hashing, exactly as you reasoned. Same story for vykar check --verify-data, which recomputes every chunk's keyed hash. One correction to my earlier comment: parallelism comes from a worker pipeline (files, and 64 MiB segments of large files, processed in parallel), not rayon — and within a segment hashing is single-threaded, so per-core hash throughput is a real ceiling there.

I measured it to keep this honest: single-thread over 1 GiB in 2 MiB chunks, keyed BLAKE2b-256 (the RustCrypto blake2 crate we use) does ~1.4 GB/s vs ~2.15 GB/s for blake3::keyed_hash — about 1.5× on Apple Silicon. The gap should be considerably larger on x86-64, where blake3 has AVX2/AVX-512 kernels while RustCrypto's blake2 is portable-only (~1 GB/s) — and those are exactly the NAS/server boxes where CPU is scarcest.

So: conceded. The rehash-heavy paths are a legitimate argument for BLAKE3, and your second point stands too — the format is new and still pre-1.0, so if there's ever a time, it's now. I've opened #177 to benchmark those paths end-to-end and make the call before the format stabilizes.