Repository navigation
Version 2.5.6 is a new tree format, built for deduplication and for trees of a terabyte and more. The Go, Rust and Swift libraries are released together at this version: from the same folder the three build the same tree, with the same root, and each reads what the others write.
It breaks with 2.4.x. It does not read trees made by 2.4.x, the same folder has a different root than it had there, and the API is new.
What's new
- Deduplication comes first. A chunk's hash covers its bytes alone and a file's hash covers its chunks alone. Names and chunk numbers moved into the parent. Earlier versions hashed a chunk's number into the chunk and a file's name into the file, so the same bytes had different hashes in different places.
- LeafSync numbers a file's chunks, not a tree's leaves. Earlier versions numbered every leaf of a tree. It now asks for a file by its hash and for its chunks by their numbers within that file, which the file's hash covers. Set reconciliation, such as Negentropy, is left with the small sets: the Merkle roots, and the file hashes that link to them.
- A folder has one place for each name, so one branch proves that a name is in a folder, or that it is not.
- Hardened against forged data. A made-up leaf, a file that comes back short, a chunk that belongs to another file, and a store that is tricked into handing back other bytes are each refused, and each has a test.
- Bounded memory and work. Every request says how long its answer may be, a tree's record says how much the tree holds, and writing a tree out counts what it would create before it creates anything.
- Built for terabytes. Building and verifying use every core, a file is never held in memory whole, a damaged chunk is named by its hash and fetched again on its own, and any byte range of a file can be read without the rest of it.
- Safe on disk. A link is never followed, in building or in writing out, even when one is swapped in while the work is under way.
- A new API.
createDag,Dagand the transmission packets of 2.4.x are replaced bybuild,verify,fetch,serveandRemote.
Install
.package(url: "https://github.com/HORNET-Storage/Scionic-MerkleTree-Swift", from: "2.5.6"),The library is the product ScionicMerkleTree. Swift 6.0 or newer, on Linux, macOS, Windows and iOS. It uses swift-crypto on Linux and Android; Apple's systems hash with CryptoKit and Windows with CNG. The library doesn't use Foundation.
How it's checked
- Every push is tested on Linux, macOS and Windows, as debug and as release builds, and again on Swift 6.0. The library is also built for iOS.
- The format is specified in
spec/FORMAT.md. The shared test vectors underspec/hold all three libraries to it byte for byte: whole trees with every byte, branches, requests with their answers, and 76 forged inputs that each library refuses. - The README ends with a benchmark of the three libraries side by side.
Full Changelog: v2.4.3...v2.5.6