Replace full-chunk audit responses with BLAKE3 verified slices#2
Closed
grumbach wants to merge 1 commit into
Closed
Replace full-chunk audit responses with BLAKE3 verified slices#2grumbach wants to merge 1 commit into
grumbach wants to merge 1 commit into
Conversation
Round 2 of the storage-commitment audit returned the complete original bytes of the sampled chunks (up to two 4 MiB chunks per response), making proof-of-storage the fleet's second-largest replication bandwidth consumer. This replaces that with a two-chain verified slice. Round-1 leaves now carry a content length and a fresh nonced block-tree root (a Merkle root over the chunk's 1 KiB blocks, each block leaf binding the audit nonce, peer, key and block bytes) instead of a flat nonced hash. Round 2 opens one freshly-random block per sampled leaf and returns a Bao verified slice plus a nonced block-tree opening; the auditor verifies each block against the chunk address (authenticity) and the round-1 root (possession) over the same bytes. The response drops from megabytes to a few KB. Verifying a slice only against the public chunk address would let a node that stores nothing pass by fetching the block on demand, so possession is bound by the nonced root: building it requires all of a chunk's bytes under the fresh nonce, and the block index is drawn after the proof is committed (cut-and-choose). Putting the nonce in every block also closes the preprocessing gap the flat nonced hash left open. The replication protocol id advances to v3 for the clean cutover.
Owner
Author
|
Opened against the fork by mistake; the correct PR targets upstream: WithAutonomi#181. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Round 2 of the storage-commitment audit (ADR-0002) made the audited node return the complete original bytes of the nonce-sampled chunks — up to two full 4 MiB chunks per response. Traffic accounting showed this
subtree_byte_responsewas the fleet's second-largest replication bandwidth consumer: ~4.2 TB out + 3.3 TB in per day, ~7.5 TB/day of chunk bytes moved purely to prove storage on a network holding ~18.6 TB.This replaces the full-byte round 2 with a verified slice: the response for one opened 1 KiB block is a few KB instead of a full chunk (~1000× reduction on the dominant message), while proving more than the old check.
How
A chunk's address is
BLAKE3(content), and BLAKE3 is internally a Merkle tree over 1 KiB blocks, so a Bao verified slice proves a block is the real content at a given offset against the address the auditor already knows — content authenticity for free. But the address is public, so authenticity alone is delegable: a node storing nothing could pass by fetching the block on demand from an honest holder (theFetchRequestpath serves full chunk bytes with no auth). Possession must therefore be bound separately.So round 1 additionally commits, per leaf, a fresh nonced block tree: a Merkle root over the same 1 KiB blocks whose leaves are
BLAKE3(nonce ‖ peer ‖ key ‖ block_index ‖ block_len ‖ block_bytes). Because the fresh per-audit nonce enters every block leaf, building the correct root requires all of a chunk's bytes under that nonce — and the block the auditor opens is drawn with fresh randomness after the roots are committed (cut-and-choose). Round 2 then verifies both chains over the same block bytes: the Bao slice against the chunk address (authenticity) and the nonced-tree opening against the round-1 root (possession).A responder that did not hold the bytes at commit time cannot have committed a correct nonced root, and cannot fold an after-the-fact-fetched block to a garbage committed root without a preimage break. Putting the nonce in every block also closes a preprocessing gap in the old flat
nonced_hash(where only the first BLAKE3 chunk mixed in the nonce, letting an adversary retain ~1.3 KiB per chunk and recompute the hash for any nonce).Shape of the change
replication::slicemodule: nonced block tree (build/open/verify) + Bao slice extract/verify helpers (bao0.13.1, shares the existingblake3).SubtreeLeafnow carriescontent_len+nonced_rootinstead of the flatnonced_hash.SubtreeByteChallenge/Response→SubtreeSliceChallenge/Response: opens one fresh-random block per sampled leaf; the auditor verifies both chains. No batching (the reply is KB-scale);MAX_SLICE_OPENINGScaps the sample.REPLICATION_PROTOCOL_IDadvances tov3for a clean cutover (matches the v1→v2 pattern; v2 and v3 nodes simply don't exchange replication traffic during the short auto-upgrade window).Why not the literal proposal
The ticket's original wording (verify the slice only against the chunk address, with a nonce-derived offset) is unsound: it drops the possession binding entirely, so a storage-less relay farms audit passes — and those passes feed ADR-0005 reward eligibility. The offset also must be fresh post-proof randomness, not nonce-derived, or the cut-and-choose collapses. This PR implements the corrected dual-chain design.
Testing
replication::sliceunit tests: Bao root equals the BLAKE3 address across lengths, slice roundtrip for every block, tamper / wrong-address / wrong-block rejection, nonced-tree binding to nonce/peer/key, relay-cannot-open-against-a-foreign-root.poc_commitment_audit_attacks(ported to the slice model): relay-holds-addresses, predict-and-fetch under fresh sampling, fabricated-fraction, deleter, plus the substitution / replay / structural attacks.poc_audit_handler_live: the production responder against real LMDB storage, including a new end-to-end test that opens a deep block (index 50) of a ~100 KB multi-block chunk and verifies both chains.cfdclean (clippy-D warnings, fmt,cargo doc --deny=warnings); all replication lib tests + both PoC suites green.Stacked on
adr-0005-earned-reward-eligibility(audit passes feed the ADR-0005 tally); rebase onto main once that lands.