Skip to content

Replace full-chunk audit responses with BLAKE3 verified slices#2

Closed
grumbach wants to merge 1 commit into
adr-0005-earned-reward-eligibilityfrom
v2-685-blake3-verified-slice-audit
Closed

Replace full-chunk audit responses with BLAKE3 verified slices#2
grumbach wants to merge 1 commit into
adr-0005-earned-reward-eligibilityfrom
v2-685-blake3-verified-slice-audit

Conversation

@grumbach

Copy link
Copy Markdown
Owner

What

Round 2 of the storage-commitment audit (ADR-0002) made the audited node return the complete original bytes of the nonce-sampled chunks — up to two full 4 MiB chunks per response. Traffic accounting showed this subtree_byte_response was the fleet's second-largest replication bandwidth consumer: ~4.2 TB out + 3.3 TB in per day, ~7.5 TB/day of chunk bytes moved purely to prove storage on a network holding ~18.6 TB.

This replaces the full-byte round 2 with a verified slice: the response for one opened 1 KiB block is a few KB instead of a full chunk (~1000× reduction on the dominant message), while proving more than the old check.

How

A chunk's address is BLAKE3(content), and BLAKE3 is internally a Merkle tree over 1 KiB blocks, so a Bao verified slice proves a block is the real content at a given offset against the address the auditor already knows — content authenticity for free. But the address is public, so authenticity alone is delegable: a node storing nothing could pass by fetching the block on demand from an honest holder (the FetchRequest path serves full chunk bytes with no auth). Possession must therefore be bound separately.

So round 1 additionally commits, per leaf, a fresh nonced block tree: a Merkle root over the same 1 KiB blocks whose leaves are BLAKE3(nonce ‖ peer ‖ key ‖ block_index ‖ block_len ‖ block_bytes). Because the fresh per-audit nonce enters every block leaf, building the correct root requires all of a chunk's bytes under that nonce — and the block the auditor opens is drawn with fresh randomness after the roots are committed (cut-and-choose). Round 2 then verifies both chains over the same block bytes: the Bao slice against the chunk address (authenticity) and the nonced-tree opening against the round-1 root (possession).

A responder that did not hold the bytes at commit time cannot have committed a correct nonced root, and cannot fold an after-the-fact-fetched block to a garbage committed root without a preimage break. Putting the nonce in every block also closes a preprocessing gap in the old flat nonced_hash (where only the first BLAKE3 chunk mixed in the nonce, letting an adversary retain ~1.3 KiB per chunk and recompute the hash for any nonce).

Shape of the change

  • New replication::slice module: nonced block tree (build/open/verify) + Bao slice extract/verify helpers (bao 0.13.1, shares the existing blake3).
  • Round-1 SubtreeLeaf now carries content_len + nonced_root instead of the flat nonced_hash.
  • Round 2 SubtreeByteChallenge/ResponseSubtreeSliceChallenge/Response: opens one fresh-random block per sampled leaf; the auditor verifies both chains. No batching (the reply is KB-scale); MAX_SLICE_OPENINGS caps the sample.
  • REPLICATION_PROTOCOL_ID advances to v3 for a clean cutover (matches the v1→v2 pattern; v2 and v3 nodes simply don't exchange replication traffic during the short auto-upgrade window).

Why not the literal proposal

The ticket's original wording (verify the slice only against the chunk address, with a nonce-derived offset) is unsound: it drops the possession binding entirely, so a storage-less relay farms audit passes — and those passes feed ADR-0005 reward eligibility. The offset also must be fresh post-proof randomness, not nonce-derived, or the cut-and-choose collapses. This PR implements the corrected dual-chain design.

Testing

  • replication::slice unit tests: Bao root equals the BLAKE3 address across lengths, slice roundtrip for every block, tamper / wrong-address / wrong-block rejection, nonced-tree binding to nonce/peer/key, relay-cannot-open-against-a-foreign-root.
  • poc_commitment_audit_attacks (ported to the slice model): relay-holds-addresses, predict-and-fetch under fresh sampling, fabricated-fraction, deleter, plus the substitution / replay / structural attacks.
  • poc_audit_handler_live: the production responder against real LMDB storage, including a new end-to-end test that opens a deep block (index 50) of a ~100 KB multi-block chunk and verifies both chains.
  • Full cfd clean (clippy -D warnings, fmt, cargo doc --deny=warnings); all replication lib tests + both PoC suites green.

Stacked on adr-0005-earned-reward-eligibility (audit passes feed the ADR-0005 tally); rebase onto main once that lands.

Round 2 of the storage-commitment audit returned the complete original bytes of the sampled chunks (up to two 4 MiB chunks per response), making proof-of-storage the fleet's second-largest replication bandwidth consumer.

This replaces that with a two-chain verified slice. Round-1 leaves now carry a content length and a fresh nonced block-tree root (a Merkle root over the chunk's 1 KiB blocks, each block leaf binding the audit nonce, peer, key and block bytes) instead of a flat nonced hash. Round 2 opens one freshly-random block per sampled leaf and returns a Bao verified slice plus a nonced block-tree opening; the auditor verifies each block against the chunk address (authenticity) and the round-1 root (possession) over the same bytes. The response drops from megabytes to a few KB.

Verifying a slice only against the public chunk address would let a node that stores nothing pass by fetching the block on demand, so possession is bound by the nonced root: building it requires all of a chunk's bytes under the fresh nonce, and the block index is drawn after the proof is committed (cut-and-choose). Putting the nonce in every block also closes the preprocessing gap the flat nonced hash left open.

The replication protocol id advances to v3 for the clean cutover.
@grumbach

Copy link
Copy Markdown
Owner Author

Opened against the fork by mistake; the correct PR targets upstream: WithAutonomi#181.

@grumbach grumbach closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant