AeroVault wrapper-stack and cryptography (design home)

[axpnet]

This is the design home for the AeroVault format on its own repo. The original, long-form design conversation lived on the AeroFTP repo and shaped both the v3 architecture and the spec; this thread carries the settled decisions forward here, where the format now lives. Full history:

• Wrapper-stack and cryptography design conversation: AeroVault Wrapper-Stack and Cryptography: Design Conversation aeroftp#276
• Wrappers/overlays roadmap: [ROADMAP] Wrappers/Overlays aeroftp#272
• Original community roadmap and attribution: [ROADMAP] Community Roadmap: big features for upcoming releases aeroftp#162

The wrapper-versus-step taxonomy, the small-file-packing model, the chunking trade analysis, and the algorithm-versioning clause are a sustained community design contribution by @EhudKirsh. The spec credits this explicitly.

Settled design

Write pipeline (v3 container), canonical order:

logical packing / small-file batching
  -> content-defined chunking (gear-CDC)
       -> keyed BLAKE3 chunk id (128-bit, over plaintext, HKDF-derived key)
  -> zstd compress each chunk / frame
  -> AES-256-GCM-SIV encrypt each compressed chunk
  -> Error Correction (v4, the fourth wrapper, after crypt)

The order is deliberate, not incidental:

• packing concatenates sub-threshold files into shared packs before chunking, so a tree of tiny files still yields multi-MiB chunks; the manifest is the index, the pack carries no per-file framing.
• chunking precedes compression so deduplication, resume, and future range semantics stay chunk-aligned.
• compression is per chunk so a reader can decompress one logical block without inflating the whole archive (this is what buys random per-file extraction; it is the deliberate trade against solid-archive ratio).
• encryption is last among the data-transforming wrappers.
• Error Correction is the fourth wrapper, applied after crypt over the ciphertext, so it never has to decode the format.

Container on-disk layout. All multi-byte integers are little-endian. The data section starts immediately after the header, so a writer can rebuild the header and manifest without shifting the data blocks.

offset 0
┌─────────────────────────────────────┐
│ Header (1024 bytes)                 │
├─────────────────────────────────────┤
│ Encrypted chunk blocks              │
│   [block_len:u64][cipher_block:N]   │
│   ...                               │
├─────────────────────────────────────┤
│ Encrypted manifest                  │
├─────────────────────────────────────┤
│ Extension directory JSON            │
├─────────────────────────────────────┤
│ Extension payloads (v4+)            │
└─────────────────────────────────────┘

Error Correction placement. One unified detached format, .aerocorrect, content-SHA bound, shared byte-for-byte between the app and the standalone aerovault crate (pinned by a cross-implementation fixture). Detached is the default placement, so the encrypted container stays byte-identical to a plain v3. embedded and both are opt-in. Why detached is the default, and the header/manifest chicken-and-egg that makes it the resilient choice, is discussed in #3.

The detached sidecar writes three windows over the container, in a fixed order, so each region's parity is found by position:

segment 0 = header window   [0, HEADER_SIZE)         -> header parity
segment 1 = manifest window [manifest_offset, +len)  -> manifest (locator) parity
segment 2 = data window     [data_offset, +len)      -> data-block parity

This is exactly why detached recovers a damaged header or manifest while embedded cannot: those two regions cannot self-locate their own recovery once corrupted (chicken-and-egg), so they have to travel outside the container.

The .aerocorrect on-disk layout: a small prefix, the locator stored in triplicate with per-copy checksums (so a lightly corrupted sidecar self-heals), and the parity region of concatenated AVEC Reed-Solomon payloads. The content-SHA binding is informational on repair; every rebuilt region is re-verified against the vault's authenticated values (header MAC / manifest cipher_hash) before it replaces the original.

Bit-rot. The corruption error correction guards against is commonly called bit-rot. Because the content is encrypted, the stored bytes have to be exact to decrypt: a single flipped bit in a chunk (a parity-byte failure) would otherwise make that block, and anything chained after it, unrecoverable. Reed-Solomon parity is what repairs those bit-rot failures, on the ciphertext, before anything is decrypted.

Overhead levels. Low / Medium / Quartile / High, or a raw percentage; recorded on the manifest and read back from the AVEC payload, so reconstruction works regardless of the level a vault was created with.

low       ~7%   K=14, P=1
medium    ~15%  K=13, P=2
quartile  ~25%  K=8,  P=2
high      ~30%  K=7,  P=2
numeric   5-50% clamped into the supported range

Cryptography. AES-256-GCM-SIV content AEAD (nonce-misuse-resistant), Argon2id (128 MiB) KDF, AES-256-KW key wrapping, optional ChaCha20-Poly1305 cascade, HKDF-SHA256 key separation, HMAC-SHA512 header MAC. The per-file 16-byte file_id plus chunk index/count is carried as AAD, so a chunk cannot be spliced across files or reordered.

Versioning, three independent axes. Container on-disk major (AEROVAULT2 / AEROVAULT3), container product revision (rev. 3 vs rev. 4 once EC is added; on-disk major stays 3), and the crate semver. A plain v3 reader opens a v4 container and skips the non-critical EC extension.

Spec: AEROVAULT-V3-SPEC.md (pipeline in section 2), AEROCORRECT-SPEC.md, AEROVAULT-V2-SPEC.md. Crate: docs.rs/aerovault · crates.io/aerovault.

Use this thread for format-level design questions going forward.

[axpnet]

A few of the core diagrams from the design conversation (axpdev-lab/aeroftp#276), kept here in the format's home so the picture lives next to the spec.

Write pipeline

The full write path. Files are packed and split by content-defined chunking, each chunk is compressed with zstd, then encrypted with AES-256-GCM-SIV, and error correction is applied last, over the ciphertext. Chunking comes before compression on purpose: it is what keeps deduplication, resume, and random per-file extraction working.

AEROVAULT3 container format

The container on disk: a fixed 1024-byte header, then the encrypted chunk blocks, the encrypted manifest, and the extension directory and payloads that carry the optional v4 error-correction data. The data section starts immediately after the header, so the manifest can grow or be rebuilt without shifting the encrypted blocks.

Where error correction sits

Error correction is the last wrapper, sitting on top of the already-encrypted bytes, so it never has to decode the format. That is what lets one parity format protect a .aerovault, a synced file, or any standalone file, and why the detached sidecar leaves the container byte-identical to a plain v3.

Full set and the original discussion are in axpdev-lab/aeroftp#276.

Parity placement (detached by default)

Where the error-correction parity lives is a choice. By default v4 is just v3 plus a detached .aerocorrect sibling, so the .aerovault stays byte-identical to a plain v3 and the parity sits outside it. embedded (parity as a non-critical extension inside the container, refreshed on every seal) and both (embed plus sibling, two independent recovery locations) are opt-in. This is the placement question under discussion in #3.

[EhudKirsh]

You wrote "Chunking comes before compression", but the diagrams shows the order as the other way around which confused me. It's these diagrams from other posts as well. Please correctly these.

Also, please write somehow that another way to call "parity byte failure" is bit-rot corruption.
Since Crypt is in bytecode and needs to be exact to be able to be decrypted.

[axpnet]

Good catch, and apologies for the confusion, that one is on me: in the rush a couple of diagrams got crossed. You're right, and the text, spec and code all agree on the correct order (chunking before compression: gear-CDC first, then zstd per chunk, spec section 2), so it was the diagram that was stale and showed it reversed. I've regenerated the pipeline diagram with the correct order.

The "container format" image was wrong too: it was actually the AeroCrypt (AECR) object format rather than the AEROVAULT3 container, now replaced with a correct AEROVAULT3 layout. And the "where error correction sits" diagram had the same Compress/Chunk order reversed, so it has been corrected and committed as well.

I checked every public document in this repo: the README, the CHANGELOG, all three specs and the diagrams are now consistent on the order (chunking, then per-chunk zstd). I also corrected the matching pipeline diagram, ASCII and prose on the AeroFTP design thread (axpdev-lab/aeroftp#276) with a dated note.

And I've added the bit-rot framing you asked for: the corruption error correction repairs is bit-rot, and because the content is encrypted the stored bytes have to be exact to decrypt, so a single flipped parity byte would otherwise make that block unrecoverable. That is exactly why crypt runs before error correction.

I'll be more careful with these. Thanks for the precision.

[EhudKirsh]

I found another diagram that was yet to be updated in https://docs.aeroftp.app/security/error-correction.html#where-it-sits-in-the-wrapper-stack.

[EhudKirsh]

I thought of another way of framing the differences in the file formats: .aerozip + .aerovault V2 = .aerovault V3.

[EhudKirsh]

Regarding the banner, please consider rephrasing it as "Military-grade encrypted container" or a similar description.
'aerovault' is singular, and it's not only "for Rust".