Error Correction default placement: embedded vs detached

[EhudKirsh]

Maybe it would actually make sense for the default place for Error Correction in AeroVault V4 to be as embedded instead of detached, because it's part of the philosophy of AeroVault that it's a single file.
So going against this philosophy should be opt-in.

[axpnet]

Good thought, and it's worth opening properly rather than me just picking. The single-file philosophy is real, but embedded and detached are not symmetric, they protect different things and cost different things, so the "default" choice has consequences beyond ergonomics. Let me lay the three paths out so we can decide on the full picture.

The asymmetry in one line: the embedded extension is located through the MAC-verified header, so if the header or manifest is the thing that rots, you can't find the embedded parity (chicken-and-egg). Only the detached sidecar, which sits outside and is content-SHA-bound, can rebuild a damaged header/manifest on open. So embedded effectively protects the data window only.

The three paths

Dimension	detached (current default)	embedded	both
Data-window recovery	yes	yes	yes
Header & manifest recovery	yes	no *	yes
Survives losing/forgetting the sibling	no recovery left	yes (self-contained)	data recovery survives, header/manifest lost
Single-file ergonomics	weakest (footgun on copy)	strongest	medium (sibling still exists)
Byte-identical to a plain v3	yes	no (carries non-critical ext, still v3-openable)	no
Parity storage cost	~P/K, once, in the sibling	~P/K, once, inside the file	~2x (two copies)
Compression	container unaffected; parity is incompressible and lives outside	container grows by the (incompressible) parity	both
Write / seal cost	decoupled: refresh on demand, no container rewrite	recomputed on every seal (write amplification)	both costs
Add-later / refresh without rewriting the vault	yes	no (needs a re-seal)	partial
Sidecar self-healing (v2 triplicate locator)	yes	n/a	yes

* the embedded extension is located via the MAC-verified header; if the header itself is damaged the parity can't be found.

Two notes that the table compresses:

• Compression: zstd runs per chunk inside the container, before EC, so placement doesn't change container compression. Reed-Solomon parity is the last wrapper, computed over ciphertext, so it's high-entropy and effectively incompressible. For the same K/P grid the parity is the same size no matter where it lives, placement only decides where the bytes sit (inside the one file, in a sibling, or duplicated).
• Performance: embedded is recomputed on every seal, so every mutation pays the RS encode and rewrites the parity region inside the container (write amplification that scales with vault size). Detached is decoupled: generate or refresh the sidecar on demand by reading the container, no rewrite. Reads/opens are unaffected in all three (parity is only touched on repair).

What each path looks like, and how it fails

1. detached (current default) -- max resilience, footgun on copy

secret.aerovault                    secret.aerovault.aerocorrect
+-----------------------+           +---------------------------+
| header                |           | seg0  header parity       |
| manifest              |  ──────►  | seg1  manifest parity     |
| data                  |           | seg2  data parity         |
+-----------------------+           +---------------------------+
byte-identical to a plain v3        content-SHA bound, self-healing

header rots  -> rebuilt from sidecar      sibling lost -> NO recovery left
data rots    -> rebuilt from sidecar

2. embedded -- truly one file, but blind to header/manifest damage

secret.aerovault  (one file, no sibling)
+-----------------------------------+
| header            ◄── locator     |  if THIS rots you can't find
| manifest                          |  the extension below
| data                              |
| ext: data parity                  |  protects the data window only
+-----------------------------------+

header/manifest rots -> UNRECOVERABLE (vault won't open)
data rots            -> rebuilt from embedded ext
sibling lost         -> N/A, nothing to lose

3. both -- self-contained for data, sidecar still covers the metadata

secret.aerovault                    secret.aerovault.aerocorrect
+-----------------------------------+   +------------------------+
| header                            |   | seg0  header parity    |
| manifest                          |   | seg1  manifest parity  |
| data                              |   | seg2  data parity      |
| ext: data parity (embedded)       |   +------------------------+
+-----------------------------------+
self-contained for data             metadata still covered, ~2x overhead

header rots  -> rebuilt from sidecar
data rots    -> rebuilt from embedded OR sidecar
sibling lost -> data still recoverable, header/manifest no longer

Where that leaves the default

The honest read is that "single file" and "survives header/manifest corruption" pull against each other, and embedded as the sole default quietly drops the protection for the failure that makes a vault completely unopenable. both is the placement that satisfies your philosophy point without that regression, at the cost of roughly double the parity overhead.

So the real question for the V4 default is probably detached vs both, not detached vs embedded. What weighs more for the default user in your view: keeping it one self-contained file, or never paying double overhead? And is the copy-the-vault-lose-the-sidecar footgun something we'd rather solve at the UX layer (warn/bundle on move) than by changing the format default?

[EhudKirsh]

I didn't realise the header/manifest damage difference between embedded and detached.
Not sure if this can be solved one day in the future.
So I'm also not sure what the default should be. But I thought to raise this as food for thought.

[axpnet]

To make the three options concrete while we weigh the default, here is how each placement actually looks on disk:

• detached (today's default): parity lives in a sibling .aerocorrect, the .aerovault stays byte-identical to a plain v3.
• embedded: parity is a non-critical extension inside the container, refreshed on every seal; the file grows a little.
• both: embed and also write the sibling, two independent recovery locations, detached-first.

Sharing it as a visual aid for the discussion, it does not pre-empt the decision.