Opening the compression chapter: where .aerozip stands, and how it gets competitive

[axpnet]

Opening the compression chapter: where .aerozip stands, and how it gets competitive

With the .aerozip lane, AeroVault stops being only an encryption tool and steps into the archiver arena next to zip, 7z, tar.gz and tar.bz2. This thread opens the compression chapter: how the format compresses today, how it compares, what we found, and the roadmap to close the gap. The numbers below are our first public benchmark baseline, reproducible, and meant to be the reference future runs are measured against.

Note

Scope: this is about the .aerozip lane (compression + error-correction, no encryption). The encrypted .aerovault lane is unchanged. All numbers are measured on the standard Silesia corpus (211,938,580 bytes), crate aerovault @ 43a8f86, error-correction OFF, zstd 1.5.5.

TL;DR

• 🟡 Today, the default .aerozip (zstd level 9, per-chunk) compresses Silesia at 3.428x, vs tar.bz2 -9 at 3.887x: we are +13% bytes behind the target.
• 🔴 The existing "archive" profile (zstd 19/22) currently makes it worse, not better (3.31x). We traced it to a real bug (Fig.2).
• 🟢 The achievable target with a per-chunk path done right (level 19) is 3.85x at 4 MiB chunks and 3.96x at 16 MiB, crossing tar.bz2 (3.887x) between 4 and 16 MiB chunks, while keeping deduplication, streaming and error-correction. A trained dictionary reaches it at smaller chunks (keeping finer dedup).
• 🟢 Matching tar.xz (4.35x) is a separate, optional high-ratio lane (LZMA), not the default.

1. The landscape (measured)

Codec	Ratio	Output	Notes
.aerozip L9 (default today)	3.428x	61.83 MB	per-chunk, cold dictionary
.aerozip L19/22 (archive profile)	3.311x	64.02 MB	🔴 regression, see Fig.2
tar.bz2 -9 (the bar to beat)	3.887x	54.52 MB	solid BWT stream
tar.zst -19 (solid)	4.003x	52.95 MB	reference
tar.zst -22 (solid)	4.041x	52.45 MB	reference
tar.xz -9 (solid)	4.346x	48.76 MB	ratio king (LZMA)

The root cause is structural: tar.bz2 compresses the whole concatenated stream (solid), while .aerozip compresses each content-defined chunk independently with a cold dictionary. That independence is what powers per-chunk deduplication, and it is also what costs ratio. This is the central tension of the whole chapter:

ratio vs deduplication granularity. Everything that makes a stream compress better (bigger frames, shared history) erodes the fine-grained dedup that is .aerozip's structural advantage.

2. A surprise: the "archive" profile regresses

Bumping zstd to level 19/22 should help. Measured, it does not: the full corpus gets worse. We localized it to a single file (samba) and a single mechanism.

The archive profile widens chunks to 16 MiB. The incompressible-probe samples only the first 64 KiB of a chunk and, if that prefix looks incompressible, stores the entire chunk raw. On samba that misfires: a chunk that zstd -19 compresses to 3.97 MB (5.44x) is stored raw at 13.60 MB (1.59x), wasting ~9.6 MB.

Reproduce the samba finding (shell)

# samba in 16 MiB blocks, pure zstd-19, NO probe:
cd silesia
split -b 16M samba /tmp/p && tot=0
for f in /tmp/p*; do tot=$((tot + $(zstd -q -19 -c "$f" | wc -c))); done
echo "pure zstd-19 @16MiB = $tot B (5.44x)"   # 3,969,144
# .aerozip L19 stores samba at 13,598,082 B (1.59x) -> ~9.6 MB wasted to raw-storage

Important

This is a real bug, not a tuning preference. The level lever itself is fine: on every other content type level 19 helps (text 3.674x -> 4.297x, binary 2.408x -> 2.672x). The fix is in how the probe and chunk-widening interact (sample a representative window, or compress-then-compare, or cap chunk size).

3. How far per-chunk can go

Per-chunk compression is not a dead end. As chunks grow, a per-chunk frame approaches the solid ceiling; measured, it crosses tar.bz2 between 4 and 16 MiB chunks (4 MiB = 3.849x, just under; 16 MiB = 3.963x, clearly over).

The model

ratio_perchunk(c)  ~=  ratio_solid * ( 1 - w(L) / c )

  c       = chunk size
  w(L)    = per-frame cold-start warmup (the leading bytes that compress against an empty history)
  ratio_solid = whole-stream ratio (the ceiling)

Bigger c approaches ratio_solid but coarsens dedup. A trained dictionary shrinks w(L) WITHOUT growing c, so it raises ratio while keeping fine-grained dedup. That is the recommended lever.

4. Roadmap

Legend: delta vs today = expected size reduction vs the current level-9 default. All format-visible changes get a new opt-in codec id, coordinated to preserve cross-open with the app.

Tier 1: quick wins (within zstd, no new codec)

Change	Delta vs today	vs tar.bz2	Effort	Dedup	Byte-compat
Fix the probe x chunk-widening bug	recover the regression	unblocks Tier 1	S	unaffected	safe
Level 9 -> 19 at sane chunk sizes	-8 to -15%	parity to better	S	unaffected	level is already a manifest field
Trained dictionary per content-class	-10 to -40% on small-file corpora	beats it	M	unaffected (keeps per-chunk)	new opt-in codec id
Explicit windowLog to cover the chunk	+2 to 6%	additive	S	unaffected	format-visible window

Tier 2: optional high-ratio lane (per-archive)

Codec	vs bzip2	Pure-Rust today	Verdict
xz / LZMA2	~10-11% smaller (Silesia)	yes, lzma-rust2 (validate encoder)	best classic high-ratio lane
bsc / BWT	~14% smaller	no (C only)	reference, not shippable as-is
brotli q11	~11% smaller	yes (full encoder)	best pure-Rust high-ratio lane

Tier 3: moonshots

Context-mixing / neural compressors reach the ratio ceiling but are non-viable for a portable, deterministic, interactive vault (10^5 to 10^6 ns/byte, GB of RAM, no cross-hardware determinism). A pure-Rust zstd encoder at C parity does not exist yet. Not on the table.

Full benchmark table + reproduction commands

# corpus
curl -sO http://mattmahoney.net/dc/silesia.zip && unzip -q silesia.zip -d silesia
IN=$(du -sb silesia | cut -f1)   # 211,938,580

# references (solid tar stream)
tar cf - -C silesia . | bzip2 -9 | wc -c      # 54,521,849  -> 3.887x
tar cf - -C silesia . | xz    -9 | wc -c      # 48,763,544  -> 4.346x
tar cf - -C silesia . | zstd -19 | wc -c      # 52,947,262  -> 4.003x

# .aerozip (plaintext lane, EC off) via the crate example examples/bench_ratio.rs
cargo run --release --example bench_ratio -- silesia 9    # 61,831,701 -> 3.428x
cargo run --release --example bench_ratio -- silesia 19   # 64,015,170 -> 3.311x  (regression)

Per-chunk zstd (fixed-block proxy, no probe/dedup)	Ratio
1 MiB blocks, L9	3.400x
1 MiB blocks, L19	3.681x
4 MiB blocks, L19	3.849x
16 MiB blocks, L19	3.963x
solid stream, L19 (ceiling)	4.000x

5. A note on the name (pre-launch)

The format name is still open (it ships next release), and the obvious name has a problem: there is already a Microsoft Store archiver called AeroZip. Naming, the candidates, the availability check, and the "one format, optional password" model are covered in a companion comment below, so this thread can stay technical.

6. What we are asking

This thread is the baseline. Next we will: (1) fix the probe bug, (2) land the level lever decoupled from chunk-widening, (3) prototype a trained-dictionary codec id, and re-run this exact benchmark. Feedback on the priority order, and on whether an optional xz/LZMA lane is wanted, is welcome.

---

_{Methodology: ratio = input / output (unit-free). Sizes in decimal MB. Per-process measurements, deterministic and load-independent. Sources: Silesia corpus (mattmahoney.net/dc/silesia.html); zstd / xz / bzip2 official docs. External ratio figures for Tier 2 carry the usual corpus caveats and are cited in the engineering notes.}

[axpnet]

Naming the format before launch: one format, optional password, and which name is actually free

Companion to the compression chapter above.

One thing up front, with credit where it is due: the name aerozip was @EhudKirsh's proposal, and it is a good instinct, it says in one word exactly what the format does. The only reason we are reopening it is a hard availability finding (below), not a change of heart. So, Ehud, you championed this one and you should have first word on it: read the table, then tell us if any of the free candidates work for you, or float a better one.

The new compression format has not shipped yet (it lands with the next release), so the name is still open. Two facts shape the decision, and one of them is a hard constraint we just verified.

The model: one format, optional password (the 7z model)

It is worth being precise about what this format is: it is the same AeroVault container (the aerovault engine does all the work), and whether a file is encrypted or not is a single header flag, not a different format. That is exactly how zip and 7z already work: one format, with or without a password.

So "compression" is a mode of AeroVault, not a separate product:

• 🟢 With a password: the encrypted vault (what .aerovault is today).
• 🟢 Without a password: an open, shareable, compressed archive (still deduplicated and error-corrected).

This reframes the naming question. We are not necessarily inventing a second format; we are deciding how to present AeroVault's open mode, and whether it deserves its own friendlier extension for sharing.

The hard constraint: most of the obvious names are already taken

Before debating taste, we checked what already exists. This is decisive.

Candidate	Status	What we found
aerozip	🔴 taken	AeroZip is already a real Windows archiver (ZIP/RAR/7Z) on the Microsoft Store by Pero Software; also aero.zip and AirZip exist. Same category. Burned.
aeroz	🔴 taken	Aeroz (aeroz.io), an existing software company.
aerox / aero-x	🔴 taken	Existing aviation / cloud products.
.av (extension)	🔴 taken	Surveillance-camera video and Final Draft scripts; also reads as "Audio/Video". Ambiguous.
.avz (extension)	🟡 taken	AviSys (birding) database backup; plus the AVZ antivirus association. Niche/legacy, but occupied.
aerozx	🟢 free	No matching product found. Reads as a next-generation zip (the docx / xlsx pattern).
aerovz	🟢 free	No matching product found. Carries the AeroVault lineage plus "z".
aeroark	🟢 free	No matching product found. "ark" = archive, no zip baggage.

Important

aerozip is not just "borrowed equity", it is an existing competing product in the exact same category. That alone is enough to retire it before launch.

Recommendation

Two coherent directions, both built on free names:

• 🟢 Arena identity (we are entering the zip / 7z / rar space): name and extension aerozx. It says "compressor", it is modern, and it is free.
• 🟢 Lineage identity (one AeroVault format, optional password like 7z): name and extension aerovz, keeping .aerovault as a backward-compatible extension. It says "this is AeroVault, open mode".

Short two or three letter extensions (.av, .avz) are best avoided: they are taken or ambiguous. If a very short pristine extension is wanted, candidates like .azx / .avx should be collision-checked first.

What we are asking

1. One format with an optional password (the 7z model), yes or no?
2. Arena name (aerozx) or lineage name (aerovz)?
3. Should open files keep .aerovault, or get a friendlier sibling extension for sharing?

Naming is reversible only before launch, so this is the moment. @EhudKirsh, over to you first.

---

_{Availability checked June 2026 via web search. Sources: AeroZip on the Microsoft Store and Pero Software; aero.zip (Capterra); AirZip; the .av and .avz extension registries; Aeroz; Aero-X. Absence of a match is not a trademark clearance; a formal check is advised before locking a name.}

[EhudKirsh]

What if you make .aerozip only a file format, and not a crate, repo or app called AeroZip? So it's just part of AeroVault.
Surely .aerozip as a file format is not taken and would solve this problem.
aero.zip is a URL, and it seems to use .zip, so that's another file format. Air in AirZip is a different word.

As for the other potential file formats you suggested, .aeroark does sound cool, concise and clear enough, because KDE also uses the word Ark in this context. I still prefer .aerozip because Zip as a verb isn't short for a longer word like Archive, as far as I'm aware
(Zipper is a noun, so it's different).
Also, I try to avoid collisions in the first letter between members of the AeroFamily who are unrelated. No member of the AeroFamily starts with a Z, but AeroAgent also starts with an A. Of course, the 4 overlays all begin with a C on purpose, because they relate.

Speaking of which, maybe .aerocompress?
If AeroCorrect can be both the name of an overlay and a file format, why can't AeroCompress?

.aerovz and .aerozx aren't sufficiently clear. Maybe .aerozst. It's short, starts with a Z, and clear. The issue is that by its name,
it would restrict itself to only the zst algorithm, or risk being confusing. I envisioned potentially incorporating multiple compression algorithms to always be on the ideal curve of % VS time. This would be the innovation of this new file format over all others, as all others innovated by introducing their own new algorithm. Your benchmarks will tell us for sure which algorithms it makes sense to use.

At any rate, a file format should only earn the Aero title if it innovates, not just coded in Rust. If it's just .zst, maybe just use that file format like .pkg.tar.zst. If you discover anything new on this topic, we can always come back here and pick a file format.

[axpnet]

@EhudKirsh you talked me around, and the answer lands on your proposal: the format name is .aerozip. Here is the full reasoning, because it is worth putting on record, and then the format's identity and a couple of final notes.

1. The decision

The format ships as .aerozip. The name you proposed from the start is the one we are adopting. AeroVault becomes the first to actually define a .aerozip format, not just an app that reads other archives.

2. Why it holds: product name vs file format

Your key distinction was right, and it gets stronger the closer you look. The trademark question is about a product/brand name in a commercial class, not about a file extension that never ships as a standalone product.

Check	Finding
aero.zip	a file-transfer service, a different category. Not an archiver, not a format. Does not bear on a compression-format name.
"AirZip"	a different word.
AeroZip (Pero Software, MS Store)	the one same-category name. But it is a multi-format extractor (opens zip / 7z / rar / tar and the rest) with no proprietary format of its own.
.aerozip as a file format/extension	unclaimed. Nobody uses it, including the AeroZip app.
Registered trademark "AeroZip"	none surfaced across USPTO / EUIPO / Trademarkia / Justia searches.

So the .aerozip format is open territory. The only residual is a brand-name echo with a same-category app, which is a name question, not a format collision, and it is low. We are defining the format they never had.

How we checked (so the table is not just assertions)

This was a due-diligence pass, not a legal opinion. The steps:

1. Identify the conflicting party. Searched for "AeroZip" and found one same-category product: the AeroZip app by Pero Software on the Microsoft Store. Read its description to confirm what it is: a viewer/extractor that creates and opens 7Z, ZIP, TAR, GZ, BZ2, XZ, WIM, i.e. standard formats, with no format of its own.
2. Separate the look-alikes by category. aero.zip resolves to a file-transfer service, a different product class, not an archiver. "AirZip" is a different word. Neither competes with a compression-format name.
3. Check the extension itself. Searched for an existing .aerozip file format / extension. No dedicated .aerozip format exists, including from the AeroZip app, which uses the standard extensions above.
4. Search the trademark registers. Looked for a registered "AeroZip" mark via USPTO, EUIPO, Trademarkia and Justia. None surfaced. Honest limit: those register databases block automated queries, so this is a search-level result, not a register-certified one. A definitive answer is a five-minute manual lookup on the official registers, which we treat as a nice-to-have before any commercial/brand use, not a blocker for an open-source file format (and publishing the format is a separate, deliberate step).

Conclusion of the analysis: the product name "AeroZip" is occupied in our class, so we never ship a product/app/crate called AeroZip; the .aerozip extension/format is unclaimed and no enforceable mark surfaced, so adopting it as a format name is sound, with a small, named residual (brand echo) that we accept knowingly.

Sources consulted: AeroZip on the Microsoft Store and the Pero Software site; aero.zip (the transfer service); USPTO / EUIPO / Trademarkia / Justia trademark search.

3. The identity: this is the part that earns the "Aero"

You framed it best, and we are adopting it as the format's official positioning: .aerozip is not "a zstd container". It is an adaptive, multi-codec archiver that stays on the ideal ratio/time curve. Where others innovated by adding one new algorithm, this format innovates by always choosing the best codec for the data in front of it. That is what earns the "Aero", not "written in Rust", and it is exactly what the #7 benchmark baseline is here to prove and to drive (the numbers will tell us which codecs are worth carrying). It also means a name tied to one codec (.aerozst) was never right: .aerozip reads as "compress" to everyone and stays codec-neutral.

4. A note of transparency: the internal codename

Inside the crate the lane carries a deliberately neutral codename, aerovz (short for AeroVault-Z, a nod to 7z), in its key-derivation labels. The reason is engineering hygiene: a format freezes those labels forever once published, so we keep the wire-format name neutral while the public name is .aerozip. If the public name ever had to change, nothing stale would be left baked into the format. So .aerozip is the name you see and use; aerovz is just the invisible internal codename that keeps our options open. No effect on the format you interact with.

5. Final considerations

• Credit: the name is yours. We pushed back on "AeroZip the product", you correctly separated that from ".aerozip the format", and that is the version we are shipping.
• Rollout: .aerozip is library-level first (the crate), surfaced as a user-facing format in a later release once the multi-codec selection is tuned against this thread's benchmarks.
• Diligence: the searches above are due diligence, not a legal opinion. Before any commercial or brand-level use we would still do a five-minute manual register check, but for an open-source file format the path is clear.
• Next on the technical side: fix the probe regression (done), level/CDC decoupled (done), then prototype the trained-dictionary codec id and re-run this exact benchmark, so the "choose the best codec" promise becomes measurable here.

Over to you: anything you want to add to the identity framing before we make .aerozip official across the docs?

[EhudKirsh]

I just realised something. Do you remember how you wrote that the ideal chunk sizes are also determined by curves?
So similar to AeroCompress/.aerozip, that can be the innovation of AeroChunk. The sweetspot on a graph.
Though it might have 0 degrees of freedom (1 exact point on a graph, to use Chemical Engineering/Thermodynamics terminology again), whereas with AeroCompress/.aerozip, it will be 1 degree of freedom, meaning anywhere on a curve that the user chooses.
Of course, chunks won't all be that exact ideal byte size, but as close to it as reasonably possible.

[axpnet]

@EhudKirsh that maps cleanly onto how the chunker already works, and the degrees-of-freedom framing is a good way to put it. There are two axes, and we just made them independent.

The ratio side follows the measured per-chunk model from the baseline above. As a first-order model:

$$\mathrm{ratio}(c_{\mathrm{eff}}) \approx R_{\mathrm{solid}}\left(1 - \frac{w(\mathrm{codec}, L)}{c_{\mathrm{eff}}}\right)$$

where c_eff is the effective average chunk size, w(codec, L) is the per-frame cold-start cost for a given codec and level, and R_solid is the whole-stream ceiling. In the implementation, the configured CDC avg is the target used by the Gear boundary rule, not a promise that every chunk has exactly that size.

• Chunk geometry (Gear content-defined chunking: min / target average / max bounds). This is the axis with the sweet spot you describe. Larger target chunks get closer to the solid-stream ratio because there is less cold-start per frame, but they coarsen deduplication; smaller target chunks give finer dedup with more cold-start. So the real target is not a magic byte count, it is an optimal region for the workload. That is close to your "0 degrees of freedom" reading in the user-facing sense: the system should target the region, the user should not have to tune it by hand. And since CDC boundaries fall on content, chunks land near the target, not exactly on it, exactly as you said.
• Compression (codec + level on the ratio/time curve). This is the 1-degree-of-freedom axis: a curve the user moves along, fast versus archive.

The relevant news: until recently those two were coupled (raising the level also widened the chunks, which is exactly what hid the samba probe regression). We just decoupled chunk geometry from the compression level (the with_cdc_bounds change in this cycle), so each axis can now be tuned on its own curve, which is precisely what your framing asks for.

On "AeroChunk": by your own rule (the Aero title is earned by innovating), the thing that would earn a name is adaptive chunk-geometry selection per corpus, the chunk-side twin of adaptive codec selection, where the format measures enough of the workload to pick the best target region instead of shipping one fixed size forever. Whether that gets its own name or stays an internal property of .aerozip is secondary. The #7 benchmarks will tell us how much the chunk axis actually moves the curve and how corpus-dependent the optimum is. Either way it is the same north-star: stay on the ideal curve, on both axes.

Edit note: clarified the formula as a first-order model (\approx), and clarified that CDC avg is the configured Gear target while c_eff is the effective average chunk size.

[axpnet]

Follow-up: T-C1 probe fix measured against the baseline above

This is the first measured update after the baseline in the opening post.

Reading it against the existing figures:

• Fig.1 established the Silesia landscape and the tar.bz2 -9 bar.
• Fig.2 identified the regression: the archive profile could store a large samba chunk raw because the incompressibility probe looked only at the first 64 KiB.
• Fig.3 showed that the per-chunk zstd path can cross bzip2 once the chunk path is behaving correctly.

T-C1 fixes that specific bug. The probe now samples representative windows across large chunks, up to a bounded probe budget, before deciding to skip the full zstd pass and store raw.

Same Silesia corpus, same benchmark example, EC off:

Run	Output	Ratio
L9 baseline from this thread	61,831,701 B	3.428x
L9 after T-C1	61,596,290 B	3.441x
L19 baseline from this thread	64,015,170 B	3.311x
L19 after T-C1	54,491,435 B	3.889x
tar.bz2 -9 local reference	54,521,847 B	3.887x

Measured effect:

Delta	Value
L19 size reduction vs baseline	-9,523,735 B (-14.88%)
L19 ratio gain vs baseline	+17.48%
L19 vs tar.bz2 -9	30,412 B smaller (-0.056%)
L9 size reduction vs baseline	-235,411 B (-0.38%)

One useful control: I also ran a temporary "no probe / always compress then compare" L19 build. It produced the exact same output as the representative-probe fix: 54,491,435 B. So this fix recovers the full ratio available from disabling the probe, while keeping the raw fast path for genuinely high-entropy chunks.

Gates were green locally:

cargo fmt --check --manifest-path aerovault/Cargo.toml
cargo clippy --all-targets --manifest-path aerovault/Cargo.toml -- -D warnings
cargo test --manifest-path aerovault/Cargo.toml

Important caveat: this is a bugfix result, not the final compression tuning pass. It proves the zstd/chunking path can already reach bzip2 parity once the false-raw decision is removed. The next step is the broader tuning matrix: decouple zstd level from chunk profile, test trained dictionaries, validate zstd advanced parameters/window coverage, and compare optional high-ratio lanes against 7z/WinZip-style competitors.

[axpnet]

Compression progress follow-up: from baseline to bzip2-parity, with the tuning boundary mapped

This follow-up summarizes the work after the original compression baseline in Discussion #7.

The short version: the plaintext compression lane is now technically in the file-compressor arena. T-C1 fixed the false-raw regression, T-C2 separated compression level from chunk geometry so we can choose the ratio/dedup tradeoff explicitly, and T-C3 showed that a naive zstd-dictionary path is not worth landing in the real AeroVault pipeline.

1. Development Progress

On Silesia, the important progression is:

Step	Output	Ratio
Baseline L9 default	61,831,701 B	3.428x
Baseline L19 regression	64,015,170 B	3.311x
T-C2 balanced L19	56,421,489 B	3.756x
T-C1 fixed L19 archive	54,491,435 B	3.889x
T-C2 archive L22	54,438,261 B	3.893x
T-C2 wide L22	53,479,885 B	3.963x
T-C2 ratio-lab L22	53,093,113 B	3.992x
tar.bz2 -9	54,521,847 B	3.887x
tar.zst -19 solid	52,947,262 B	4.003x
tar.xz -9 solid	48,763,544 B	4.346x

The meaningful result is not just "we found a larger chunk size". The meaningful result is that the curve is now mapped:

• balanced keeps the original dedup-first posture.
• archive crosses tar.bz2 -9 while keeping bounded per-chunk compression.
• wider ratio-first profiles approach solid zstd, but should remain explicit opt-in modes because they coarsen deduplication.

2. T-C2 Profile Sweep

The best product compromise from this sweep is still the current archive geometry:

1 MiB min / 4 MiB average / 16 MiB max

At zstd level 22 it reaches 3.893x on Silesia, just above bzip2. Increasing only the max from 16 MiB to 32 MiB gives a small extra gain (3.902x), while moving to 8 MiB or 16 MiB average chunks gives stronger ratio but with a more obvious dedup tradeoff.

So the product framing should probably be:

Profile	Purpose
balanced	default, dedup-first, smaller chunks
archive	bzip2-parity/high-ratio default candidate
ratio-first / lab	explicit opt-in, larger chunks, more ratio

3. T-C3 Dictionary Result

We also prototyped a real zstd-dict path inside the AeroVault pipeline:

• opt-in compression wrapper;
• dictionary trained from the first add batch;
• training over large CDC chunks and individual small files before batching;
• dictionary stored in the manifest for the prototype, so overhead was counted;
• round-trip test passed.

The result was negative. Proxy tests looked promising on source-code corpora, but the real container got larger once small-file batching and dictionary storage overhead were included.

That is an important engineering result: the existing pack-before-chunk path already captures a lot of the small-file cold-start win that a naive dictionary would otherwise target. We should not land a format-visible dictionary path unless a more selective/class-specific design beats the current pipeline in real container measurements.

Current Conclusion

The compression lane is now strong enough to enter the file-compressor comparison honestly:

• It beats tar.bz2 -9 in the archive profile on Silesia.
• It stays per-chunk, streamable, deduplicable, and compatible with the AeroVault wrapper-stack model.
• It does not pretend to beat tar.xz; that remains a separate high-ratio lane question.
• It has a clean technical story: bug fixed, tradeoff exposed, naive dictionary rejected by data.

The next serious push, if we want more ratio beyond archive zstd, should be an explicitly separate high-ratio lane or carefully scoped zstd advanced-parameter work, not a naive dictionary feature.

Local Evidence

• tc2-tuning-matrix-2026-06-21.csv
• tc3-zstd-dict-prototype-2026-06-21.csv
• NOTES-2026-06-21-tc2-tuning.md
• NOTES-2026-06-21-tc3-zstd-dict-negative.md

[EhudKirsh]

In axpdev-lab/aeroftp#276 (reply in thread) you wrote that:

The KDF floor is real and deliberate. On the smallest input AeroVault spends about 778 ms to compress against zip-aes at 46 ms, because Argon2id at 128 MiB runs once per container. That is brute-force resistance, not a slow compressor. The clear lever is a cached-key batch mode so the KDF is paid once across many small containers rather than per file.

For this reason alone, it might've been interesting to retest compression without encryption. And so you have.
And then you found this bug, when you removed this encryption variable that should've been a control variable.
Events like these are expected when changing the variables in experiments. I'm glad that my input indirectly led to the discovery of the bug.

[axpnet]

@EhudKirsh exactly, and credit where it is due: your KDF-floor point is what made it worth isolating compression from encryption in the first place. Treating encryption as a control variable and removing it is what put the per-chunk probe under a clean light, and that is where the archive-profile regression showed up. Good experimental instinct. The fix (sampling the probe across the chunk) is already in, and the updated baseline is in this thread. Thanks for the nudge.