Testing AeroVault V3

[EhudKirsh]

I tested AeroVault V3 Beta in AeroFTP v4.0.3. Here is my feedback:

Compression tests I made:

The numbers below are the final compressed byte sizes (the smaller the better),
the percentage that was compressed (the higher the better) and the time elapsed (the lower the better):

Files	Fast (zstd -3)	Balanced (zstd -9)	Archive (zstd -19)
🗒 3 .txt notes totalling 1.14MB	439kB, 62%, <1s	408kB, 65%, <1s	367kB, 69%, <1s
📄 125 .pdf academic articles totalling 534MB	473MB, 11%, 9s	471MB, 12%, 10s	13%, 462MB, 1 min & 39s
🖼️ 363 .jpg images totalling 73MB	69MB, 6%, 2s		68MB, 7%, 15s
🎵 138 .mp3 music files totalling 1GB	0.98GB, 2%, 12s	0.98GB, 2%, 20s	1GB, 0%, 3 mins & 3s
🎞️ 27 of .mp4 & .mkv videos totalling 4.8GB	4.7GB, 2%, 48s	4.7GB, 2%, 1 min 48s	4.7GB, 2%, 17 mins & 19s
16 .sh3d (Sweet Home 3D) files totalling 5.05GB	1.46GB, 71.1%, 51s	1.29GB, 74.5%, 2mins & 20s	1.07GB, 78.8%, 35mins & 58s
6 .3mf files totalling 119MB	118MB, 1%, 2s		117MB, 2%, 21s
A 150kB AllOnes.bin file and a 150kB AllZeros.bin file	2kB, 99%, <1s
150kB AllRandomNumbers.bin file	152kB, 0%, <1s
📁 1.55GB unencrypted .zip	1.55GB, 0%, 14s	1.55GB, 0%, 18s	1.55GB, 0%, 5.4 mins
🛡🔒 .aerovault itself! Trying to compress an encrypted file just to see what happens. This is a .aerovault V2 of the 3 1.14MB .txt files, to test if encrypting without first compressing leaves room for compression afterwards.	1.14MB, 0%, <1s

It seems that .aerovault incurs a 2kB minimum byte size. So even when the receipt says that the all Ones and Zeros files can be compressed to under 100B, the resulting file is around 2kB. Also, the .aerovault of files with random numbers adds 2kB over the inputs. This is just my observation.

I tried to compress over 16GB of music files, but I ran out of RAM at AeroFTP using 8GB, but it was nice to try.
I ended AeroFTP with task manager and deleted the 2 temporary files (56B .aerovault.lock and 8GB .aerovault-v3-nSTAnw).
It's probably worthwhile to write the a warning to not try to enter too much byte size to one .aerovault file.

These files weren't optimised, such as by using apps that compress images.

I rounded to whole numbers for simplicity.

I also tested V2 Standard, Advanced and Paranoid modes, and there's no noticeable (or expected) compression.

I also tested that the b3sum of decrypted and extracted files is identical to what it was before compressing and encrypting them.

Here's how I created the all 1s, all 0s and all random numbers binary files using NodeJS:

const{writeFileSync}=require('fs')
let size=1024*150 // Adjust the byte size as you wish.

writeFileSync('AllZeros.bin',Buffer.alloc(size))
writeFileSync('AllOnes.bin',Buffer.alloc(size,255))
writeFileSync('AllRandomNumbers.bin',require('crypto').randomBytes(size))

This script saves those files to the folder in which the terminal is open in. You can then view these binaries in an IDE. I use VSCodium.
Careful not to make files too large, or the IDE will crash.

Stages note:

I asked AI, and you might find it interesting to ask AI and verify for yourself, what comes first, chunking or compression, in .zip, .rar, Restic and Kopia. It seems like .zip is the exception, and it compresses files individually before grouping small files together. So for the overlays/wrappers, it indeed seems to be that chunking comes first, but for .aerovault, you might want to look into the reasoning that .zip has to compress first. Maybe it's just old and inefficient, or maybe it can help shape part of the .aerovault design. It's a trade-off of: do you want maximum compression or maximum speed of compressing and extracting individual files. Just to clear the confusion, it seems that in all of these, batching small files and splitting large files is a step in chunking, not compression like I thought before.
This paragraph should be cleared up before releasing AeroVault V3, and the overlays/wrappers in AeroFTP after it.
It'd be nice to see definitive sequence like:

1. Chunking
   i. Batching small files and splitting larges files
   ii. Hashing
   iii. Deduplication
2. Compression
3. Encryption
4. Error Correction (V4)

Where pretty much the only main thing that I can see changing is putting compression before chunking,
while keeping the steps inside the same. Actually, I'm not sure.

Usability (General, not specific to V3):

• I tried to drag & drop files from File Explorer to the page that allows me to add files, and it didn't let me.
  It would be nice to have a drag & drop functionality for AeroVault, similar to the custom icons library.
  While on this point, AeroFile can also benefit from a drag & drop feature from File Explorer.
• Further on the previous point: even when I browse for files to add to AeroVault, I can't select folders.
  Please allow selecting multiple files and folders at the same time to give a regular zipping experience.
• There's a >_ Technical Receipt page with 📥 Export .txt and 📥 Export .json buttons,
  but unfortunately they don't say there they're exporting to the Downloads folder.
  It's be nice to be able to decide where to save these to, and get some sort of feedback that the files are successfully exported.
• We're already touched on this, but I'd like to ask you directly: are you sure you don't want to give an option to use AeroVault without encryption? Some users might want it just for the error correction (V4) and compression aspects. Especially when you use the word Archive in the context of zstd -19, as archives aren't always encrypted. If you don't want users to fall into the trap of not setting a password or setting a weak password without realising, let me suggest this: consider making a new .aerozip file format. It's basically like AeroFTP V3 & V4, but without encryption. I was even thinking of suggesting Encryption(.aerozip) = .aerovault, but this might be problematic since Error Correction is done after encryption and I'd like both file formats to be able to support it.
• When decrypting .aerovault, I see that it's possible to extract files individually. Please add an option to extract everything to some folder path. Is the only way currently to extract and save hundreds of files from .aerovault in the AeroFTP GUI at the moment by extracting each by manually clicking on each? There has to be a button to extract all the files and subfolders at the same time.
• When importing .aerovault, to the right of the '🗝 Change Password' button, please add a button to change the mode.
  Maybe also show the 📥 Export .txt and 📥 Export .json buttons there as well.
• Please consider adding the word 'TIME' before 'ELAPSED' to make it clear.
  Also, rephrase ms_total to time_elapsed in the .txt and .json receipts, and measure it in seconds, even if with a decimal place.

[EhudKirsh]

I added a couple of more rows to the table. I was curious to find how much various 3D object file formats can be compressed.
I think part of why .sh3d are so highly compressible is because I'm compressing slightly different variants of the same few designs,
so some of these files are nearly duplicates of each other.

By the way, as you can see, some of these tests took a long time. Can you please implement a progress bar and write the percentage of the progress, both in the GUI and CLI? Thanks.

Afterthought:
The tiny amount of compression of .jpg and .mp3 isn't worth it. Converting .jpg to .jxl can be done losslessly and save much more on byte size. Similarly with using .opus instead of .mp3 (although, it's lossy). So compression is simply not worth it for these.
What I haven't tested is trying to compress massive databases. I'm guessing that these will be the main applications,
along with notes and articles, where zst compression will be useful for AeroVault and AeroSync.

[axpnet]

Thank you for this, the depth of the test pass is exactly what shaped v4. As promised in axpdev-lab/aeroftp#276, here are the specifics, point by point. Going forward AeroVault format and feature discussion lives on this repo; the settled design is summarized in the new Design Threads home (#4), and the long-form history stays in axpdev-lab/aeroftp#276 / #272.

Tags: ✅ shipped · 🔭 follow-up (on record) · 🆕 new answer.

Compression tests

• Already-compressed inputs (jpg/mp3/mp4/pdf/zip) gaining ~0-2%. ✅ Agreed, and acted on: ZIP now decides per entry and stores a file as-is when a deflate pass would not shrink it, instead of spending time to grow it (shipped, see AeroVault Wrapper-Stack and Cryptography: Design Conversation aeroftp#276). The benchmark conclusion is the same one you reached: on near-random bytes every format sits near 100%, and the small delta over 100% is the honest cost of framing.
• The ~2kB minimum floor. 🆕 That floor is the fixed framing + crypto cost of a container, constant regardless of payload: the header, the Argon2id-wrapped key material, the manifest, the per-chunk AEAD tags, and the HMAC-SHA512 header MAC. On a tiny or fully incompressible input there is nothing to compress against it, so it shows up as a ~2kB floor (and as the ~2kB you saw added over the random file). It is not a bug, it is the cost of an authenticated encrypted container; it just stops being visible once the payload is non-trivial and compressible.

Out of memory on 16GB + leftover temp files

• Pre-flight memory guard so a very large input warns instead of OOMing: ✅ shipped (AeroVault Wrapper-Stack and Cryptography: Design Conversation aeroftp#276).
• True streaming vault I/O (so peak memory is bounded by a window, not the file): 🔭 on record as a follow-up. The detached .aerocorrect path is already windowed; the vault content path is the remaining piece.
• The leftover .aerovault.lock (56B) and .aerovault-v3-XXXX (8GB) after you killed it: 🆕 that is a real gap, thank you for the exact filenames. The temp + lock should be cleaned on abort, not just on the success path. Tracking it as a cleanup-on-abort fix.

Stages note (pipeline ordering)

Your proposed sequence is exactly the AeroVault v3 design, and it is settled and shipped:

1. packing / small-file batching
2. content-defined chunking  (+ keyed BLAKE3 id, dedup)
3. zstd compression (per chunk)
4. encryption (AES-256-GCM-SIV)
5. Error Correction (v4)

So chunking comes first and compression is per chunk, for the reasons in the spec (section 2): chunking before compression keeps dedup, resume, and range semantics chunk-aligned, and per-chunk compression lets a reader inflate one logical block without unpacking the whole archive. That is the deliberate trade: we keep random per-file extraction and chunk-level dedup, at some ratio cost versus a solid archive.

The .zip "compress each file first" model you flagged is the solid-vs-non-solid trade. Compressing before chunking would widen the window and squeeze a bit more ratio, but it breaks chunk-aligned dedup, resume, and per-file random access, which matter more for a backup/sync container. So we deliberately do not do it. You had it right in your own sequence; the "maybe compress first" was the only part to set aside. For the record, the wrapper-stack taxonomy and the chunking trade analysis here are credited to you in the spec and in axpdev-lab/aeroftp#162.

Usability

• Drag & drop from File Explorer onto the create screen (and AeroFile): 🔭 follow-up, on record.
• Selecting folders, and multiple files+folders together: ✅ a first-class add-folder picker shipped. If the mixed multi-select (files and folders in one pick) still feels short of a regular zipping experience, tell me and I will treat that as a separate refinement.
• Technical Receipt export location + feedback: ✅ a save dialog shipped, so you choose where the .txt / .json go and get confirmation, instead of a silent drop into Downloads.
• Unencrypted option / .aerozip: 🔭 follow-up, on record. Two notes: AeroVault has no unencrypted mode by design (the whole point is the authenticated encrypted container), so an unencrypted lane would be a separate format, which is exactly your .aerozip framing. Meanwhile, if the motivation is error-correction-only, aerovault correct gen already produces an .aerocorrect for any file with no encryption involved, today. The "compression + EC without encryption container" is the part that needs .aerozip, and your point about EC sitting after encryption is the real design knot to untie there.
• Extract-all to a folder: ✅ shipped, you no longer click each file.
• Change-mode button on the import view (next to Change Password), and the export buttons there too: 🔭 the change-mode action is on record as a follow-up; surfacing the export buttons on that view I will fold into the same change.
• Labels: ✅ the visible wording is now "Time elapsed". 🆕 the receipt field rename (ms_total -> time_elapsed, in seconds with a decimal) is a clean, low-risk change and I will make the receipts match the wording.

Follow-up comment

• Progress bar + percentage in GUI and CLI for long operations: 🆕 reasonable, and your .sh3d / video timings make the case. Tracking it; the CLI and GUI should both show real progress on compress/extract/correct.
• jpg -> jxl, mp3 -> opus, and databases/prose as the real wins: ✅ agreed on the analysis. We reached the same conclusion: zstd pays off on structured text and prose, and barely moves already-compressed media. Transcoding (jxl/opus) is a different feature class (it is re-encoding, not packing) and out of scope for the container itself for now, but the corpus insight is exactly right and is why we are re-evaluating the default codec presets rather than promising ratio on media.

Thanks again. A lot of v4 is the way it is because of this report.

[EhudKirsh]

Meanwhile, if the motivation is error-correction-only, aerovault correct gen already produces an .aerocorrect for any file with no encryption involved, today.

I only came up with the idea of being able to generate a .aerocorrect file to any file after writing this. Although, this begs the question: Can EC be embedded at the end of any file? If not, then .aerovault, .aerozip and whatever file format AeroCrypt chunks have can be designed especially for the embedding of EC.

Regarding .aerozip, you can frame it as: .aerozip = .aerovault V3 - AeroCrypt.
Similar to .aerovault V4 = .aerovault V3 + .aerocorrect. Something like that.

Follow-up comment

For the record, I'm not sure if it makes sense to convert .mp3 to .opus, because that's lossy. It might be better to redownload music and record new audio in .opus to begin with, and simply keep some existing .mp3 where no equivalent of them as .opus can be found.

On the other hand, converting .jpg to jxl can be lossless, and I have ideas on that.
Including: perhaps AeroFile can be a GUI for libjxl. That will be cool and useful. More on that elsewhere.

[axpnet]

@EhudKirsh good set of design questions. A quick technical read on each, and since they straddle two repos I have split them to the right home so nothing gets lost: the format-level ones stay here on aerovault, the AeroFile/app ones go to the AeroFTP tracker.

Format level (here, aerovault):

On embedding error correction at the end of any file: appending parity to the tail of an arbitrary file is possible, but the file stops being a valid file of its original type unless the reader knows to strip that trailing region, so a foreign .mp3 or .pdf with recovery bytes glued on would no longer open in a normal player or viewer. That is exactly why the standalone path uses a detached .aerocorrect sidecar: the original stays byte-identical and valid, and the recovery data lives beside it. For formats we define ourselves the picture is different, because our own reader already knows the layout: a container we control (.aerovault, a future .aerozip, the AeroCrypt chunk format) can reserve a trailing error-correction region by design. So the honest split is embedded EC for formats we control, detached sidecar for arbitrary foreign files. This is the open EC-placement design question in the rolling tracker (#5) and the format design home (#4).

On .aerozip = .aerovault V3 - AeroCrypt: that is a clean way to frame the unencrypted archive lane, and it is already tracked as item 7 in #5. The knot you spotted is the real work: in the vault, error correction currently sits over the final encrypted container, while an archive-only lane wants EC over the compressed-but-unencrypted bytes, which is a different layering. Untangling that ordering is the open design question, the packaging name is the easy part.

AeroFile / app level (tracked in AeroFTP, axpdev-lab/aeroftp#340):

On .mp3 to .opus: agreed, that is a lossy-to-lossy re-encode, so it throws away quality that is already reduced in the source, and a non-destructive tool should not do that quietly. The principle we want to keep is lossless or pass-through only, never a destructive re-encode.

On .jpg to .jxl: this one is genuinely interesting, because JXL can transcode an existing JPEG losslessly and reversibly (the bytes reconstruct back to the original JPEG exactly), with a real size win and no quality loss, which fits a non-destructive file tool well. "AeroFile as a GUI for libjxl" is a nice angle. Both of these are AeroFile features rather than format questions, so I logged them in the AeroFTP tracker axpdev-lab/aeroftp#340.

Thanks, these are good threads. Nothing here is a commitment yet, but the EC-placement and the unencrypted-archive layering are the questions worth settling first on the format side. For next time: format-level ideas land best here on aerovault (#4 / #5), AeroFile and app features on AeroFTP (#340).

[axpnet]

@EhudKirsh follow-up with the counter-tests for the two issues your V3 Beta run actually hit (the kill-mid-seal leftovers and the 16GB out-of-memory), plus the round-trip integrity you verified. All three are now closed; details and the exact regression tests are below.

The problems your tests surfaced, and where they stand now

• 🟢 Leftover temp on a hard stop (your 8GB .aerovault-v3-nSTAnw): the v3 container now writes through a NamedTempFile that auto-deletes on the error/drop path, and the standalone .aerocorrect repair scrubs its temp on the persist-error branch too (audit M1). A killed or failed run no longer leaves a decrypted-plaintext .aerovault-v3-XXXX beside the target.
• 🟢 Stale .aerovault.lock (your 56B leftover): a lock orphaned by a crashed seal is auto-reclaimed when its recorded owner PID is provably dead, via an atomic rename-aside (audit M9), instead of blocking the next writer ~30s and then hard-erroring until you delete it by hand.
• 🟢 Out of memory at 8GB on a 16GB input: the pre-flight memory guard now estimates the working set and warns before it would exhaust RAM, so you get a clear "too much for one vault" message instead of an OOM. The standalone .aerocorrect path is already windowed; true windowed streaming for the vault content path is tracked in the rolling tracker ([Tracker] AeroVault follow-ups, patches & roadmap #5).
• 🟢 Round-trip integrity (your b3sum check after decrypt/extract): still byte-identical. Round-trip plus byte-identity is part of the audited test suite and is green on both targets.

Counter-tests (current status)

Regression tests added per issue, all green (crate aerovault @ c5bc3f6, app aeroftp @ 1067e81e)

Your observation	What we re-tested	Result
8GB plaintext temp left after kill	v3 container writes via auto-deleting NamedTempFile; standalone repair scrubs temp on the persist-error branch (M1)	🟢 repair_restores_and_leaves_no_temp_artifact green (crate + app, in-memory and streamed): byte-restore confirmed AND no temp artifact left
56B .aerovault.lock stale, blocked the next run	PID-liveness auto-reclaim, atomic rename-aside (M9)	🟢 stale_lock_with_dead_pid_is_reclaimed green; live_owner_lock_is_not_stale green (a live owner is never stolen)
OOM at 8GB on the 16GB music set	pre-flight memory guard warns before allocating beyond budget	🟢 shipped; warns instead of OOMing
b3sum identical after decrypt + extract	round-trip + byte-identity across the full matrix	🟢 part of the 116-test crate suite + the app aerovault_v3 suite, green on Linux and Windows

These sit inside the broader 2026-06-18 security audit: 1 High, 1 Medium, 3 Low, 4 Info, 0 Critical, all fixed and verified on both the crate and the app, controaudit grade A with 0 open findings. The kill-cleanup pair (M1 + M9) is exactly the gap you reported here. Full write-up: #1.

Shipped in aerovault 0.6.2 (crates.io) and in AeroFTP 4.0.7. Thanks again for the precise filenames and the b3sum check, they made the cleanup gap trivial to reproduce and close.

(Your usability asks in this thread, the drag-and-drop, multi-select files+folders, extract-all, Technical Receipt save location, the TIME ELAPSED / time_elapsed wording, and the .aerozip unencrypted-archive idea, are tracked in the rolling tracker (#5); several already shipped. They are out of scope for this audit delta.)