Skip to content

v4.3.0 - Two new tools, a second tool chain, and the release where the plugin contract finally stopped assuming a source is one file.

Choose a tag to compare

View all tags
@pacnpal pacnpal released this 09 Aug 19:23
· 47 commits to latest since this release
v4.3.0
0b1077c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Compressatorium 4.3.0

Released 2026-08-09. Two new tools, a second tool chain, and the release where the plugin contract finally stopped assuming a source is one file.

4.2.0 pushed the architecture past two boundaries: a tool whose input is a folder, and a job that chains two tools together. 4.3.0 pushes past a third — a source that is twelve files — and adds two consoles' worth of coverage on the way: nkit2iso, which restores an NKit-shrunk GameCube or Wii image back to a full ISO, and JWUDTool, which converts Wii U disc images between .wud and .wux. That makes nine tools plus the synthetic chain tool.

Neither tool needed new special cases in shared code. That is the point of the release. Five more places where a shared path branched on one tool's identity — overwrite cleanup, tool availability, chain resolution, "can this tool convert this path", "what else belongs to this source" — became plugin hooks, each one pinned by a test against a synthetic second tool rather than the single real instance. Extension matching moved to a longest-suffix match so a format like .nkit.iso can be declared at all. The per-change breakdown lives in docs/RELEASE_NOTES.md.

The short version

  • An eighth tool: NKit → ISO (nkit2iso). Restore a .nkit.iso / .nkit.gcz back to a full-size, CRC32-checked .iso.
  • NKit info (GET /api/nkit-info) reads the NKit/disc header directly — console, game ID and title, restored size, CRC32, ratio — before you spend a restore on it.
  • A second chain: nkit_to_rvz. NKit → ISO → RVZ in one job, with the full-size ISO living in scratch instead of your library.
  • A ninth tool: Wii U (jwud). .wud ↔ .wux via JWUDTool, byte-for-byte verified during conversion, with a Skip verification option that roughly halves the runtime.
  • Split Wii U dumps work. game_part1.wud … game_part12.wud are joined on the fly, offered as one source, named after the disc, and deleted as a set.
  • WUX structural verification and info (/api/jwud-verify, /api/jwud-info) — header fields plus every sector-index entry against the file's real length.
  • Five more per-tool branches became plugin hooks: overwrite_targets, is_ready, converts_path, source_companions, scannable_path, delete_on_verify_is_safe, expected_output_size, and per-spec chain resolution.
  • Extension matching is a longest-suffix match now, so compound extensions like .nkit.iso are declarable without displacing the generic .iso owners.
  • A dangling symlink on an output path is removed instead of silently written through.
  • The "adding a tool" guide is back in sync with the code after drifting behind the Phase 7/9 registry cleanups.

New tool: NKit → ISO (nkit2iso)

NKit shrinks a GameCube or Wii disc image by dropping everything a program can recreate — junk padding, gaps, all-junk files, and for Wii the AES encryption and the H0–H3 hash tree. No emulator reads the result. NKit → ISO restores it: one mode, nkit_restore, no settings, using nkit2iso (MIT, a static Go binary built into the image for linux/amd64 and linux/arm64). The rebuild is CRC32-checked against the value stored in the NKit header, so a completed job is bit-exact. Archive members work too: an NKit image inside a .zip/.7z/.rar converts in place.

This is a one-way tool — Compressatorium never writes NKit — so there is no compress direction. It registers no verify route: nkit2iso has no such subcommand, integrity is the header CRC32 checked inline during the restore, and for the same reason delete-on-verify is not offered.

The one caveat, stated plainly. A Wii image whose update partition was removed at shrink time cannot come back byte-exact — that data is not in the file. By default (NKIT2ISO_RECOVERY=none) the region is zero-filled, exactly as official NKit does without its recovery files, and the job's final message says the ISO is playable but not redump-verifiable. Set NKIT2ISO_RECOVERY=download to have nkit2iso fetch the publicly archived recovery partition and splice it in for a bit-exact restore; that is the only path on which the tool touches the network, which is why it is opt-in.

NKit info (GET /api/nkit-info). nkit2iso has no info subcommand, so this reads the NKit/disc header directly — the same one the binary reads — and reports the console, game ID and title, disc number and version, the size the restored ISO will occupy, the CRC32 the restore is checked against, and the shrunk-to-original ratio. Worth a look before spending a restore on a multi-gigabyte image. A .nkit.gcz is handled by inflating a single zlib block to reach the header, with every header-supplied length bounded against the file's real size first, so a crafted container can't drive a multi-gigabyte read. A file named .nkit.iso whose bytes carry no NKit marker reports 422, not 500.

New one-step pipeline: nkit_to_rvz. The second ChainSpec, alongside cso_to_chd. The restored ISO is nearly always a stepping stone to RVZ — Dolphin reads it natively and it compresses better than NKit anyway — so chaining keeps the full-size intermediate in a scratch directory instead of your library. Neither NKit mode offers delete-on-verify: Dolphin's RVZ verify is structural, and a Wii image restored without its update partition yields a cleanly verifying RVZ that is not bit-exact — deleting the source on that evidence would destroy the only file a later recovery-enabled restore could use. The final RVZ uses Dolphin's defaults; run the two steps as separate jobs to pick a codec and level.

New settings: NKIT2ISO_PATH, NKIT2ISO_RECOVERY, plus the usual COMPRESSATORIUM_NKIT2ISO_NICE / _IOPRIO_CLASS / _IOPRIO_LEVEL overrides.

New tool: Wii U (JWUDTool)

Every Wii U disc image is exactly 25,025,314,816 bytes regardless of how much of the disc a game actually uses, so the padding compresses dramatically: WUX deduplicates repeated 32 KiB sectors and stores each distinct one once, and Cemu reads the result directly. Two modes, jwud_compress and jwud_decompress, both lossless, reversible, and with nothing to configure.

No keys. This only repacks the image, it never decrypts it — a .wux carries exactly the same encrypted content as the .wud it came from, so no common key or title key is involved and none ships with the app. (JWUDTool's decrypt/extract features do need your own keys; they are out of scope here.)

Every conversion is verified against its source. JWUDTool re-reads both images and compares them byte for byte before the job reports success, so a compress job takes about twice as long as writing the output alone and a mismatch fails the job instead of leaving a bad image in place. The progress bar covers both phases (conversion 1–50 %, verification 51–99 %). The verification pass is now a choice: it stays on by default, but the tool picker's dropdown — where other tools put their codec, since WUX has none — offers Skip verification, which passes -noVerify and roughly halves the runtime.

Structural verification for .wux. /api/jwud-verify checks header magic and fields, then every sector-index entry against the file's real length, which catches a truncated copy or a corrupt index table and is as deep as the format allows since WUX stores no content checksums. Both header size fields are pinned to the format's only real values (a 32 KiB sector, a whole disc image) rather than merely sanity-checked, since the index table's size scales inversely with the declared sector size — so no file, however crafted, can dictate how much work a verification does. Only .wux is verifiable, so delete-on-verify is offered on compress but not on decompress. /api/jwud-info reads the same header, reporting the original image size and the ratio.

Split dumps work too. wudump writes game_part1.wud … game_part12.wud and JWUDTool joins them when you pick part 1. Only part 1 is offered as a source; the rest stay listed (so you can see and delete them) but aren't convertible alone, since a part is not a disc image. A complete set's product is named after the disc (game.wux, not game_part1.wux), and delete-on-verify takes the whole set instead of orphaning eleven parts and ~23 GB of silent leftovers. A half-copied set, one with a gap, an orphan part, and an archived part all keep their own name — none of them can produce the disc, so none may claim its output and risk overwriting an unrelated finished image.

Both directions also accept a member straight out of a ZIP/7z/RAR. New setting: JWUDTOOL_PATH (default /usr/local/bin/jwudtool). JWUDTool is a Java program, so the image now installs a headless JRE plus the pinned, SHA256-checked upstream release jar behind a small launcher; the jar is architecture-independent, so both linux/amd64 and linux/arm64 are covered. The tool hides itself in the UI when no launcher is present, which is what a local checkout without Java sees.

A split set is a source, and it's guarded like one

Three ways a twelve-file source costs something a single file never did:

  • Completeness is checked part by part, not just as a total. A short part balanced by a long one sums to a whole image and is still unjoinable, because the reader computes each part's offset from its index rather than from the sizes before it. Every part must now be exactly the split part size until the remainder, which happens once, on the last part of a whole image. Eleven parts that happen to sum to a disc are rejected too.
  • Companion inputs are checked against the volume boundary. The sibling list is name-derived, and JWUDTool opens those files itself rather than taking a list from us — so a game_part2.wud symlink pointing outside the configured volumes would have had its bytes folded into the output, with only the submitted path having passed the route's containment check. A symlinked companion, or one resolving outside the volumes, is now rejected. Checked twice, like the PS3 folder walk it mirrors: once when the job is queued and again in the worker after the job takes its locks, since a queued set's part can be swapped in between. The re-check runs before any existing output is cleared, so a rejection on an overwrite job leaves your previous file intact.
  • Split parts are dropped from the DAT-match walk. .wud is a produced output, so the metadata scan would have SHA1'd twelve 2 GiB files per set — ~25 GB of reads to match nothing, because a part is a slice of a disc image rather than one. The veto is on membership of a real set (part 1 only when a part 2 is beside it, other parts only when part 1 is), so a whole disc image that merely happens to carry a _partN name stays scannable.

The set is protected end to end: the companion list also feeds job path tracking, so renaming or deleting part 5 while the conversion is running is refused like any other in-use path.

The plugin contract grew — and stopped assuming

Every hook below replaced a branch that sat on code every tool runs through, where a second tool of the same shape would have silently inherited the first one's behavior. Each is pinned by a test against a synthetic second tool (a fake directory tool, a gated non-nsz tool, a .foo→.bar chain), which is what catches a regression back to an identity branch — asserting on the single real instance passes either way.

Input-side hooks, for sources that aren't one file:

  • converts_path(path) — the input-side mirror of verifies_path. The file listing used to offer all twelve parts of a split dump as sources, eleven of which could only fail.
  • source_companions(path) — the input-side mirror of companion_outputs, read by the delete planner so delete-on-verify takes the whole set.
  • scannable_path(path) — the per-file veto on the DAT-match walk. Bounded I/O: a stat or two, never a read.
  • delete_on_verify_is_safe(...) — a per-job answer, because ModeSpec.supports_delete_on_verify only knows about the mode. jwud's structural verify is not what justifies deleting a 25 GB source — JWUDTool's byte-for-byte comparison is, and -noVerify turns that off. The shared request validator rejects the combination for single and batch alike.

Shared-path hooks, generalized out of single-tool special cases:

  • overwrite_targets() — overwrite cleanup branched on input_kind == DIRECTORY and called makeps3iso's remover directly, so any future folder-input tool would have had its outputs swept by makeps3iso's split-part logic: wrong files deleted, its own companions left behind, no error. The sweep now asks the owning plugin and runs one validate-then-unlink pass for every mode, file or directory. The blocking stat/unlink pass also moved off the event loop for file jobs.
  • is_ready() — GET /api/tools computed nsz's key probe and branched on tool.id == "nsz", so only Switch could ever be hidden; a second tool needing operator-supplied keys would have been advertised while unable to run.
  • Per-spec chain resolution — ChainTool.detect_output hard-coded a .chd candidate and the final-tool lookup read modes[0], so a second ChainSpec would have badged the first chain's product and delegated verify/info to the first chain's tool. Both now resolve per spec. ChainTool also names its product from the first step's stem plus the chain's own output extension, or nkit_to_rvz would have produced Game.nkit.rvz.
  • expected_output_size(input_path, mode) — the chain's disk-headroom preflight imported maxcso's size reader directly, so a second chain starting with a different tool would have silently fallen back to a ratio. Preflight-only, never a correctness input.
  • detect_output(..., from_archive) — so an archive member is never mistaken for a real file with real neighbours.
  • Chain steps can flag a caveat that survives to the terminal message. A later step's messages replace earlier ones, so the NKit "playable, not bit-exact" warning was being lost to a bare "Conversion complete". A step now marks such an update as a warning and the chain carries it into its own final message.

Two deliberate refinements came out of unifying the overwrite sweep:

  • A dangling symlink on an output path is now removed rather than ignored. exists/isfile both follow symlinks and report False for a dangling one, so the old check skipped it silently and the converter could write through the link, landing the output wherever it pointed, outside the validated volume. The sweep now tests with lexists/islink and still rejects any other non-file occupant before removing anything.
  • An authorized overwrite always clears the output's verification record, even when the sweep removed nothing — otherwise a record could survive something else deleting the prior output while the job sat queued, and report the freshly built, unverified artifact as verified.

Compound extensions

Every "does this tool handle this file?" check was Path(name).suffix.lower() in declared, re-typed at each site. That can only ever see one trailing component, which made .nkit.iso undeclarable: its trailing component is the generic .iso that CHDMAN, Dolphin, and maxcso already own, so the choice would have been over-claiming every ISO or hard-coding a content sniff.

The check now lives behind one shared helper that returns the longest declared extension a name ends with, used by the registry's input/verify/archive-member lookups, the archive-member gate, output detection, and the plan-time input gate. No behavior change for any existing tool: every other declaration is a single component, and game.iso ends with .iso and nothing else. Matching is per-tool, so an NKit image is claimed by nkit and by the generic-tail owners — the same way a raw .iso is already claimed by chdman, dolphin, and cso — rather than displacing them.

The frontend now ranks claiming tools by matched-extension length instead of declaration order. Dolphin's header reader genuinely succeeds on an NKit image (it carries a real GC/Wii disc header), so without the ranking the Info modal would have described a .nkit.iso as an ordinary disc.

One test changed meaning in the same pass: the old delete-on-verify test asserted equivalence in both directions. The safety-critical direction still holds for every mode with no exceptions (delete ⇒ the output must be verifiable), but the converse is wrong — verification proves an output is well-formed, not that the conversion preserved the source. A mode whose pipeline can be lossy now declines the flag via an explicit, documented allow-list.

Documentation and CI

  • The "adding a tool" guide is back in sync with the code (docs/ADDING_PLATFORMS_AND_TOOLS.md). It had drifted behind the Phase 7/9 registry-driven cleanups and still told contributors to write code that no longer exists — per-tool FileEntry flags, a routes/files.py edit, a FileRow.svelte fallback, an if-block in plan-time validation. Corrected across all four views of the same list (the inventory, the walkthrough, the checklist, the file map), and extended to cover the parts of the contract that had never been written up: ModeSpec.companion_exts / input_kinds, ChainSpec, accepts_directory(), companion_outputs(), the split convert kwarg, the full registry query surface, the fact that info and verify routes are optional, the synthetic ChainTool registration and its ordering constraint, the five-stage Dockerfile, and PYTHONPATH=app on every test command. Also fixed an internal contradiction about what tests/conftest.py stubs (nothing — it is DB-only), filled the missing "which existing tool should I copy?" section, and added the screenshot-refresh step. It is now linked from README.md, AGENTS.md, and CLAUDE.md, which previously pointed only at the design doc.
  • makeps3iso is named in the FastAPI app description (#254) — the OpenAPI blurb still listed six tools after the seventh shipped.
  • The screenshot workflow's commit job no longer runs on tag pushes (#252), so cutting a release doesn't race a screenshot commit onto the tagged tree.

Upgrade notes

No database migration. The schema is unchanged from 4.2.0 (0003 remains the head).

New settings, all optional:

Variable Default What it does
NKIT2ISO_PATH /usr/local/bin/nkit2iso Path to the nkit2iso binary
NKIT2ISO_RECOVERY none none zero-fills a removed Wii update partition (offline; playable but not redump-verifiable); download fetches the archived recovery partition for a bit-exact restore. The only path on which the tool uses the network.
JWUDTOOL_PATH /usr/local/bin/jwudtool Path to the JWUDTool launcher
COMPRESSATORIUM_NKIT2ISO_* / COMPRESSATORIUM_JWUD_* unset The usual _NICE / _IOPRIO_CLASS / _IOPRIO_LEVEL per-tool priority overrides

The image is larger. JWUDTool is a Java program, so the container now ships a headless JRE plus the pinned upstream jar, alongside the new static nkit2iso binary. Both new tools are built for linux/amd64 and linux/arm64.

Behavior worth knowing if you script against the API:

  • Extension matching is longest-suffix, not last-component. Existing single-component declarations behave identically; a .nkit.iso is now claimed by nkit in addition to the generic .iso owners.
  • A dangling symlink on an output path is now removed by an authorized overwrite instead of being ignored.
  • An authorized overwrite always clears the output's verification record, even when it removed nothing.
  • Delete-on-verify can be refused per job, not just per mode — a jwud_compress job with Skip verification on and delete-on-verify on is rejected up front, for single and batch alike.
  • .wud split parts no longer appear in the DAT-match scan, and only part 1 of a set is offered as a conversion source.

Everything else is backward compatible: mode strings, endpoint paths, and JSON shapes are unchanged, and all 4.2.0 environment variables (including the CHD_* aliases) keep working.

Full commit list since v4.2.0 (11 commits, oldest first)

  • d0ff5b8 chore: sync package.json to v4.2.0 [skip ci]
  • 688e9ac ci: don't run the screenshots commit job on tag pushes (#252)
  • 7faf2b4 Update Web UI screenshots — Sun Aug 9 01:26:59 UTC 2026
  • c6ff9a5 fix: name makeps3iso in the FastAPI app description (#254)
  • 022adf3 refactor: turn three per-tool branches into plugin hooks (overwrite_targets, is_ready, per-spec chain resolution) (#255)
  • ba29785 docs: bring the "adding a tool" guide back in sync with the code (#253)
  • ada8447 Add nkit2iso: NKit → ISO restore, NKit info, and the nkit_to_rvz pipeline (#257)
  • 17dc223 Update Web UI screenshots — Sun Aug 9 17:29:40 UTC 2026
  • 7f9311d feat: add JWUDTool (Wii U) as a ninth tool — .wud ↔ .wux (#256)
  • 2f29854 Update Web UI screenshots — Sun Aug 9 17:58:22 UTC 2026
  • 0b1077c fix: harden split Wii U sets — exact part sizes, companion containment, scan cost (#258)
Assets 2
Loading