→ In plain English — Mathpressor is a lossless compressor with a specialty: numbers. Give it the kind of data that's made of measurements — scientific simulation grids, sensor/telemetry logs, audio, raw float/int arrays — and it beats the popular tools (zstd, xz, 7-Zip), because it understands that consecutive values are close, not random. It also does something a normal
.zipcan't: a program can run directly from the compressed file, decoding only the bits it needs. It is not a magic shrink-ray for everything — on already-squeezed data like game art, photos, or video, the specialist tools win. (We did run a 1 GB game live off a 320 MB archive at full speed — a neat trick, but the real, measured edge is on numerical data.)This README is written two ways at once: the technical text, and an In plain English box like this after the jargon-heavy parts. It also flows shallow → deep — what it is and how to use it come first; the byte-level internals (opcodes, wire format) are at the bottom. Read whichever you like, scroll as far down as you want.
A lossless compression engine, container format, and live VFS — 100% Zig. Its measured edge is binary numerical & structured data: it routes each file to whichever reversible transform is smallest — a value-domain predictor (1D delta / 2D Lorenzo over float & int arrays, the stencil the scientific compressors ZFP/SZ use), columnar transpose, audio LPC, 2D image prediction, x86 BCJ2/RIP filters, or demoscene procedural synthesis — then a strong general backend (zstd / LZMA / context-mixing) or raw storage, always keeping the smallest. The insight: statistical compressors (zstd, xz, 7-Zip) hunt for repeated byte substrings; numerical data's redundancy is that values trend and vary smoothly, which those tools can't exploit but a predictor can. A host can then run straight off the compressed archive — random-access, decode-on-demand, nothing inflated to disk. Ships as a CLI, a desktop GUI, and a libmathpressor.so over a plain C-ABI.
Measured, lossless, on a Ryzen 5 5600X: a smooth scientific float field compresses 3.16× where xz manages 1.32× and zstd 1.11×; real sensor/telemetry columns beat xz on 11/11; a monotonic counter goes 9070× vs xz's 29×. And a real 1.06 GB Unreal game runs live off a 320 MB archive at native 60 fps, nothing inflated to disk. Same bits on every CPU architecture. (See bench/STRUCTURED-DATA-RESULTS.md.)
Mathpressor isn't trying to beat everything at everything. It has a real, measured edge on one kind of data and honestly loses on others — knowing which is which is the whole point.
Built for — data with numerical / predictive structure, in binary form (it beats zstd and xz here, often by a lot, while staying lossless and live):
- scientific & simulation arrays — float/double grids and fields;
- sensor / telemetry / IoT time-series;
- columnar / tabular binary data;
- audio (PCM);
- monotonic sequences — timestamps, counters, IDs.
Not built for (the specialists win — use them):
- games / already-compressed assets (Oodle/IoStore, Bink video) — incompressible by everyone (~1.0×);
- authored media (photos, video) — already encoded;
- text (logs, source, JSON, CSV-as-text) — LZMA/7-Zip's substring modeling wins; a number stored as ASCII hides its structure;
- high-entropy data (random, encrypted, embeddings) — incompressible by everyone.
The keep-smaller guard means Mathpressor never makes a file bigger — on "not built
for" data it simply matches the general backend (zstd/LZMA). The win is specifically on
numerical/structured data; that's the lane. One caveat worth stating up front: the
numerical predictors need the data in binary form — the same telemetry that loses as
a text CSV wins 9/11 as binary float columns. Full results, including the honest losses
and the cases that flip on format, are in
bench/STRUCTURED-DATA-RESULTS.md.
The core idea: don't store the data — store how to reconstruct it. General compressors shrink data by finding repeated byte-strings. Mathpressor adds a second axis they lack — prediction:
- Predict the next value, store only the (tiny) error. A float in a smooth field is
close to its neighbors; an audio sample is close to the last one; a counter is the
previous plus a constant. Predict it and the residual collapses to near-zero runs the
codec crushes. This is
MATH_FLOAT's 2D Lorenzo,MATH_AUDIO's LPC,MATH_IMAGE2D's MED — the predictors general compressors don't have, and the source of the numerical wins. - Or store a generator instead of the output. If a texture is something a little
math formula can draw (clouds, noise, gradients), store the recipe — a few bytes —
and re-draw it on demand (
MATH_BYTECODE/MATH_RESIDUAL). Rare on real authored data, but unbeatable when it applies: a 26-byte program expands to a 9,216-pixel texture.
Most files have neither kind of structure, so Mathpressor falls back to a strong general backend (zstd / LZMA / context-mixing) or raw storage — always keeping the smallest, so it never inflates a file. The predictors double as the ideal live primitive: cheap, exact, decode-on-demand.
→ In plain English — Instead of one way to shrink a file, Mathpressor tries several and keeps the smallest. The fanciest trick: if a texture is something a little math formula can draw (like clouds or static), it stores the recipe (a few bytes) instead of the picture (thousands of bytes) and re-draws it on demand. Most files aren't drawable that way, so for those it uses normal-but-strong compression. It never makes a file bigger — worst case it just stores it as-is.
When you pack a directory, Mathpressor automatically routes each file to one of these (the diagram shows the original four; the table lists them all):
┌─────────────────────────────────────┐
│ entropy gate (7.5 b/B) │
└──────────────┬──────────────────────┘
entropy < 7.5 │ entropy ≥ 7.5
│ │ │
▼ │ ▼
┌──────────────────┐ │ ┌──────────────────┐
│ Math search │ │ │ addBinary() │
│ 5 000 attempts │ │ │ STORE guard │
└──────┬───────────┘ │ └──┬───────────────┘
│ │ │
┌───────────┼───────────┐ │ ┌─────┴──────┐
▼ ▼ ▼ │ ▼ ▼
[MATH_BYTECODE] [MATH_RESIDUAL] [FALLBACK_STREAM] [STORE]
exact match ≥70% match + gzip wins gzip inflates:
tiny program gzip'd delta raw bytes
| Route | Type byte | When | Container overhead |
|---|---|---|---|
MATH_BYTECODE |
0x01 |
Program found, bit-perfect | ~10–30 bytes per asset |
FALLBACK_STREAM |
0x02 |
Structured/text data | gzip compressed block |
STORE |
0x03 |
Already compressed / random | raw bytes (guard prevents inflation) |
MATH_RESIDUAL |
0x04 |
≥70% match found | program + gzip'd delta |
MATH_BLOCKS |
0x07 |
Pages part-equation, part-data | per-block descriptors + literal stream |
MATH_FILTERED |
0x08 |
Reversible transform helps the codec | filter id + compressed filtered stream |
MATH_COLUMNAR |
0x09 |
Record array (vertex/float tables) | AoS→SoA transpose + compressed |
MATH_IMAGE2D |
0x0A |
Raw raster (TGA/PGM/PPM) | reversible subtract-green + planar + 2D MED predictor, compressed |
MATH_DICT |
0x0B |
Many similar small files (JSON/strings/shaders) | zstd frame primed with a shared trained dictionary |
MATH_AUDIO |
0x0C |
16-bit PCM WAV | fixed-order LPC (per-channel sample predictor) + compressed |
MATH_BCJ2 |
0x0D |
Full mode's solid tar (x86 code) | 4-stream range-coded BCJ2, each LZMA'd |
MATH_CM |
0x0E |
Full mode's solid tar (text/general) | context-mixing coder (orders + match + logistic mixer + SSE) |
MATH_CHUNKED |
0x0F |
Large live files (> 256 MB) | independently zstd-compressed 4 MB chunks + seek index — a read decodes only the covering chunk(s), never the whole file (true live random access); each chunk optionally primed with a per-file content dictionary so cross-chunk redundancy survives |
MATH_FLOAT |
0x12 |
Binary float/int arrays (scientific grids, sensor/telemetry, time-series) | value-domain predictor: map each value to a sort-order-preserving integer → 1D delta or 2D Lorenzo (left+up−up-left, the ZFP/SZ stencil, with auto-detected row width) → byte-plane the residual → compress. Net-new ground vs zstd/xz, which have no value-domain predictor |
All routes reconstruct to bit-identical original bytes. To the caller, they are invisible.
Many small similar files (per-language strings, JSON manifests, shader variants) compress far better when the codec can reference patterns shared across files. A solid block does that but destroys random access, so it is banned from live mode. A trained zstd dictionary gets the same cross-file sharing while keeping every entry independently decodable: one dictionary is trained per file-extension group at pack time, shipped once per archive, and each entry is its own dict-primed frame (still random-access, still live).
It is fully guarded so it never costs bytes:
- A file becomes a
MATH_DICTentry only when its dict-primed frame is smaller than the size it would otherwise get from the real backend (the honesty guard — at Max that baseline is per-entry LZMA, so a smaller LZMA block is never traded for a larger dict one). - A dictionary is kept only when the group's total saving repays the bytes the dictionary costs to ship (the amortization gate). Several dictionary sizes are tried and the best net is kept.
- Files that don't benefit fall through to normal per-file routing.
Set MATHPRESSOR_NODICT=1 to disable the route (diagnostic / A-B switch).
16-bit PCM samples are strongly correlated sample-to-sample, but general
compressors only have byte-level delta — they can't predict across a 2-byte
sample or per channel. MATH_AUDIO deinterleaves a WAV's channels and applies a
fixed-order linear predictor (the FLAC/Shorten family, orders 0–3, best chosen
per file), storing the residual. The WAV header and any post-data chunks stay
verbatim; only the PCM region is predicted; wrapping integer arithmetic makes it
an exact involution. Because the predictor is net-new ground, this beats general
compressors — even solid ones — on raw audio while staying per-entry/live
(measured: a 7-file WAV set packed to 11.9 KB vs solid 7z 18.5 KB, solid xz
23.5 KB, per-file xz 25.3 KB). Honesty-guarded like every route.
→ In plain English — Two settings. Regular is the "play it live" setting: a bit bigger, but you can run the game straight from it. Full is the "shrink it as much as possible" setting: smaller, but you must fully unpack it before using it (like a backup or a download). Same files, you pick the trade — run-it-now vs smallest-possible.
Mathpressor has two packing modes with different purposes, not just different ratios:
-
Regular mode — live-runnable. Every asset is an independent, randomly accessible entry. The design goal is that a host (e.g. a game) loads the
.mathand synthesizes or extracts only the assets it needs, the moment it needs them — a virtual filesystem where a procedural asset effectively doesn't exist until it's requested, then is generated on the spot. This is why regular mode never uses solid blocks (which would force decompressing a whole block to read one file) and why decode/synthesis latency matters as much as size.MATH_BYTECODEentries are the ideal live primitive: near-zero storage and no decompression — pure on-demand VM synthesis. Two passes share data across files without breaking random access: whole-file dedup (identical files share one blob) and the trained-dictionary route (MATH_DICT, above) for many similar small files. Both keep every entry independently decodable. -
Full mode — cold archive. The whole selection becomes one solid tar, then at Max it races several backends and keeps the smallest:
- BCJ2 (
MATH_BCJ2) — a 4-stream, range-coded x86 branch converter that splits CALL/JMP target addresses into their own LZMA streams and models the convert/skip decision adaptively. Wins on code; on libc it lands within ~0.6% of 7-Zip and beatsxz --x86by ~4.4%. - CM (
MATH_CM) — a pure-Zig context-mixing coder: indirect context models (each context → an 8-bit bit-history state → a StateMap), orders 1..16 from the full history buffer, two match models, an online logistic mixer, SSE, binary arithmetic coding. On text/code it beats every general compressor outright: a 116 MB/usr/includecorpus packs to 7.05 MB vs 7-Zip-mx9's 8.34 MB (−15.4%) and xz's 8.40 MB — measured, repro inbench/CM-UNCAP-AND-DICT.md. CM also feeds the BCJ2 main stream (~6% on de-addressed x86 code). It is cold-only and slow (~0.3 MB/s); since cold storage doesn't care about speed, it runs up to a 1.5 GiB memory-bounded cap — large enough to cover whole archives, where a prior 96 MiB cap used to silently drop CM and fall back to LZMA. Never on the live path; regular/live mode keeps the fast LZMA main stream. - RIP filter — an x86-64 length decoder rewrites
[rip+disp32]references (the GOT/data refs that fill PIC.so/PIE binaries) to position-absolute so repeated refs match for the LZ stage. 7-Zip's x86 filter doesn't do this. It's restricted to ELF executable sections (verify-then-use safe) and feeds LZMA, so it's fast-decode and used in regular/live mode too — which is how regular mode also beats 7-Zip on binaries while staying live. - plain LZMA / in-place x86 BCJ as the fast baselines.
The keep-smaller guard means full mode never loses to any single backend. Where full mode beats 7-Zip / xz (measured): on structured data its context-mixing coder + domain transforms win — text/source −16 %, x86-64 binaries −5 %, raw images −2 % vs
7z -mx9. Where it doesn't: on opaque / already-dense data (e.g. an Unreal asset pak) 7-Zip and xz are smaller, and on small raw audio 7-Zip edged it in testing. So the honest claim is full mode beats general compressors on text / code / image; it is competitive, not uniformly smaller, on opaque data. Full mode is maximum ratio but must be fully expanded to use — not live-runnable. - BCJ2 (
→ In plain English — Compared to the popular shrink tools (7-Zip, xz), Mathpressor wins on things it understands well — text, program code, and pictures (it's noticeably smaller there). It loses on stuff that's already hard to shrink, like a game's pre-packed artwork. So it's not "smaller than everything at everything" — it's "smaller on the things it has special tricks for, and you-can-run-it-live on the rest."
The intended hierarchy: full mode is #1 on ratio for the data types it models
well; regular (live) mode trades some ratio for random access and on-demand
decode (it still beats 7-Zip on x86 binaries while staying live). The two
constraints — "smallest on size" and "run live" — are in genuine tension (a
stronger but slower backend helps the first and hurts the second), which is why
regular mode keeps a fast path and treats heavier backends as a ship/cold option.
See bench/REPORT.md and docs/DISCOVERIES.md
for the per-type numbers and an honest prior-art assessment.
One workload stays structurally full-mode's: many tiny near-duplicate files. A solid block references the full window across every file with zero per-frame overhead; a per-entry dictionary captures the shared patterns but still pays a small per-frame cost, so it closes most of the gap (e.g. 2.4× better than the old per-file regular mode on such a corpus) without matching solid. That is the live-vs-cold trade working as intended — random access has a price, and full mode is the place you choose to stop paying it.
Regular mode's promise is that a host reads assets live from the compressed
archive — the archive never inflates to its full size on disk. src/mathfs.zig
makes this real: a read-only FUSE filesystem that exposes the packed tree and,
on each read, decodes only the bytes that read touches, into a bounded RAM cache.
Mount it over a game's install path and the game runs off the .math with no idea
it isn't reading plain files.
The enabling piece is MATH_CHUNKED storage. A whole-file zstd stream isn't
seekable — reading one byte of a 961 MB pak means inflating all of it (a multi-
second stall). So large live files are stored as independently-decodable 4 MB
chunks plus a seek index; a read maps to its chunk(s) only (addChunkedStreamingFile,
Reader.readChunk, ABI mp_entry_chunk_size / mp_read_chunk). mathfs adds a
concurrent decode engine on top: decodes run outside the cache lock (refcounted,
so many cores decode at once), an adaptive prefetch pool reads ahead only on
detected sequential runs, and a large RAM LRU keeps the working set warm.
→ In plain English — A normal squeezed file is like a vacuum-sealed bag: to grab one sock you must open the whole thing. That's why an early version froze for ~8 seconds whenever the game needed something. The fix: pack the big file as many small sealed bags (4 MB each) with a table of contents, so the game opens only the bag it needs (a few milliseconds) and leaves the rest sealed.
mathfsis the helper that pretends to be ordinary folders to the game, but secretly opens those little bags on demand — using all your CPU cores, guessing what's needed next, and remembering recent ones. Result: the game reads from the 320 MB file as if it were the full 1 GB sitting on disk, smoothly.
Measured on a real Proton game (FPS Chess, UE4, 1.06 GB install, 961 MB pak):
it runs live off a 320 MB .math at native 60 fps with zero frametime
hitches (worst frame 22 ms; a single-byte pak read dropped from an 8.1 s
whole-file stall to ~14 ms). Storage 3.3×, nothing inflated to disk, bit-perfect.
Full methodology, numbers, and the harness are in bench/REPORT.md;
the development story is in docs/JOURNEY.md; an honest novelty /
prior-art assessment (what's new vs known, with a per-type ratio comparison vs 7-Zip
and xz) is in docs/DISCOVERIES.md.
zig build tools # builds vfs_runner, concread, mathfs (needs libfuse3-dev)
mathfs <archive.math> <mountpoint> -f -o ro # mount; reads decode chunks on demand
fusermount3 -u <mountpoint> # unmountzig build # ReleaseFast, stripped — produces exe + .so
zig build test # run the unit test suite (52 tests)
zig build run # synthesis demo (96×96 ASCII preview)
# Modes
./mathpressor # demo: synthesise and preview a texture
./mathpressor bench # benchmark: 5 asset types × 512×512
./mathpressor pack_demo # showcase: all 4 routes, verify bit-perfect
./mathpressor pack <dir> <out> # pack a directory tree → .math archive
./mathpressor packfull <dir> <out> [tier] # full mode: solid TAR → zstd .math
./mathpressor unpack <in> <dir> # unpack a .math archive → directory
./mathpressor <prog.mpc> <out.pgm> # synthesise a .mpc bytecode file → PGM imageFull mode trades per-file random access for the best ratio on file trees: the
selection is written as one solid uncompressed tar (std.tar.writer — pure
Zig, no system tools), then the whole stream is compressed into the .math
container at the effort tier (FLAG_FULL_TAR). The tar is ordered by
(extension, path) so similar files sit adjacent in the stream, keeping the
.math header, FAT checksum, and GUI integration. Unpack detects the flag and
expands the inner tar with std.tar.pipeToFileSystem, preserving symlinks and
executable bits.
The full-mode backend is LZMA/xz (liblzma, the same kind of system C
dependency already used for libzstd), at preset 6 / 6 / 9-extreme by tier —
a stronger entropy model than zstd (range coder + adaptive bit-contexts +
match model). On real Steam binaries this turns the one case that used to lose
to xz into a win: linux64 (47 MB) packs to 10.84 MB vs 11.31 MB for stock
tar | xz -9e (and 12.81 MB for tar | zstd -19). Only xz -9e --x86 —
xz's own x86 filter, which the user must know to enable — edges it out, by ~2%.
Before the tar is built, a parallel math pre-pass runs the translator over
every file: anything expressible as a bit-perfect program (sparse/zeroed files,
byte ramps, tiled patterns — and at Max, procedural noise) is lifted out of the
tar into a MATH_BYTECODE entry, and x86 executables are lifted out as
individual BCJ-filtered LZMA entries — so full mode genuinely combines
mathematical synthesis and reversible transforms with solid traditional
compression. A real Chrome profile's 4 MB zeroed metrics file becomes an
8-byte program (524,288×). A program is only accepted when it is strictly
smaller than the file — the math route must earn its place. The iterative noise
search runs at Max only; benchmarked across real corpora it matches nothing the
analytic detectors miss, while the detectors are O(n) and effectively free.
./mathpressor pack Steam/ steam.mathOn a real Steam installation (~8.6 GB, 58 000+ files): the streaming builder keeps peak RAM under 15 MB regardless of archive size. Files are routed by entropy — compiled DLLs and .so files get gzip'd at 2.4–2.5×, shell scripts and HTML at 3–9×, already-compressed .tar.xz / .gz files hit the STORE guard and are kept raw.
| Constraint | Implementation |
|---|---|
| 100% Zig | No C, C++, or Python anywhere in the engine or build system. |
| Zero float in the core loop | The VM, math generators, and all runtime paths use strict 32/64-bit integer arithmetic. f64 is used only in the offline translator (Shannon entropy). Verified bit-identical on x86-64 LE, big-endian s390x, and aarch64 — all produce checksum 0x5757ceb1. |
| No hidden allocations | Every asset is synthesized inside its own std.heap.ArenaAllocator torn down on return. The pack CLI uses a streaming writer (one file's compressed data in RAM at a time, not the whole archive). |
| Custom PRNG | A hand-rolled, frozen XorShift32 with shift triple (13, 17, 5). Never std.Random — the stdlib's internals can change between Zig releases. Zero-seed remapped to 0xDEAD_BEEF. |
| Wrapping arithmetic for delta | Residual delta uses -% / +% wrapping subtraction/addition, keeping reconstruction in-range without any clamping or branching. |
mathpressor/
├── build.zig ReleaseFast + strip by default; produces exe + .so
└── src/
├── math_gen.zig Deterministic integer generators (PRNG, noise, cellular, warp)
├── vm.zig Bytecode interpreter + Builder assembler (11 opcodes)
├── translator.zig Opportunistic translator: entropy gate → math search → delta
├── container.zig .math archive format: Builder, StreamingBuilder, Reader
├── abi.zig Mathpressor C-ABI boundary (mp_* exports)
└── main.zig CLI entry point (demo, bench, pack, unpack, pack_demo)
Build produces two artifacts:
zig-out/bin/mathpressor— standalone CLI (demo, bench, pack, unpack)zig-out/lib/libmathpressor.so— C-ABI shared library host applications link at runtime
A host application loads libmathpressor.so and drives the engine through a plain
C-ABI — only integers and raw pointers cross the boundary, so it is callable from
C, C++, Rust, or any FFI. All exported symbols use the mp_ prefix.
The core runtime entry point is asset synthesis:
pub export fn mp_synthesize_asset(
asset_id: u32,
bytecode_ptr: [*]const u8,
bytecode_len: usize,
out_buffer_ptr: [*]u8,
out_buffer_len: usize,
) i32;Returns bytes written (≥ 0) or a negative MP_ERR_* code. Each call constructs and destroys its own ArenaAllocator over page_allocator — no retained state, no hidden heap growth.
The same library also exposes the tooling entry points used by the bundled desktop GUI (these walk the filesystem and write archives, so they are heavier than the stateless synthesis call above):
| Symbol | Purpose |
|---|---|
mp_synthesize_asset |
synthesize one asset from bytecode (runtime path) |
mp_pack_directory_auto / _vfs / _solid |
pack a directory into a .math archive |
mp_extract_file |
extract one file from a .math archive (re-parses each call) |
mp_open / mp_close |
open an archive once (parse FAT + build a path index), then close |
mp_read_entry |
decode one asset by path from an open handle — O(1) lookup, no re-parse |
mp_entry_chunk_size / mp_read_chunk |
chunk geometry + single-chunk decode — live random access into large MATH_CHUNKED files (used by mathfs) |
mp_entry_size / mp_entry_count / mp_entry_name / mp_entry_size_at |
size/enumerate entries on a handle |
mp_fnv1a |
FNV-1a checksum helper |
Live VFS for a game engine. A host that streams many assets from one archive
uses the open-handle API: call mp_open once (the archive bytes must stay mapped),
then mp_read_entry(handle, path, buf, len) per asset on demand — an O(1) path
lookup plus a single-entry decode (true random access; no FAT re-parse). Every
regular-mode route (LZMA, BCJ2+RIP, dict, audio, image, columnar, math-bytecode)
decodes per-entry, so the whole live VFS works at Max. Measured on a 1245-entry
archive: reading every asset via the handle is 3.2× faster than re-parsing per
call, and each asset decodes at 38–244 MB/s depending on route.
The pack functions run serially: std.Thread.Pool cannot initialize inside a dlopened shared library (the Zig start code that sets up thread-local storage never runs), so the parallel pack pipeline lives only in the standalone CLI. The GUI calls these from a background thread, so the UI stays responsive.
src/translator.zig runs offline (at pack time) to decide the route for each asset.
Any byte length qualifies: the canvas is the smallest covering rectangle
(side = ceil(√len)), and extraction truncates the padded tail.
Phase 1 — Structural gate (O(1)):
Reject empty files and anything beyond the 4096×4096 canvas (≈16.7 MB).
Phase 2 — Analytical detectors (O(n), no search):
One linear scan recognises three structures that occur in real files and
constructs the program directly — no iteration:
CONST_FILL— padding / sparse / zeroed files (the modal byte)RAMP— arithmetic byte sequences (start + step·i mod 256, lookup tables)REPEAT— short-period tiled patterns (repeated structs, stride records)
An exact hit is verified once through the VM and stored as MATH_BYTECODE.
These run before the entropy gate deliberately: a perfect byte ramp has a
uniform histogram (8.0 bits/byte) and the entropy heuristic would reject it.
Phase 3 — Entropy gate (O(n)):
If ≥ 7.5 bits/byte (encrypted, already-compressed, random) → skip the search
(a qualifying analytic approximation is still carried through).
Phase 4 — Iterative math search (budget set by the effort tier):
Sweep (seed, template, frequency) combinations through the VM. For each candidate, compute the L1 norm (sum of per-byte absolute differences).
- L1 = 0 → exact match →
MATH_BYTECODE - L1 > 0 but ≥70% of bytes are exact →
MATH_RESIDUAL- Delta compiled as:
delta[i] = raw[i] −% approx[i] - Delta compressed (exact positions are 0, compresses well)
- Delta compiled as:
- Nothing qualifies →
addBinary()(compress vs STORE guard)
The STORE guard (inside addBinary):
Always compare compressed output size vs raw. If it inflates, store raw bytes instead — the container never inflates any file.
The residual guard (pack paths):
A MATH_RESIDUAL is stored only when program + compressed delta is smaller
than compressing the whole file — the math route must earn its place, never
be overhead dressed up as a win.
Phase 5 — Per-block decomposition (MATH_BLOCKS, fallback files only):
Files whose pages are part-equation, part-data decompose into 4 KB blocks:
each block gets an exact analytic check (constant / ramp / repeat), equation
blocks become 1–3 byte descriptors, and the literal blocks concatenate into
one stream compressed conventionally. The same honesty guard applies — the
decomposition is stored only when it beats whole-file compression. Measured
honestly: LZ codecs already capture constant and repeated pages nearly free,
so this route fires where analytic pages cost LZ real bytes (e.g., files of
distinct lookup-table ramps: 3.5% smaller than zstd on the same data) and
stays silently out of the way everywhere else.
Phase 6 — Reversible math filters (MATH_FILTERED):
A filter is a length-preserving, exactly-invertible integer transform applied
before the codec — it shrinks nothing itself, it rewrites the bytes so the
LZ/entropy stage finds more redundancy. This is the literal sense of "use math
to make the file cheaper," and it's how xz beats plain DEFLATE:
- delta (distance 1 / 2 / 4) —
out[i] = in[i] −% in[i−d]; turns counters, gradients, and PCM-style data into runs of small values the codec crushes. - x86 BCJ — rewrites
CALL/JMPrel32 operands to absolute, so the same function called from many sites becomes byte-identical and the codec matches it. Reversible because the opcode byte is never touched and the 4 operand bytes are skipped, so encode and decode walk identical positions.
The pack path tries the viable filters (BCJ is gated by a cheap x86-density
prescreen), compresses each, and keeps the smallest — but only if it beats the
unfiltered representations (honesty guard). On a real 46 MB Steam .so the BCJ
filter is ~5% smaller than zstd-19, bit-perfect; full mode lifts such
executables out of the solid tar into individual MATH_FILTERED entries, so
the binary win and the cross-file solid win compose.
Built-in templates: single_noise, noise_invert, noise_bright, blend_mult, cave, marble, plus the three analytic constructors above.
All multi-byte operands are little-endian, decoded with explicit std.mem.readInt calls — never raw pointer casts — so bytecode is fully portable across architectures.
The VM has 4 buffer slots (indexed 0–3). OP_INT_NOISE fills a slot and makes it the current buffer. Post-processing ops (OP_INVERT, OP_ADD_CONST, etc.) act on the current buffer. OP_BLEND_MULT, OP_MIX, OP_WARP, and OP_COPY combine slots.
| Opcode | Mnemonic | Payload | Effect |
|---|---|---|---|
0x01 |
SEED |
u32 |
Reseed the PRNG |
0x02 |
INT_NOISE |
u8 dst, u16 w, u16 h, u8 freq |
4-octave fractal integer noise → slot dst; select it |
0x03 |
INVERT |
— | 255 − p on current buffer |
0x04 |
ADD_CONST |
i16 |
Saturating brightness offset |
0x05 |
BLEND_MULT |
u8 src |
cur[i] = cur[i] * src[i] / 255 (multiply mask) |
0x06 |
COPY |
u8 src, u8 dst |
Copy slot src → slot dst |
0x07 |
CELLULAR |
u8 steps, u8 birth, u8 survive |
Moore-neighbourhood CA smoothing |
0x08 |
WARP |
u8 src, u8 strength |
Domain-warp current buffer using slot src as displacement |
0x09 |
LEVEL |
u8 lo, u8 hi |
Contrast stretch: remap [lo, hi] → [0, 255] |
0x0A |
THRESHOLD |
u8 pivot |
Binarise: ≥pivot → 255, <pivot → 0 |
0x0B |
MIX |
u8 src, u8 alpha |
Linear blend current ← alpha/255 * src |
0x0C |
CONST_FILL |
u8 dst, u16 w, u16 h, u8 value |
Fill slot with a constant byte (padding/sparse files) |
0x0D |
RAMP |
u8 dst, u16 w, u16 h, u8 start, u8 step |
buf[i] = start + step·i mod 256 (lookup-table ramps) |
0x0E |
REPEAT |
u8 dst, u16 w, u16 h, u8 plen, plen×u8 |
Tile a literal pattern: buf[i] = pat[i % plen] |
0xFF |
HALT |
— | Lock current buffer as output, end execution |
Wire layout (all integers little-endian):
[12 B] Header: "MATH" magic, version u16=1, fat_count u32, reserved u16
[280 B × N] FAT: one entry per file (path[240], comp_type, offsets, sizes, checksum)
[variable] Data region: compressed blocks in FAT order
FAT entry layout (280 bytes):
path[240] null-terminated UTF-8 relative path
comp_type u8 0x01 / 0x02 / 0x03 / 0x04 (see table above)
_pad[7]
data_offset u64 byte offset from start of data region
original_size u64 uncompressed byte count
compressed_size u64 size of stored block (total, including framing)
checksum u32 FNV-1a of original uncompressed bytes
_pad2[4]
[u8: bytecode_len]
[bytecode_len bytes: the approximate program]
[u64 le: gz_delta_len]
[gz_delta_len bytes: gzip-compressed residual delta]
Reconstruction: vm_execute(bytecode)[i] +% delta[i] == original[i] — always exact, wrapping arithmetic, no clamping.
Cross-compiled to big-endian s390x and ran under QEMU. All three architectures (x86-64 LE, s390x BE, aarch64 LE) produce FNV-1a checksum 0x5757ceb1 for the same 26-byte program.
The key invariant: all multi-byte operands in the ISA are decoded with std.mem.readInt(..., .little) — never via raw pointer casts that would be affected by host endianness.
45 tests across all modules, run with zig build test:
| Module | Tests |
|---|---|
translator.zig |
L1 norm, qualifies-for-approx, entropy, applyDelta, exact match, approx match (dirty 16×16 + 25% corruption), fallback, isCandidate |
math_gen.zig |
XorShift32 frozen sequence, zero-seed remap, value noise determinism, fractal noise, warp, level remap, cellular automaton |
vm.zig |
All 11 opcodes, determinism across calls, error paths (missing HALT, bad opcode, truncation, slot range, dimension mismatch) |
container.zig |
math/fallback round-trips, mixed entries, bad magic rejection, STORE guard (gzip wins / guard fires / never inflates), MATH_RESIDUAL round-trip |
main.zig |
C-ABI integration, pack_demo all five benchmark programs, gzip helper |
| File | Lines | Role |
|---|---|---|
src/math_gen.zig |
~340 | Integer PRNG, lattice noise, domain warp, level remap, cellular automata |
src/vm.zig |
~400 | Bytecode interpreter, 11-opcode ISA, Builder assembler |
src/translator.zig |
~380 | Entropy gate, iterative math search, L1 tracking, delta compilation |
src/container.zig |
~700 | .math format: in-memory Builder, streaming StreamingBuilder, Reader, 4 extraction paths |
src/abi.zig |
~60 | C-ABI export, per-call arena, error codes |
src/main.zig |
~580 | CLI modes, benchmark, pack demo, recursive directory walker |
src/mathfs.zig |
~480 | Live read-only FUSE VFS: on-demand chunk decode, concurrent + adaptive prefetch, bounded RAM cache (zig build tools) |
src/vfs_runner.zig |
~230 | C-ABI live-VFS runner: lossless verify + per-asset decode throughput |
src/concread.zig |
~60 | Concurrent read benchmark (parallel-decode scaling) |
examples/*.mpc |
6 files | Pre-built example programs (26–40 bytes each) |