shrimp

A binary file compressor and inspector, written in Zig.

Status: early development. Compression picks the smallest of four block encodings — raw, bit-level run-length encoding, Huffman coding, or greedy LZ77 matches (with one-position lazy lookahead) coded through two Huffman tables (DEFLATE-lite) — per 256 KiB block, wrapped in a checksummed .shrimp container. Code-length tables are RLE-packed, so small blocks and sparse alphabets stay cheap. See ROADMAP.md for the plan and scripts/bench.sh for a benchmark against gzip.

Requirements

• Zig 0.16.0
• raylib (GUI only): brew install raylib, or pass -Draylib-prefix=<prefix> to zig build (default /opt/homebrew/opt/raylib)

Build

zig build          # installs zig-out/bin/shrimp and zig-out/bin/shrimp-gui

GUI

zig build run-gui              # or: zig-out/bin/shrimp-gui [file]

A single-window raylib app: drop any file onto it for an instant analysis (entropy, byte histogram, bit-run stats, exact predicted compressed size), then compress or decompress in place — outputs land next to the source file (<name>.shrimp, or the name minus .shrimp, never clobbering an existing file). Every compress is verified with an in-memory round-trip. .shrimp files get a container view with checksum status. --smoke [file] runs a headless two-frame self-test including the compress/decompress action.

Usage

shrimp compress   <input> <output>   # e.g. shrimp compress fixtures/hello hello.shrimp
shrimp decompress <input> <output>   # verifies the checksum while decoding
shrimp inspect    <input>            # works on plain files and .shrimp containers

inspect on a plain file shows a hex dump, byte histogram, Shannon entropy, bit-run statistics, and the exact size shrimp compress would produce. On a .shrimp file it shows the container breakdown and verifies the checksum.

Test

zig build test

The .shrimp format (v4)

header:  "SHRM" | version:u8 | original_len:u64le | crc32:u32le
block:   type:u8 | raw_len:u32le | payload_len:u32le | payload

• Data is processed in 256 KiB blocks; each block uses whichever encoding is smallest, so a .shrimp file never inflates its input beyond small fixed headers.
• Block types:
  • raw (0) — bytes verbatim.
  • rle (1) — bit runs across the whole block: a starting-bit byte followed by alternating run lengths (0 = empty run, continues runs past 255 bits).
  • huffman (2) — canonical Huffman codes: num_syms:u16le, mode:u8, then the code-length table (see below), then the MSB-first packed codes. Code lengths are capped at 15 bits.
  • lz77 (3) — greedy hash-chain matches (lengths 3–258, distances up to the whole block) through two canonical Huffman tables (literal/length and distance, DEFLATE-style), with the MSB-first token stream after them.
• Code-length tables come in two modes (mode:u8):
  • 0 — num_syms raw length bytes.
  • 1 — table_bytes:u16le, then 2-byte ops over the symbol sequence: {0x00, len} a symbol of len bits, {0x01, n} skip n zero-length symbols, {0x02, n} repeat the previous length n times. Trailing zero-length symbols are implicit. A table is stored packed only when that is smaller than the raw form.
• crc32 (CRC-32/ISO-HDLC) covers the uncompressed data and is verified during decompression. v1–v3 files (64 KiB blocks, raw length tables) remain readable.

Layout

• src/bits.zig — shared MSB-first bit writer/reader
• src/rle.zig — bitstream RLE encoder/decoder
• src/huffman.zig — canonical Huffman encoder/decoder (any alphabet size), with packed code-length tables
• src/lz77.zig — hash-chain LZ77 encoder/decoder with lazy lookahead (DEFLATE-lite, 256 KiB window)
• src/format.zig — .shrimp container (block selection, compress/decompress, file-level compressFile/decompressFile)
• src/inspect.zig — file analysis: entropy, histograms, run stats, analyzePath shared by CLI and GUI
• src/main.zig — CLI entry point
• src/gui.zig — raylib GUI
• scripts/bench.sh — benchmark vs gzip
• fixtures/ — test fixtures (a compiled Mach-O binary, its source, text)