The separate Glasses Lossless Codec v0.1 experiment—shared token
supersequences plus per-record visibility masks—is documented in
GLASSES_CODEC.md. It includes a binary .glc format,
encode/decode/benchmark/analyse CLI, synthetic generators, diagnostics and tests.
It is also registered in the Windows Compression Lab as Glasses supersequence
v0.1 (custom). Selecting that row reveals remembered line/sentence segmentation
and 1–64 MiB block-size settings. The app can benchmark it, export its detailed
byte-accounting report, include it in pipelines, and save or restore it through
the normal self-describing .clab workflow.
A dependency-free Python workbench for comparing compression algorithms and prototyping your own byte-oriented codecs.
This project requires Python 3.10 or newer. It has no third-party package dependencies; Tkinter is included with the standard Windows installer from python.org.
On Windows, double-click run_compression_lab.cmd. The portable launcher looks
for the standard Python launchers first and can also use Codex's bundled Python
when the project is opened in Codex. It does not contain a user-specific path.
You can also run the app from a terminal:
python compression_lab.pyPaste text or open any file, choose how many timing repetitions to run, and click Run benchmark. The table reports compressed size, compression ratio, space saved, median compression/decompression time, and round-trip correctness.
Click Load Huffman article to download only the readable plain text from the Wikipedia Huffman coding article. The text is shown in the editor before benchmarking, and requires an internet connection. Wikipedia content is available under its stated content license; retain attribution if you redistribute the downloaded text.
Click Load all-hit sample for a generated 256 KiB repeated-pangram test. It
is deliberately artificial: after learning, the split-Huffman model should
report zero literal misses and zero backoffs. The app sets timing repeats to one
because retraining every learned codec on this larger input takes longer. The
same data is supplied as all_hit_sample.txt. Benchmarks run on a background
thread, with a codec-by-codec progress bar and a prominent label naming the
codec currently running, so this larger comparison can finish without making
the window appear frozen. The editor displays only a 4 KiB preview; the full
256 KiB sample remains in memory and is used by the benchmark.
Files opened with Open file… use the same preview behavior. Only the first 4 KiB is decoded for display (with invalid UTF-8 replaced safely), while the benchmark receives every original byte unchanged, including binary data. If you edit a loaded preview or article, the app immediately switches the active source back to the editor, and the next benchmark uses its current UTF-8 text.
python compression_lab.py --text "hello hello hello" --repeat 10
python compression_lab.py --file .\sample.bin --repeat 5
python compression_lab.py --file .\sample.bin --json
python compression_lab.py --huffman-article --repeat 5
python compression_lab.py --all-hit-sample --repeat 1In build_codecs(), append a Codec. Both functions accept bytes and must
return bytes:
Codec(
"My codec",
my_compress,
my_decompress,
"A short note about the experiment",
)The app catches failures per codec and always checks that decompression exactly reproduces the original input. Built-in comparisons include RLE, zlib/DEFLATE, gzip, bzip2, and LZMA.
Follower 2-bit (custom) implements the context idea explored in this project.
After each lowercase letter, its three predicted followers use 00, 01, and
10. Code 11 is an escape followed by the literal 8-bit byte, which makes the
format lossless for text and arbitrary binary data. The first byte is literal;
the decoder then selects each new prediction table from the previous decoded
byte. A compact variable-length header records the original byte count so that
padding at the end is unambiguous.
Follower 1-bit-first (custom) uses the same prediction arrays for a controlled
comparison, but assigns prefix codes 0, 10, and 110 to the three predicted
followers. A miss is 111 followed by the literal byte. The first-ranked hit is
therefore cheaper than in the 2-bit codec, while third-ranked hits and misses are
one bit more expensive. The benchmark reveals which side of that trade-off wins
on the selected input.
Follower 2-byte sequences (custom) learns four common two-byte continuations
for profitable one-byte contexts. A pair-table hit uses 0xx: three bits emit
both bytes, averaging 1.5 bits per output byte. Code 1 falls back to the fixed
single-character predictor. The learned tables are embedded in the compressed
stream, and their full storage cost is included in the reported size.
Selective top-1 transform (custom) is intended as a pipeline preprocessor.
For each transition, control bit 1 means the most likely follower was matched
and the byte is omitted; bit 0 means the original byte is copied unchanged to
a separate literal stream. A hit therefore costs one bit and a miss effectively
costs nine bits. The output stores packed control bytes first and untouched
literals second, giving a following zlib, LZMA, or other codec two more regular
regions to compress. Its standalone result is expected to be worse on inputs
with too few top-prediction hits; the important comparison is the complete
pipeline against the second codec by itself.
Adaptive context arithmetic (custom) performs genuine fractional-bit entropy
coding. It maintains a probability table for each preceding byte and updates it
as the stream is processed. A context emits an escape into a global adaptive
byte model the first time it encounters a follower. The decoder starts with the
same empty context models and repeats every update, so no dictionary needs to be
stored in the file. Highly predictable followers can approach a fraction of one
bit each over a long input, while uncommon followers consume more of the shared
arithmetic-coded range.
Phrase dictionary + Huffman (custom) learns repeated byte sequences from 3 to
16 bytes long, including word fragments, whole words, and short multi-word
patterns. It retains at most 128 candidates whose estimated token savings pay
for their dictionary storage, tokenizes with longest matches, prunes entries
that are not actually used profitably, and canonical-Huffman-codes the combined
literal/reference token alphabet. Dictionary entries, token code lengths, and
all framing are included in the compressed size. Phrase discovery happens
before Huffman coding so similar source strings remain visible to the learner.
For files above 2 MiB, discovery samples the beginning, middle, and end to bound
memory and training time; encoding and round-trip verification still use the
complete file.
Phrase dictionary + arithmetic (custom) uses the same learned phrases and
longest-match tokenization but removes the Huffman stage entirely. Literal bytes
and phrase-reference IDs form one adaptive context-arithmetic alphabet, allowing
frequent references to average fractional bits per token. The serialized phrase
dictionary can be stored raw, compressed by the custom adaptive arithmetic
coder, compressed by zlib, or automatically assigned to the smallest of those
three representations. Forced modes intentionally retain their real cost even
when they grow the dictionary, making their individual contribution measurable.
The result
includes the complete dictionary, model mode, token count, payload length, and
all framing. Its before/after figure compares phrase-Huffman with the integrated
phrase-arithmetic representation on the same input.
Selecting that codec in the desktop comparison reveals its remembered tuning panel. Controls cover maximum dictionary entries, minimum/maximum phrase length, minimum occurrences, training sample size, arithmetic context order 0–8, a bounded exact-size pruning pass, and auto-tuning across three dictionary sizes and all nine context orders. Use custom arithmetic token coder can be cleared to store the token stream at a fixed number of bits per token instead. This remembered switch applies to both phrase-arithmetic codecs; select Raw only for dictionary storage as well if no custom arithmetic coding should be used anywhere in the result. Exact pruning tests up to 16 weak entries against the complete framed size; auto-tuning is more CPU-intensive and is off by default. The chosen token mode and arithmetic order are stored in the compressed stream.
Phrase dictionary + PPM arithmetic (custom) keeps the same phrase learner,
tokenization, dictionary modes, and tuning controls, but changes how token
probabilities are chosen. It first tries the configured longest context. When
that context has not seen the next token, it emits an escape and tries each
shorter context in turn before falling back to the global token frequencies.
This is a PPM-style hierarchical backoff experiment; it deliberately remains a
separate codec so its size and speed can be compared directly with the original
single-context phrase-arithmetic codec. Auto-tuning evaluates orders 0–8 using
the PPM output's complete framed size.
Arithmetic token coder only (custom) applies the phrase codecs' ordinary
adaptive arithmetic token engine directly to the original byte values 0–255.
It uses the shared arithmetic order 0–8 setting but has no phrase learner,
dictionary, PPM backoff, zlib, or other secondary compressor. This provides a
clean complete-size baseline for measuring what the arithmetic layer achieves
on its own. Its explanation report automatically compares all nine orders and
shows complete sizes, context hits, escapes, context-model counts, and a
single-run timing diagnostic.
Word references + zlib (custom) counts exact ASCII letter-words using the
configurable minimum length and occurrence thresholds shown in the desktop UI
(both default to 6). The first qualifying occurrence is emitted as an in-stream
definition; later occurrences become compact global IDs, so there is no separate
copy of the word dictionary. The transformed token stream is compressed with
zlib level 9 and compared with zlib on the untouched input. A one-byte mode flag
selects the smaller representation, preventing a bad word transform from causing
more than one byte of regression. Selecting the result reports unique entries,
repeat references, referenced source bytes, and the zlib-alone comparison.
Follower 2-level (custom) adds 128 corpus-selected two-letter contexts plus the
experimental an → t/o/d array. A two-letter hit
costs 2 bits. Code 11 falls back to the one-letter table, where a hit costs 2
more bits; another 11 introduces an 8-bit literal. Contexts absent from the
sparse two-letter table go directly to the one-letter model without spending a
fallback code. The contexts are retained only when their corpus-weighted savings
beat the fallback penalty. Keeping both variants in the benchmark makes the
trade-off measurable instead of assuming that a more complex model is always
better.
Follower learned (custom) retrains on every input. It counts the followers for
each two-byte context, proposes a top-three array, and retains an override only
when its measured bit saving exceeds the five bytes needed to store that array
plus a safety margin. The learned arrays and their count are embedded in the
compressed stream, so decompression is self-contained and their overhead is
included in every reported size. Selecting its benchmark row shows the learned
array count and model size. The Learning: before → after column compares the
static two-level result with the trained rerun on exactly the same input; select
the row to see the percentage improvement or regression.
Follower learned 3-context (custom) adds a learned three-byte context layer.
It tries a triple array first, then learned pair arrays, the fixed one-byte
table, and finally a literal. Pair entries cost five model bytes and triple
entries cost six; both are selected only when their measured savings exceed
their serialized cost. Its before/after comparison uses the learned pair codec
as the baseline, isolating whether the additional triple layer helped.
Follower 3-context + residual zlib keeps the same learned triple/pair model
but splits its output into a packed prediction-control stream and a literal
residual stream. The residual bytes are compressed together with zlib level 9.
The control-stream length, zlib wrapper, learned arrays, and all other headers
are included in the reported result. Its before/after comparison uses the plain
learned three-context codec, isolating the benefit of residual compression.
Follower 3-context + zlib model/residual additionally compresses the serialized
learned-array block. It stores a mode byte and block length, uses zlib level 9
only when that is smaller than the raw arrays, and otherwise falls back to raw
model storage. Its before/after comparison uses the residual-zlib codec, so the
display isolates the value of compressing the arrays themselves.
Follower 3-context + Huffman streams is a zlib-free entropy-coding experiment.
It turns every consulted prediction result into a symbol (0, 1, 2, or
fallback 3) and canonical-Huffman-codes that token stream. It separately
Huffman-codes the literal bytes and serialized learned model. Every block stores
its uncompressed length, sparse code-length table, encoded length, and payload.
Its before/after baseline is the best zlib model/residual variant, making it
clear whether the custom Huffman stage wins after all codebooks are counted.
Follower 3-context + split Huffman separates the prediction outcomes into
triple-, pair-, and single-level streams. Each gets its own four-symbol canonical
Huffman table, allowing different rank/fallback distributions at each level.
Model and literal blocks remain independently Huffman-coded. Its before/after
baseline is the combined-token Huffman codec, so the reported change isolates
the value of conditioning outcome probabilities by model level.
Follower 3-context + RLE/Huffman run-length-encodes each prediction-level
stream before canonical Huffman coding. Each level independently compares the
fully framed raw-Huffman and RLE/Huffman blocks and stores a one-byte mode flag
for the smaller representation. This targets long runs of identical prediction
ranks while retaining ordinary split Huffman on mixed streams. Its before/after
baseline is the split-Huffman codec.
The benchmark exposes prediction Hits, pair-table Backoffs, and Literal misses for the custom codecs. All-hit size is the theoretical packed size with literal misses replaced by the cheapest prediction. Fixed-width codecs use 2 bits per predicted byte; the 1-bit-first codec retains the observed rank costs of 1, 2, or 3 bits. The calculation still includes the original-length header, first literal byte, and byte-alignment padding. Select a result to see its hit rate and theoretical percentage saving.
The Use column in Compare codecs controls which codecs a benchmark runs.
Click its ☑ or ☐ value to toggle a codec. Disabled codec names remain visible
but are skipped unless Hide unchecked codecs is selected. The enabled codecs
and this display preference are saved in .compression_lab_settings.json in the
current user's home folder so they survive app restarts. Newly introduced codecs
start enabled by default. Select a codec row to show only that codec's controls in
the shared Codec settings area.
Select any codec and choose Export explanation… to create a Markdown report using the current input and settings. Phrase-arithmetic reports include the actual learned phrases and usage counts, token reduction, dictionary storage choice, arithmetic payload and framing sizes, PPM backoff levels and escapes, the measured benefit of each stage, standard-compressor comparisons, and the reverse decompression steps. Phrase-learning advice interprets whether entries hit the length or count limits, relates occurrence thresholds to actual usage, identifies ineffective training-size changes, and explains the likely benefit and cost of each suggested experiment. The detailed report is generated independently of the results table, so it can be exported before running a full benchmark. Every phrase-arithmetic report also calculates four complete framed sizes: no phrases with fixed-width tokens, phrases with fixed-width tokens, phrases with ordinary custom arithmetic, and phrases with PPM-backoff arithmetic. It then states explicitly whether phrase matching, ordinary arithmetic, and PPM backoff helped or hurt. The automatic no-phrases baseline includes framing and token storage, so it exposes the phrase layer's exact total contribution rather than only an estimated saving from individual matches.
The phrase settings can create and load reusable .wphrases.json candidate
models. Create word model… scans the current text for exact, case-sensitive
overlapping word sequences from two through Maximum words per phrase
(default 9), plus complete repeated sentences including final punctuation. It
saves only entries meeting Model minimum occurrences (default 10). Load
word model… makes those entries available to the two phrase-arithmetic codecs;
legacy .wpairs.json v1 files still load. The file is guidance rather than
required decoder state: every candidate is recounted against the current input,
rejected when it is below the configured threshold or unprofitable, and compared
with the ordinary and combined dictionaries using complete compressed size.
Selected entries are embedded in the normal phrase dictionary, so saved .clab
files remain self-contained and do not require the model during decompression.
The remembered Phrase contribution tests in report setting defaults to 25
and accepts 1–256. For each top-ranked phrase within that limit, the report
removes the phrase and recompresses the complete stream. It shows raw phrase
bits, source bits covered, estimated gross bits avoided, and the measured
whole-file bits saved or cost. Larger limits make report generation slower, and
the marginal results are not additive because removing one phrase can change
other matches and adaptive probabilities. Phrase text renders spaces as ␠,
newlines as ↵, carriage returns as ␍, and tabs as ⇥, preventing
whitespace-only entries from appearing blank. When every retained phrase is
whitespace, the advice calls that out and suggests a lower occurrence threshold
for exploring word and fragment phrases.
When PPM backoff grows the output, Step 4 also measures orders 0–8 against the
same final dictionary and recommends the best measured order or the original
non-backoff codec. It explains escape traffic, warns when a high hit share is
still a net loss, separates dictionary settings from backoff costs, and identifies
PPM symbol exclusion as a possible algorithm experiment rather than a current
setting.
Select a codec and choose Save compressed… to create a .clab file from the
current full input. The app compresses and immediately decodes the payload before
saving, refusing to write a codec result that does not round-trip exactly. The
small self-describing container records its format version, codec name, original
filename and byte length, CRC-32 checksum, payload length, and codec bytes.
Choose Decompress file… to open the container, select a restored destination,
automatically run the recorded codec, and verify both length and checksum before
reporting success. The benchmark's compressed-size column measures the codec
payload; the saved .clab file is slightly larger because it includes this
metadata.
Open the Compression pipeline tab to chain codecs. Click the checkbox beside each codec to include it, select codecs and use Move up or Move down to change the order, then choose Run checked pipeline. Every step reports its input and output sizes, step saving, cumulative saving, and round-trip result. The complete chain is decoded in reverse order and compared byte-for-byte with the original input. Reported experimental sizes do not include metadata that a standalone file format would need to record the chosen codec order.
Timing is intended for quick comparisons, not rigorous performance science.
For reliable benchmarks, use larger inputs, increase --repeat, close noisy
background applications, and test several kinds of data.