Conversation
first checkpoint of the experiment to replace the libzstd crate with a pure codec. two layers, both tested end to end. the bitstreams are zstd's two shapes: a backward reader/writer (entropy payloads pack lsb-first with a sentinel above the last real bit, and the reader walks down from just below it) and a plain forward reader for table descriptions. the writer records reads in decode order and reverses, so a round trip through it is the test. fse is the entropy coder both the decoder and the encoder turn on: the reference table-build (spread symbols by the fixed stride, derive each slot's bit count and base from its occurrence), a decoder, and an encoder that walks a symbol sequence backward choosing states. the proof is a full round trip — encode a real sequence, decode it back through an actual backward stream — on both a uniform and a skewed distribution. next: xxhash64, frame and block parsing, huffman literals, the sequence fse tables, and lz77. the crate stays on main; this builds beside it so the two are directly comparable when the encoder lands.
a module constant set to a hex or binary literal silently became zero. the emitter passes an integer literal's source text into the ir unchanged, and the consumer parsed the global initializer with a decimal-only str::parse whose failure it swallowed with unwrap_or(0) — while the identical literal written inline went through parse_i64_operand and compiled correctly. so MAGIC := 0xFD2FB528 was 0, with no diagnostic anywhere: a wrong value that type-checks, links, and runs. found writing a frame parser whose magic number stopped matching. the global initializer now uses parse_i64_operand, which accepts the same 0x/0b/0o forms as every other numeric operand and reports malformed input instead of coercing it — the rule #552 established for the rest of the ir.
xxh64 is the checksum a zstd frame carries over its decompressed content, so a decoder needs the whole 64-bit hash to verify a frame. it is wrapping 64-bit arithmetic throughout, which pith's Int supports directly: * and + wrap rather than trap and bits.shr is logical, so the bit patterns match the reference exactly. the published empty-string vector pins it. the frame layer parses the magic, the variable-length header (content size, window descriptor, dictionary id, checksum flag), and the block headers, and decodes raw and rle blocks. compressed blocks fail with a clear message until the entropy layers land. that is already enough for real interop: the decoder reads frames produced by the system zstd binary from incompressible input — which zstd stores as raw blocks — and reproduces the original bytes, with the frame's own checksum verified through our xxh64. two independent implementations agreeing on a real frame is the only proof that counts.
zstd describes a literals huffman table by per-symbol weights rather than code lengths: weight w means a code of (max_bits + 1 - w) bits, 0 means absent, and the last symbol's weight is inferred from the gap left in the probability space — which is also the validity check, since that gap has to be a power of two. the decode table is flat, so decoding is peek the table log, index, consume the entry's length; the bitstream gained peek and skip for exactly that. build_codes derives each symbol's canonical code from the finished decode table rather than recomputing it, so an encoder and the decoder cannot disagree by construction — the round-trip tests encode with those codes and decode back, on a uniform and a skewed table. also here: the direct (4-bit packed) weight description and the literals section header, whose size fields sit at bit offsets that vary by block type and size format across five layouts. the fse-coded weight description and the four-stream jump table come with the sequence layer next.
the normalized counts an fse table is built from travel in a variable-width forward bitstream: the field narrows as the remaining probability space shrinks, a stored value is biased so zero can mean the 'less than one' probability, and a zero count is followed by a repeat field so runs of absent symbols cost two bits each. this reads that, and writes it — the writer exists so the reader can be round-tripped without a reference frame, and the encoder will need it anyway. writing the round-trip test found a real gap in the reader: the forward bitstream yields zeros past the end of its data rather than refusing, so a truncated description decoded as a run of low-probability symbols that balanced perfectly against the accuracy log while reading bits that were not there. it now fails when the description consumed more than the data held. two zero bytes was the case that caught it. this is the last shared piece before sequences: the same reader serves the huffman weight description and all three sequence tables.
a compressed block is literals plus sequences — copy N literals, then copy M bytes from D back — with the three code streams fse-coded and interleaved into one backward bitstream carrying three live decoder states. this adds the baseline and extra-bit tables, the three predefined distributions, the mode byte and its table descriptions, the interleaved decode loop, offset resolution against the rolling repeat history, and lz77 execution that copies byte-at-a-time so a match can overlap itself. the decoder now reads real zstd output: sixteen frames across five input shapes and levels 1, 3, 9 and 19, byte-identical, checksums verified. the four that do not are all huffman-coded literals, the one documented gap. two bugs only a real frame could have found, both of which my round-trip tests were structurally unable to catch: the fse table gave each symbol's next-state a run starting at one, with floor-log2 miscomputed as a bit length. the encoder built its codes from the same wrong table, so the two agreed perfectly — self-consistent and wrong. the spec starts each symbol's run at its own normalized count, which is what pushes a frequent symbol's states into the table's cheap half. and the predefined match-length distribution had four low-probability symbols where the spec has seven. it summed to sixty-four either way, so the sum invariant — the only property a synthetic test can check without a reference — passed happily. the shape test now pins the counts and the low-probability boundaries, not just the total.
huffman literals close the last gap: fse-coded weight descriptions decoded by two interleaved states, and the four-stream layout whose jump table gives the first three lengths while the output splits into four segments. treeless blocks reuse the previous block's tree. widening the corpus past the first happy cases found two more bugs, both only reachable across block boundaries. repeat mode for the sequence tables was unimplemented — a later block naming the tables an earlier one described. and the repeat-offset history lives for the whole frame, not the block; resetting it per block produced output of exactly the right length with the wrong bytes, which the frame checksum caught and nothing else would have. both are frame-level state now, threaded alongside the huffman table. thirty-six frames from the system zstd binary decode byte-identical: nine shapes — prose, source, markdown, random binary, structured json, wide alphabet, empty, one byte, 100 bytes — across levels 1, 5, 12 and 19, covering raw, rle and compressed blocks, direct and fse-coded huffman weights, one and four streams, every sequence mode, and multi-block frames. make zstd-pure-check pins eighteen of them as a ci gate. measured decode throughput against the linked crate on the same frames: 3.4 MB/s vs 437 MB/s on prose, 17 vs 3178 on a wide alphabet, 42 vs 5449 on incompressible input — call it 130-190x slower. that is the honest cost of bit-level work in a language with no inline assembly and arc-managed containers, and it is the number the decision should be made on.
append bytes copied from the buffer's own contents — the back-reference every lz-family decoder emits. when the source range overlaps the destination the copy repeats, which is not a bug to work around but the encoding itself: a run of a hundred identical bytes is one short match. the runtime copies forward a byte at a time in that case and uses a bulk extend when the ranges are disjoint. zstd matches were reading and writing one byte per position through separate runtime calls, which is what motivated this.
extracting n bits walked the stream one bit per iteration, each with a divide and a modulo, so an eleven-bit huffman code cost eleven iterations. it now loads the bytes the field spans as one word and shifts. on top of that the backward reader caches a sixty-four bit window. reads walk downward, so one window serves about fifty bits before it has to be refilled, and a field read becomes a shift and a mask against a value already in hand. the empty-window sentinel sits above every valid bit index rather than below it: the test is `at < window_base`, and a short stream's reads all fit under sixty-four bits, so a negative sentinel would let them read an all-zero window instead of refilling. the unit tests caught exactly that while every real frame still decoded.
a compressed block used to build its own byte buffer and then be appended to the frame's, and it took a copy of everything decoded so far so matches could reach back into it. blocks now decode straight into the frame buffer: a match into an earlier block just indexes further back. literal runs are written as one slice instead of a byte at a time, and matches go through copy_within. the fse and huffman decode tables hold one packed Int per entry rather than a struct. decoding a sequence indexes those tables six times, so a struct entry meant six heap handles and six reference-count round trips per sequence. read_ncount caps the accuracy log at 22, which keeps base under 2^22 and the packed fields from colliding. resolving an offset against the repeat history now updates the history in place. it ran once per sequence and returned a fresh three-element list each time, which cost more than everything else in the loop put together. the two implementations that briefly existed are back down to one, so the rules can't drift apart.
builds its corpus from the repo — docs, std sources, generated log lines — so there are no binary fixtures in tree and the shapes stay realistic: between 2:1 and 11:1, which is what a decoder meets in a content-encoding or an otlp export. the highly-compressible case is reported last and labelled, not led with. a frame that packs 80KB into 22 bytes decodes almost entirely through the match copy, so it measures memory bandwidth rather than entropy decoding and reads five times better than any real workload. each phase runs to a wall-clock budget rather than a fixed rep count. the kernel decodes some of these in tens of microseconds, and a fixed count lands inside the millisecond clock's granularity and reports its quantization as though it were throughput.
a literal-length code needed a baseline from one list and an extra-bit count from another; the two are now packed as baseline * 256 + bits, so the decode loop indexes once per code instead of twice. the length bounds check reads a hoisted length rather than calling len() every iteration. the packing is written out by hand, so the test pins it against the format: every fixed baseline and width, plus the invariant that each baseline sits exactly one past the span its predecessor covers. that last check is what matters — a mispacked entry leaves some length unreachable and another ambiguous, and it caught a wrong value in the first draft of this table. worth about 1.5% on its own. the interesting part is what it ruled out: a list index measures 18ns in isolation but only about 2ns of marginal cost here, because the surrounding loop has enough independent work to overlap them. isolated per-operation costs overstate what removing an operation actually buys.
the stats hooks sit on the hottest runtime entry points — every list get, byte-buffer write, and struct alloc probes whether PITH_PERF_STATS is set. that probe went through a OnceLock, which showed up as a measurable slice of a list index. it is now a three-state atomic with an inlined accessor and a cold probe path, so with stats off the cost is a predictable branch on one relaxed load.
three per-operation extern calls become inline code. xs[i] called pith_list_get_opt, which heap-allocated a two-slot optional tuple the caller immediately unpacked and released. a conservative pre-scan proves a call's result is consumed only by that exact unpack pattern — any other use of the register disqualifies it — and rewrites the call, the allocation, and the release into inline loads with null, magic, element-size, and bounds checks. a failed check still produces is_some == 0, so the loud out-of-bounds error is unchanged. escaping optionals from .get(i) keep the real call. xs[i] = v on primitive lists gets the matching store fast path; tagged lists and out-of-bounds writes fall back to the runtime call so retain, release, and the oob-is-a-no-op contract stay where they were. bits.band(x, y) compiled to a call to the std.bits wrapper whose body called the ffi extern — two call layers per bit operation, and the sequence decoder does about thirty per sequence. a pass now structurally verifies a two-parameter function whose body is exactly load, load, call bit builtin, ret, and aliases calls to it (and to the builtins directly) to the single native instruction. shr stays logical, matching the extern. an edited wrapper stops matching and keeps its calls; local shadowing is respected. the wrapper detector and the unpack scan have unit tests. decoding throughput on the pure zstd corpus roughly doubles from these two files alone; the full regression corpus, the leak gates, and both zstd interop checks pass unchanged.
the sequence and literal decode loops now mirror the bit reader's window in locals and do their peek, read, and skip as shift and mask inline, going back through the reader only on a window miss and at the stream tail. resolve_offset is likewise inlined over three local repeat variables, written back after the loop; the named function and its tests are unchanged, and a comment ties the two copies together. the three predefined fse tables are built once and shared behind a mutex instead of rebuilt per block — repeat mode already established that a borrowed table is safe. build_decode_table pushes entries instead of zero-filling and assigning. the readable helpers stay for every cold path; this trades inline arithmetic for calls only in the two loops the profile says dominate.
three new builtins, each replacing a per-byte call loop with one call: write_range appends a range of another bytes value without materializing an intermediate slice; write_word appends up to eight bytes packed in one integer; read_word is its mirror, a little-endian load of up to eight bytes. out-of-range and bad counts return failure or zero rather than partial work, and callers that rely on zero-past-the-end tolerance check bounds themselves. copy_within's overlapping case is also reworked: an overlapping range means the output repeats with period (len - src), so each chunk copied is a whole number of periods and the chunk doubles as the copy proceeds — a long run costs a logarithmic number of memcpys instead of a byte loop. run-heavy zstd frames decode an order of magnitude faster for it. the bootstrap seed is regenerated for the new checker builtins; the bootstrap fixed point verifies.
a word load from a bytes value is one 8-byte load and a mask when the whole word is in bounds; the call survives only for the tail and for bad counts, keeping the runtime's edge-case behavior word for word. the bitstream refill and the xxhash stripe loop sit on this, so the call overhead was the dominant cost of both.
decode_and_execute decodes each sequence and runs it straight into the frame buffer. the list of sequence structs that used to sit between the two loops cost more to build and read back than the arithmetic that produced it; the fused loop never materializes it. decode_sequences and execute_into stay as the separately-testable reference forms — the fused body mirrors both, and a comment ties the three together. literal runs, trailing literals, and raw blocks write through write_range instead of slicing first. huffman literals decode eight symbols into a packed word and append it with one call instead of eight. the backward reader's window refill and the forward extract load words with read_word, as does the xxhash kernel — which also hoists its primes and collapses its rotates, taking the checksum from half of a checksummed frame's decode time to a twentieth.
a performance.md section with the three-pass optimization arc — where the time went, which fixes were decoder-side and which the compiler now applies to every program — and the two measurement lessons worth keeping: isolated microbenchmark costs overstate marginal cost about 5x, and a representation change has to price the read side, not just the write. bench/README.md gets the corpus and methodology notes, including why the run-heavy case is reported but not treated as the headline.
compress() emits real zstd any decoder reads: raw and rle blocks split at the format's 128kb cap, huffman-coded literals with exact length-limited codes (package-merge, so the kraft sum is exactly one — the implied-last- weight description demands it), and lz77 sequences over the predefined tables with repeat-offset codes. the greedy finder hashes 4-byte prefixes frame-wide, so matches reach across block boundaries. huffman weight descriptions are fse-coded when that pays, which is what unlocks literals for alphabets past symbol 128. compress() never fails — any internal error falls back to a stored frame — and compress_checked() is the loud form the tests and gates use. every wire decision is derived from the decoder's own tables: codes come from build_codes over the decode table the weights build, fse state paths from the decode tables the counts build, repeat-offset candidates are validated by running the decoder's resolve_offset. the two sides cannot drift apart, and the spec anchor is external: make zstd-encode-check requires the system zstd binary to decode every pith-compressed frame byte-identical, because a round trip through our own decoder has twice been self-consistent and wrong. sizes: within 1.16x of zstd -3 on text, parity on runs and incompressible input, 1.44x on a megabyte of source; ~4 mb/s encode. remaining size levers, documented in the module: custom fse descriptions for the sequence streams and lazy matching.
the example now proves interop in both directions — the reference codec reads a pure-pith frame and the pure decoder reads a reference frame — rather than a round trip through our own code. performance.md gets the encoder's standing next to the decoder story.
the benchmark only ever timed decoding, so the encoder's throughput was undocumented and its size claims came from a different corpus than the one the performance notes cite. it now times both directions over the same shapes: the pure encoder's frame is decoded by the kernel and checked against the input before timing, and the reported size is a percentage of the kernel's own output so throughput cannot be bought by compressing worse. renamed to bench/zstd_codec.pith to match what it measures. the numbers correct an overstatement in the previous notes. sizes are within 1.15x of the kernel on small text but 1.39-1.88x on prose, source, and json — the earlier "parity on runs and incompressible input" came from encode-check shapes that degenerate to rle frames, not from these. encode runs 30-65x slower than the reference. the decoder earned its numbers over three passes; the encoder has had none, and the docs now say so.
sizes come down from 1.15-1.88x the reference codec's output to 1.01-1.19x, and throughput rises 2.5-3.8x. both metrics move together; neither was traded for the other. most of the size came from per-block fse tables. all three sequence streams used the predefined distributions, which cost most on data whose histogram is nothing like them — json logs were the worst at 1.88x. the encoder now histograms the codes a block actually uses, normalizes to the accuracy log (rare symbols to the "less than one" slot, leftovers by largest residual), and emits a table description only when the estimated bits beat predefined by more than the description costs. the finder also prefers the previous offset, since a repeat code costs less than an offset's magnitude, and holds a match until the next position has been checked for a clearly longer one. most of the speed came from writing bits forward. the backward bitstream writer recorded every field as a value and a width on two lists and packed them on a second pass; a packer that flushes four bytes at a time into the buffer runs the same loop once. per-sequence scratch histories and their failable calls are gone, the fse state walk computes its slot from the table's tiling instead of scanning for it, and the finder's hash table is sized to the input rather than always 65536 entries — which was 65536 pushes to encode eight kilobytes. the packer landed with a test asserting it packs byte-identical to the recording writer over 400 varied fields across every flush boundary, before anything switched to it.
sizes 1.01-1.19x of the reference codec (was 1.15-1.88x), throughput up 2.5-3.8x, and the profile's largest cost is now the match finder rather than allocation churn. records where the remaining size gap actually is: fewer sequences rather than cheaper ones.
the encoder's spec anchor is the system zstd binary decoding every frame we produce; that gate only protects the tree if it runs on every change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
zstd, both directions, in pure pith — no c, no crate in the path.
std.compress.zstd_pure_*decodes and encodes real frames: raw, rle, and compressed blocks; huffman literals in one and four streams; fse-coded sequences with repeat offsets; xxh64 content checksums.the correctness anchor throughout is the system zstd binary, not our own round trip. twice while building this an encoder and decoder agreed with each other and disagreed with the spec — the fse next-state offset, and the predefined match-length distribution, both of which are invisible to a round trip because the same wrong table sits on either side. so
make zstd-pure-checkrequires our decoder to read 18 shapes of system-produced frames byte-identical, andmake zstd-encode-checkrequires the system binary to read every frame we produce, across 9 shapes plus a 60-shape seeded differential run. both now gate ci.where it lands, against the crate-backed kernel on a corpus built from this repo (
make zstd-pure-bench, 2:1 to 11:1):the decoder started at 129x the kernel and took three passes to get here; the run-heavy row is faster than the kernel because an overlapping match copy became a period-doubling memcpy instead of a byte loop. the encoder has had one pass and produces frames within 1-7% of the reference codec's size on real content.
three of those passes were not really about zstd. the hot loops kept hitting per-operation overhead that every pith program pays, so the fixes landed in the compiler and runtime:
xs[i]used to call into the runtime, heap-allocate an optional tuple, unpack it and release it — the ir consumer now collapses that whole pattern to inline loads keeping the same loud out-of-bounds failure;bits.bandcrossed two call layers and is now one native instruction; a word load from bytes inlines to an 8-byte load and a mask; the disabled perf-stats probe became a single relaxed load. new runtime primitives —write_range,write_word,read_word, and a chunkedcopy_within— replace per-byte call loops for anything doing bulk byte work.what was tested
every gate below was run on the final tree, and the two subagent-produced passes were independently re-verified rather than taken on report:
make zstd-pure-check: 18 system frames decoded byte-identicalmake zstd-encode-check: 9 shapes read back by both our decoder and the system binary; plus a 60-shape seeded differential runmake zstd-interop-check: both directions at levels 3 and 19make run-regressions-only: 303 passed — the compiler changes touch every program, so this is the one that matters mostmake leak-check-only: flat across 200k and 800k roundsmake bootstrap-verify: fixed point verified (the seed changed for the new checker builtins)cargo test -p pith-codegen: the new inline passes have unit tests for their pattern detectorsnotes
bench/zstd_decode.pithis nowbench/zstd_codec.pithand times both directions. it hands the pure encoder's frame to the kernel's decoder before timing, and reports output size as a percentage of the kernel's, so encode throughput cannot be bought by compressing worse.decode_and_executeis the fused hot path;decode_sequencesandexecute_intoremain as separately-testable reference forms with a comment binding the three. the hand-built sequence tests they support are the module's spec anchor and were kept deliberately — a round trip cannot check them.compress()call. it predates this pass, which halved it by allocating less. worth its own change before anything depends on the encoder in a long-lived process.docs/performance.md: the encoder's match finder is now 31% of its profile, the last of the size gap is fewer sequences rather than cheaper ones, and sequence-table repeat mode is unimplemented.