Repository navigation
Releases: M-Compressors/mzstd
Release list
v0.2.9
-
Session API in
mzstd-ffi. C callers can now compress and
decompress incrementally through reusable handles:mzstd_encoder_create,
_reset(with a pledged size orMZSTD_CONTENT_SIZE_UNKNOWN),
_process(MZSTD_CONTINUE,MZSTD_FLUSH,MZSTD_END) and_free, and
the matchingmzstd_decoder_*functions (MZSTD_DECODE_PROCESS,
MZSTD_DECODE_FINISH, awindow_log_maxof10..=31or0for 27).
Each process call takes in/out lengths and reportsMZSTD_NEEDS_INPUT,
MZSTD_NEEDS_OUTPUTorMZSTD_FINISHED. Encoder frames are
byte-identical toZSTD_compressStream2for the same inputs, output
capacities and operations; the decoder reads concatenated and skippable
frames asZSTD_decompressStreamdoes. Pointers, lengths, overlaps and
operation values are validated before use, and a panic poisons the handle
instead of unwinding into C. A newsessionCriterion benchmark times
both against zstd's streaming API with reused contexts. -
The
fastlevels no longer zero a block's room per dense block.
Promoting a dense full block to the fixed-size sequence cursor resized
the store's vectors to a full block and truncated them again every time,
writing 384 KiB of zeros per 128 KiB block. The store now keeps one
full-block destination, zeroed once, and the promoted block's entries
stay in it until the next block. Silesia nci at level -1 with a reused
context is 5-6% faster (it had fallen 4.5% behind the main branch),
cyclic Alice 2.4-2.9% at levels -5 to 2. Frames are unchanged. -
Benchmarks refreshed on AMD, against C and the main branch. The
published comparison is a new complete sweep of all 384 encoder and 384
decoder cases on a Ryzen 9 7950X, and its pages and charts carry a third
series, the main branch measured beside the working tree
(--main-csvinlevel_docs.py,plot.pyanddecoder_docs.py). A new
silesia.pytimes C, main and the working tree in random order on whole
Silesia files with reused contexts. The earlier Intel sweep and the
interim AMD tuning reports are removed. See the
benchmark results. -
Reused-context and literal-decoding performance without unsafe. The
row match finder views its kept tables a row further on at every context
reset: entries a previous input of similar content left in the
never-cleared tag table otherwise shared the new input's tags, and on
repeated Alice frames the greedy and lazy levels 4-10 took up to 6.5%
longer from finding and dismissing them. The reference salts its hash
instead (ZSTD_advanceHashSalt), at an XOR per hash; the view costs
nothing per position, so cold searches are unchanged. The
decoder runs its block loops with AVX2's features on AVX-512 hosts (the
compiler vectorized the Huffman streams' lookups into gathers there),
decodes large four-stream Huffman sections into a literal buffer of a
fixed size that lets the pair loop keep one output position per stream,
and reads offset bits with one shift. Thefastmatch path extends
backwards with the reference's byte loop, compares a match's first word
with one bounds test and keeps its sequence cursor's counts in
registers: 1.4-1.7% faster on Silesia at levels -1 and 1, still 7-8%
behind the reference there. Encoded bytes remain identical to
zstd 1.5.7. See the benchmark results. -
Alice performance follow-up without unsafe. Dense full
fastblocks
promote their sequence writer after 1024 sequences, lazy prefix parsing
uses relative positions, and sequence code/width maps share one lookup.
The decoder skips zero-width length reads, builds described FSE rows in
the selected feature context, and continues Huffman pairs into the tails
before exact stream-end validation. -
AMD performance tuning without unsafe. The encoder's block
pre-splitter runs in the selected SIMD feature context, uses exact 32-bit
fingerprint arithmetic and keeps its fixed histograms on the stack. The
decoder borrows compile-time predefined FSE tables and symbol rows, avoids
an enum-conversion table lookup, and interleaves three Huffman streams
while decoding the fourth separately. Encoded bytes remain identical to
zstd 1.5.7. -
Fixed a false
framed_roundtripfuzz crash. With one-byte chunks the
target writes a chunk per payload byte plus metadata, which exceeded the
4096-chunk decode limit of the framed targets. The limit is now 8192, and
the ClusterFuzzLite input is kept as a regression. The library is
unchanged. -
Formal verification against RFC 8878, ported from mbrotli. Charon and
Aeneas translate selected Rust items into Lean 4
(scripts/extract_lean.py), andverification/leanproves them correct
against a transcription of RFC 8878 itself, never against the C library:
the frame header (descriptor, window formula, dictionary ID, content size)
and the block header on both sides, with round trips from the encoder's
writers through the decoder's parsers;Number_of_Sequencesin its
shortest form and every form the RFC defines; the literals length and
match length codes, whose encoder picks the only code whose RFC range holds
each length and whose decoder tables read it back; the offset code tables,
the default distributions, the initial repeat offsets and the magic
numbers; the encoder's repeat offset update for every offset value; and the
FSENumber_of_BitsandBaselineof every state (Table 21). Each proof
also shows the translated code cannot panic or overflow on the inputs it
covers. The transcription checks itself against the RFC's worked examples;
RFC 8878's Table 18 misprints its last two rows, and the proofs follow the
RFC's text.ci-lean.ymlchecks that the translation matches the Rust
source and that the proofs build with only Lean's standard axioms. To make
the code translatable,FrameHeader::parsereads its fields by index,
Number_of_Sequencesis read and written by the newsequence_countand
write_sequence_count, and the encoder's frame header is written from a
FrameHeaderFieldsvalue; behaviour and output are unchanged. -
hotpath Cloud with Callgrind, ported from mbrotli. New opt-in
hotpath-callgrindandhotpath-cloudprofiling features, and the
optionalhotpathdependency is updated to 0.29.2. The new
profile_compressorexample drives the encoder at levels -5 to 22 on
repeated text, on a 17-byte input and on an optional corpus. The
hotpath benchmarkworkflow runs it on Alice for every push tomainand
pull request in merged mode: natively for timing and allocations, then
under Callgrind for instruction counts, in one report. It uploads the
report to hotpath Cloud (pull requests throughhotpath-relay.yml),
judged byhotpath/policy.toml. The published crate is unchanged apart from the
example. -
Continuous fuzzing with ClusterFuzzLite, ported from mbrotli.
fuzz/libfuzzerwraps all 14 AFL target bodies (11 stable, 3 framed) as
libFuzzer targets, which.clusterfuzzlite/build.shbuilds with
AddressSanitizer, debug assertions and overflow checks and seeds from the
AFL corpora and committed regressions. Pull requests that touch code get a
ten-minute run on the changed code (ci-cflite-pr.yml). A daily workflow
runs 5.5 hours of batch fuzzing, then corpus pruning and a coverage report
(ci-cflite-cron.yml).ci.ymllints the adapters. The published crate
is unchanged. The coverage build callscargo builddirectly, because the
image's cargo wrapper turnedcargo fuzz buildinto a build of the root
crate, so no targets were built. Pruning gets 10800 s instead of 600 s,
which timed out merging the first target's corpus.run_fuzzerssteps
declarelanguage: rust. -
Miri and AddressSanitizer workflows, ported from mbrotli and run on
everyv*tag or on demand.ci-miri.ymlrunsscripts/miri.sh, which
interprets only the tests that reachunsafein dependencies: the
fearless_simdvector kernels (opt_countmatch-length scans, the row
finder's tag masks, the decoder's shuffle wildcopy and sequence execution
around it) on the scalar, SSE2, SSE4.2 and AVX2 backends, plus AVX-512
when the runner supports it (MIRI_TARGET_CPU=nativein CI), and the
borrowed-bufuninitialized output of both codecs. Tests are named
exactly and each run checks its pass count.ci-sanitizer.ymlruns every
integration test withframed, then the same kernel modules natively,
under-Zsanitizer=addresswith LeakSanitizer. Tests that leaked their
inputs or&'staticdictionaries throughVec::leak/Box::leaknow
borrow locals or astatic OnceLock; the two tests that forget an
owning guard on purpose run in a separate step without leak detection.
Undercfg!(miri)theopt_countwindow
test keeps only lengths next to 16-byte boundaries and
compress_to_uninit_writes_the_one_shot_framecompresses 2 KB; native
runs check a superset of the previous window cases.
v0.2.8
-
Framed containers (
framedfeature).mzstd::framingports
mbrotli's structured framing API one to one —FramedCompressorwith
borrowed one-shot input, native and owned sessions, resource guards,
FramedWriter/ResourceWriterandFramedEncoderReader;
FramedDecompressorwith owned, appended and sliced one-shot output,
event sessions,FramedReaderand the random-accessFramedSeekReader;
resources, metadata (resource, footer, global, repeated), padding,
hidden resources, explicit dictionary references, a central directory and
a footer. In place of RFC 9841's Brotli wire format the container is made
only of Zstandard frames: visible resource chunks are regular frames,
byte-identical toZSTD_compress2of each chunk, sozstd -dprints the
visible resources; headers, metadata, hidden resources and the directory
live in skippable frames; and the container ends with a seek table in
zstd'scontrib/seekable_formatlayout. Tests check every container
againstZSTD_decompressand the contrib seekable decoder, which
zstd-ffinow builds as an oracle (along with the contrib seekable
compressor, the benchmarks' baseline). The fuzz package's opt-in
experimentalfeature is replaced byframed, which gates the
framed_encode,framed_decodeandframed_roundtriptargets. The
format is specified inarchitecture/framing.md; the API may change in
patch releases while the feature is opt-in. -
Clarified
ParallelTaskQueuedocumentation. Stopped executors can
leavewait, the blocking reader and writer and the writer's finishing
waiting indefinitely; dropping queue handles or tasks reports closure or
abandonment when the session observes it. Tasks executors already hold
must still be run or dropped. -
Parallel batches over random-access sources.
ParallelCompressor::prepare_sourceplans a batch over a
RandomAccessSourceinstead of a borrowed slice, as mbrotli's does, and
returns anOwnedParallelBatchwhose tasks are'static, so they run on
detached threads or any executor. Each task reads its segments and their
overlap into one reused buffer; the frame is byte-identical to
prepare_sliceover the same bytes.ArcBytesSource(sharedArc<[u8]>),
SeekSource(anyRead + Seek + Sendreader behind a mutex) and
FileSource(positional reads of a regular file) implement the trait,
andSourceIdentitylets a source carry a version token.finishchecks
that the source's length and identity are unchanged
(ParallelEncodeError::SourceChanged) and, for a frame with a checksum,
reads the source once more to hash it. Source failures are
SourceMetadataandSourceRead { segment }, carrying theio::Error
in aSourceIoErrorso the error staysClone + Eq.
v0.2.7
- Faster Huffman tables for small inputs. The literal table's sort
(HUF_sort) no longer places symbols that do not occur: the reference
sorts them into a last bucket of their own, so the other symbols' order
is unchanged, and counting them was the longest chain of increments to
one bucket. One array replaces the reference's start and cursor per
bucket, since a bucket ends where the next lower one starts. The
quicksort for buckets from count 164 up (HUF_simpleQuickSort) keeps
waiting ranges on a fixed 28-entry stack of byte pairs instead of
recursing, and its insertion sort stays hand-written: the standard
library's stable sort, tried for it, linked about 1,200 instructions of
driftsort forNodethat runs of at most nine nodes never reach. The
sort's stack frame shrank from 1,248 to 496 bytes. On histograms of
256-byte and 1 KiB slices of the test corpora the sort took 0.12-0.25 µs
instead of 0.16-0.77 µs, and 0.19-0.78 µs instead of 0.32-0.82 µs on
16 KiB and 128 KiB ones (an ad hoc timing loop, not checked in).
Compressing zstd's 1189-bytehttpfile took 8.5% less time at level 1,
6.2% at level 3, 2.4% at level 9 and 7.3% at level 19 (Criterion,
benches/compress.rs, against the previous commit in back-to-back runs
on an Apple M5 Pro); 1 MiB inputs are unchanged. Frames are
byte-identical: new tests compare the sort with a verbatim port of
HUF_sortand the quicksort with the recursive one on heavily tied
counts, and a rebuilt table drops symbols that no longer occur. - Faster fast and dfast levels (-131072 to 4). The fast search tests a
candidate on a filling table by clamping its offset into the history and
loading its bytes there, which needs neither a branch nor a stand-in
word; a clamped offset never matches, so the reference's matches are
found. The select it replaces rebuilt a stand-in's address at every
position. Both searches now prefetch the input when their step grows,
as the reference does. Against the previous commit, level -5 took 2-18% less
time, level 1 0.3-15%, level 2 0.3-14% and level 3 1-11% on 1 KiB to
1 MiB of text, binary and long-literal data (1 MiB of text at level 3
unchanged), with the largest gains on small inputs; level 4 moved by -3% to +1% (one reused context per input,
interleaved runs of both builds on an Apple M5 Pro). 1 MiB of binary data
at level 2 measured 5% slower, as it does with dfast untouched: code
placement, not the loop. Frames are byte-identical. Clamping dfast's
candidates too was up to 18% faster on single small blocks but up to 16%
slower on multi-block inputs, and was not kept. - Fixed an x86 slowdown in the fast and dfast levels. On x86 the
compiler split the clamped candidate test above back into two branches,
the range test mispredicting on text while the table fills: Alice and
1 MiB of text took 5-17% longer at levels -5 to 2 than before the
change (i7-13700KF, interleaved process pairs). The test now folds both
conditions into one value, which restores the previous time on text and
keeps a 1-5% gain on 64 KiB of binary and random data. dfast's inlined
match-length compare, also measured 3-6% slower on text at levels 2 to
4 on x86, was reverted. Frames are byte-identical. - NEON tag mask for the row match finder. 16-slot rows (levels 5 to 9
on inputs whose window exceeds 16 KiB) narrow their tag compare to four
bits per slot withvshrn, the reference'sZSTD_row_getNEONMask,
instead of emulating a byte-to-bit mask NEON lacks. The intrinsics run
in afearless_simd::kernel!; mzstd's source stays free ofunsafe.
Levels 5 and 7 took 1-6% less time on 64 KiB and 128 KiB of text and
binary data on an Apple M5 Pro. The reference's 32- and 64-slot NEON
masks and a NEON match-length kernel measured no faster than the
portable code and were not kept. x86 is unchanged. - The Huffman encoder no longer zeroes its code table. The 2 KiB of
packed codes are built witharray::from_fn, so each entry is written
once and thememsetin front of the fill is gone. - Compiled-in table widths for small inputs at the fast levels. The
fastsearch compiles in the hash widths of the reference's defaults
for inputs up to 128 and 256 KiB (minimum match and hash log (5,12),
(5,13) and (6,14), levels -5, -1 and 1), together with the window's
first index, for blocks whose prefix starts the window (one-shot frames
and a fresh context's first frame); other prefixes keep the generic
loop. Before, only the large-input widths were compiled in, so Alice
and every 64 KiB input ran the generic loop. Its short literal runs are
appended straight from the input as one 16-byte chunk, without a stack
copy or an overflow test (store_seq_wild). Cold one-shot calls on
Alice take 3-5% less time at levels -5, -1 and 1, and 64 KiB inputs
1-6% less (i7-13700KF, 10 interleaved process pairs per case); the fast
kernel executes 9% fewer instructions. About 30 KB of code. - dfast's table widths are compiled in, one function per width. The
six (minimum match, hash log, chain log) sets the default parameters
give on inputs up to 64 KiB, 256 KiB and above, at levels 2 to 4, each
run in a separate#[inline(never)]search function, so their shifts
are immediates and their table bounds constants. Inlined side by side,
LLVM had merged the instances back into one loop with run-time widths,
which is why the compiled-in widths once measured no faster. The first
word of a match length is compared inline again, now neutral to 3%
faster inside the separate functions. After a match, dfast loads the
word and long-table slot where the match ends and prefetches their
candidates before the backward extension and the sequence store, so the
next search no longer waits for them (the kernel is bound by the latency
after each mispredicted match, not by its instruction count), and short
literal runs are appended straight from the input as fast's are.
Together, Alice takes 5% less time at level 2 and 7% at level 3, 1 MiB
of text 5% at levels 3 and 4 and random data 3% (12 interleaved pairs).
About 160 KB of code over the four x86 SIMD levels; the 44-byte text at
level 3 is 1.5-3% slower (0.5 µs calls). - The row match finder is chosen once per block. greedy, lazy and
lazy2 with the row finder picked among nine (minimum match, row size)
searches at every searched position, and each parser inlined all nine.
The search is now picked when a block starts, so each parser inlines
one. Levels 8 to 12 take 1-2.5% less time on Alice and 1 MiB of text,
with 6-8% fewer mispredicted branches; greedy and lazy are unchanged.
The dictMatchState parsers keep the per-search choice. About 570 KB of
code. - Cold calls no longer zero tables they may never use. The Huffman
table, the three sequence tables and the optimal parser's state are
allocated on their first write (a privateLazyBox: an unwritten table
reads as one shared all-zero value, as the reference leaves its
workspace untouched until a block writes it). The engine is built once
instead of copied, and is 2 KB smaller. Empty input compresses in half
the time (289 to 145 ns at level 1), the 44-byte text 30-40% faster at
levels -5 to 9, 64 KiB inputs 1-11% faster; a 44-byte call executes
5.6K instructions instead of 17.6K (zstd: 6.0K). - The block pre-splitter is compiled per sampling rate and hash log,
as the reference'sZSTD_GEN_RECORD_FINGERPRINTdoes, and its distance
loop vectorizes:ZSTD_splitBlockexecutes 40% fewer instructions,
below the reference's, and 1 MiB of text takes 2.7% less time at
level 9, 1 MiB of repeated bytes 4-8% less at levels 3 to 19. - The decoder no longer zeroes output ahead of each block.
decompressanddecompress_intozeroed a whole block of room before
decoding a compressed block into it. Sequences of 1 KiB or more, a
block's trailing literals and literal-only blocks are now appended to
the vector, so their bytes are written once, and room for shorter
sequences is zeroed 4 KiB at a time as the sequence loop reaches it. A
BlockOutputtrait carries both kinds of destination; the sequence loop
is compiled for fixed destinations (decompress_to_slice, streaming
history), which keep the previous loop, and for growing ones, which
leave it for a cold path when a sequence needs more room. 1 MiB of text
decodes 8-16% faster, the 64 KiB binary corpus 14-22% (now faster than
zstd at every level) and 1 MiB of repeated bytes 10-19%.
decompress_to_uninitzeroes on demand, a chunk at a time. - Faster Huffman literal decoding. The four-stream pair loop follows
the reference's fast loop: 11-bit pair tables (half the size and build
cost), five lookups per refill and two-byte stores whose window the
compiler proves from the row's two-bit symbol count, and refills clamped
instead of bounds checked: 13.2 instructions per lookup instead of 17.7
(zstd: 11.25). The single-symbol loop stores symbols straight into
fixed-size chunks of the four outputs instead of gathering them in an
array that spilled to the stack (8.8 per symbol instead of 11.6; zstd
8.2). Single-symbol tables are filled by weight in fixed-size runs
after a counting sort (HUF_readDTableX1_wksp), weight streams decode
four symbols per refill, and sequence FSE tables are spread without
per-cell tests when no symbol has a "less than one" probability
(ZSTD_buildFSETable_body): reading a Huffman tree costs 35% fewer
instructions, a sequence table 20%. Alice and 1 MiB of text decode
3-5% faster at levels 1 to 19. These table builds were measured slower
on 2026-10-15; that run compared builds whose unchanged sequence loop
moved by ±1.6% instructi...
v0.2.6
- Decoder output into uninitialized memory.
DecoderSessionand
DecoderSessionOwnedhaveprocess_uninit, andDecompressorhas
decompress_to_uninitanddecompress_with_dictionary_to_uninit, as in
mbrotli. Each takes&mut [MaybeUninit<u8>], such as
Vec::spare_capacity_mut, and writes exactly the bytes it reports,
nothing past them; a session'sDecodeFailurehas written exactly its
producedprefix. Bytes, counts, statuses and errors matchprocess,
decompress_to_sliceanddecompress_with_dictionary_to_slice, and
processandprocess_uninitmay be mixed on one operation. The crate
stays free ofunsafe. Sessions only ever copy finished output out of
their history, soprocess_uninitwrites the caller's memory once and
never zeroes it. The one-shot direct path reads its output back as history,
so it zeroes the destination throughborrowed-bufone block ahead of the
decoder and never past a frame's declared end; raw and RLE blocks are
written once without zeroing. Thedecompressionfeature now pulls in
borrowed-buf. The decoder is generic over its destination rather than
matching on it, soprocessruns the code it ran before: streaming 4 MiB
of mixed data took 2.81 ms against 2.84 ms before the change (Criterion,
benches/decompress.rs, Apple M5 Pro). Againstdecompress_to_slice,
decompress_to_uninittook 0.97-1.05 times as long on 4 MiB of text and
binary at levels 1, 3 and 19 and on a 12 MiB LDM frame, and the same on
random data;process_uninittook 2.83 ms against 2.81 ms forprocess.
New Criterion casesmzstd-uninitcover both paths. mzstd_decompressno longer zeroes the destination. The C ABI views
the caller's buffer asMaybeUninit<u8>and calls
decompress_to_uninit, so on success no byte past the result is written.
mzstd-ffidrops itsborrowed-bufdependency. On
mzstd-ffi/benches/one_shot.rs, 256 KiB of random data decodes in
2.93 µs instead of 3.84 µs and 1 MiB of text 0-2.5% faster. The
large-literal corpus is 2-4% slower, and a 64-byte frame takes 30 ns
instead of 25 ns at level -5 and 1.5-3% longer at other levels.- Fix
FrameMode::Singlewith skippable frames. A single-frame
operation skipped skippable frames and kept waiting for a Zstandard frame,
unlikeZSTD_decompressStream, which reports a frame complete after
either kind. A trailing skippable frame, or a stream made only of
skippable frames, failed withUnexpectedEndOfInputalthough zstd reads
it, and a skippable frame followed by a Zstandard frame decoded both in
one member where zstd stops after the skippable one.Singlenow ends
after exactly one frame of either kind: a skippable frame finishes the
operation with no output and leaves what follows unconsumed (one-shot
calls report it asTrailingData). Sessions, one-shots and theio
adapters change alike;FrameMode::Concatenatedis unchanged. Regression
tests compare member-by-member decoding with the reference through the
newzstd_ffi::oracle::decompress_first_frame. 0restores a compression parameter's default. Every numeric
with_*setter ofCompressionParamsandLdmParamsstores0as
unset, asZSTD_CCtx_setParameter(.., 0)does. The last value given now
wins, and a caller translating zstd-style parameters can apply each one
directly in its loop instead of collecting them inOptions first.
Before,with_window_log(0)storedSome(0)andCompressor::newrejected
it as out of range, and no setter could clear a value.0was already
accepted bywith_target_lengthandLdmParams::with_hash_rate_logand
meant the default there, so no frame changes.- Tests and fuzzing.
tests/decoder_uninit.rscompares the new
methods with their initialized counterparts across corpora, levels, input
chunks, output windows, failures, frame sequences and dictionaries, with
sentinels past every reported length. The AFL decoder targets alternate
processandprocess_uninit, anddecompresschecks
decompress_to_uninit;parameter_parsingaccepts a zero window log.
The architecture adds a
decoder uninitialized output
specification.
v0.2.5
- Faster fast levels (-5 to 2). The hash widths of the default
parameters for inputs above 256 KiB are compiled in, which turns the hash
shift and table mask into constants, and a candidate's validity is
checked with a branch once the history before a block is several times
the table's size, where that branch predicts well (the branch-free check
stays for blocks whose table is still filling). On the first 20 MB of
Silesia (silesia.tar, warm contexts) levels -5 to 2 compress 5-10%
faster. Frames are unchanged. - btlazy2 (levels 13-15) is as fast as zstd. The binary-tree walk picks
the side to descend with a branch, as the reference does, instead of a
branch-free select: the walk is a chain of dependent cache misses, and a
predicted branch starts the next node's load before the comparison that
picks it resolves. On the first 5 MB of Silesia levels 13 and 15
compress 17% and 21% faster. Frames are unchanged. - Sessions without a dictionary are small again.
PreparedDictionary
keeps its tables behind oneBox, so it is pointer-sized and an
EncoderSessionOwned(defaultD = PreparedDictionary) no longer reserves
about 8 KiB inline for a dictionary it may not have: 40 bytes instead of
8192 on 64-bit targets. The public API is unchanged. - Benchmarks. Both comparison suites were rerun on this release; the
previous run's archives were replaced.
v0.2.4
-
Dictionaries.
mzstd::dictionaryreads raw-content and
Zstandard-format dictionaries (DictionaryContent::{Auto, Raw, Zstd}).
DictionaryBuilderprepares aPreparedDictionaryfor one
EncoderConfig, andCompressorgainscompress_with_dictionary,
compress_with_dictionary_into,_to_slice,_to_uninit,
start_with_dictionary,into_session_with_dictionaryand, withstd,
writer_with_dictionaryandreader_with_dictionary; a dictionary
prepared for another configuration (frame options aside) is
EncodeError::DictionaryMismatch. Frames are byte-identical to zstd
1.5.7'sZSTD_compress2andZSTD_compressStream2after
ZSTD_CCtx_loadDictionary_advanced: the dictionary's own tables and
parameters (tagged fast/dfast tables), the reference's choice between
attaching the dictionary and copying its tables, dictMatchState variants
of every match finder (fast, dfast, greedy/lazy/lazy2 with hash chains and
rows, btlazy2, btopt, btultra, btultra2), the dictionary's entropy tables,
repeat offsets and ID, and the optimal parser's first-block statistics
seeded from them. One exception, found by fuzzing: with a dictionary whose
literal or match length table omits its last symbols, the reference's
optimal parser reads uninitialized memory and its frames vary from run to
run; mzstd's stay deterministic. -
Dictionary decoding.
DecodeDictionaryparses dictionaries as
zstd 1.5.7'sZSTD_DDictdoes, rejected exactly when the reference
rejects them; aPreparedDictionarydecodes too, through
DictionaryRef::Prepared, without a second copy of its bytes.
Decompressorgainsdecompress_with_dictionary,_into,_to_slice,
start_with_dictionary,into_session_with_dictionaryand, withstd,
reader_with_dictionary/writer_with_dictionary. A frame naming another
dictionary ID fails with the newDecodeError::DictionaryMismatch
(DictionaryUnsupportedstill reports one decoded without a dictionary).
EncoderSessionOwnedandDecoderSessionOwnedgain a defaulted type
parameter for the dictionary they own. Decoding without a dictionary is
unchanged. -
Parallel segments are primed with the input before them instead of
compressing it. For fast, dfast, greedy, btopt, btultra and btultra2,
each segment loads its overlap as a raw-content prefix, the way ZSTDMT
primes a job (ZSTD_compressBegin_advanced_internalwith
ZSTD_dtlm_fastandforceWindow; checked byte-identical against
ZSTD_CCtx_refPrefixwithZSTD_c_forceMaxWindowat every level, LDM
included), where the overlap used to be compressed and thrown away.
Lazy, lazy2 and btlazy2 still compress it: loading made them up to three
times slower on random input (lazy, lazy2) and 9% slower on text
(btlazy2). Overlap sizes are unchanged. With four threads and 4 MiB
segments on 16 MiB of text, binary, random and mixed data and on the
vendored zstd sources (i7-13700KF,benches/parallel.rs
parallel-corpora, new): levels 18-22 run 15-27% faster (binary level
19: 2.34 s instead of 2.77 s) with frames 0.01-0.27% larger, levels 1-5
up to 27% faster and levels 16-17 1-7% faster with frames within 0.12%;
levels 6-15 are unchanged. Parallel output changes; serial output does
not. Each stream segment's buffer is now allocated once at its full size,
and batch assembly reserves the frame. -
Cheaper Huffman code packing. Literals packed their codes symbol by
symbol with a branch; the packing is now branch-free over the whole
alphabet in the backend's feature context, which matters for a
dictionary's 256-symbol table (one 107-byte record with a dictionary: 9.2k
instructions instead of 11.5k). Output is unchanged. -
Measured against 0.2.3 on the Criterion suites (
benches/compress.rs,
benches/decompress.rs, pinned to one core, alternating runs): every
compression and decompression case without a dictionary is within ±2%,
and 64-byte and 1 KiB inputs execute 0.3% more instructions per call.
Compression with a dictionary (benches/dictionary.rs, trained 64 KiB
dictionary, JSON-like records) runs at 0.85-0.91x zstd 1.5.7's time at
levels 16 and 19 on 100 and 1000 records, 1.09-1.19x at levels 1-9, and
1.20-1.29x for a single 107-byte record. -
New fuzz target
compress_dictionaryanddecode_dictionary, and the
dictionaryCriterion benchmark; theparallelbenchmark gains a
per-level group.
v0.2.3
-
Faster fast and double-fast match finding (levels -131072 to 4).
Candidate validity is a select between the candidate's bytes and a
stand-in word that can never match (the reference's
ZSTD_match4Found_cmov), so only equality branches; this removed most of
the finders' branch mispredictions. Blocks that cannot read below the
prefix take a specialization with prefix-relative positions, hash slots
are held as cell references, and exit tests share the bounds checks of the
loads they guard. Frames remain byte-identical to zstd 1.5.7. -
Faster lazy finders (greedy to btlazy2). Row tables are viewed from
a 64-byte boundary as the reference's workspace aligns them (half of the
tag rows used to straddle cache lines); search state is built once per
block; the row update checks one range per run and steps through a
constant table; collected candidates are prefetched as in the reference;
btlazy2's tree walk counts short runs inline, hands long ones to the
vector scan, and picks its side without a branch. -
Faster optimal parser (btopt, btultra, btultra2). A per-block
Finderprecomputes the search's parameters, the tree update runs out of
line, the walk exits early when the first candidate is outside the window,
hash cells are prefetched four positions ahead, long literal runs refresh
all literal weights at once, and the position and match tables are sized
to what a block can use. Levels 16-22 now run faster than zstd 1.5.7 on
every dataset of the comparison corpus. -
Cheaper sequence coding.
fill_codesbuilds the codes, extra-bit
counts and the three code histograms in one pass,sequences::encode
adds each sequence's extra bits with one write when they number fewer
than 31, and the bit writers and FSE table build drop their bounds checks.
The sequence encoder now executes fewer instructions than the reference's. -
Updated benchmarks for this working tree
(2026-10-14): compression median 1.092× zstd 1.5.7 (was 1.000×), faster in
158 of 192 dataset/level pairs (was 96); decompression median 1.022×, with Alice 0.92× (was 0.89×) and cyclic
Alice 0.85× (was 0.82×). All frames remain byte-identical to zstd 1.5.7.
Text at the fastest compression levels and compressible-text decoding
remain slower than C. -
Faster literal and sequence decoding. Huffman pair tables are built
from one series per code length with one add per row (3.4x fewer
instructions), sequence bit counts carry their range so reads need no
masking, the first chunk of a long-offset match copy and the Huffman fast
loops' refills need no bounds checks.
v0.2.2
-
A session holds a first
Endcall to its declared size. With
InputSize::Exact(n), a session whose whole input arrived in one
Operation::Endcall of any other length used to write a complete frame
for that length, asZSTD_compressStream2does when it replaces the
pledged size. It now fails withEncodeError::InputSizeMismatch, like
every other contradiction of a declared size, so a truncated or overlong
input never yields a frame. Frames whose input matches its declaration
are unchanged and still byte-identical to zstd 1.5.7. -
process_uninitandcompress_to_uninitwrite in place. Both used to
allocate and zero a temporary buffer as large as the destination on every
call, then copy the result over. The encoder now writes into the caller's
MaybeUninitmemory directly, zeroing it only just ahead of each block it
writes there (tracked withborrowed-buf, withoutunsafe), and copies
buffered output in. Bytes, counts, statuses and errors are those of
processandcompress_to_slice. As with those, bytes past the reported
output may now hold zeros or scratch bytes; only the reported prefix is
meaningful. Theencoder_sessionandoutput_capacityfuzz targets cover
both paths, andbenches/compress.rstimes them: with 128 KiB outputs
over 1 MiB of mixed data in 16 KiB chunks,process_uninitat level 1
takes 1.25 ms (was 1.35 ms;process1.26 ms), and with 4 KiB outputs
1.22 ms (was 1.28 ms).compress_to_uninitof 1 MiB of text is 1-2%
faster (level 3: 1.334 ms, was 1.350-1.363 ms). Measured with
cargo bench --bench compress -- 'stream-output|compress_to_uninit'on
Apple Silicon (aarch64-apple-darwin); output is identical.
v0.2.1
-
Reuse the Huffman tree during depth selection. Optimal-depth trials
restore leaf lengths from the original internal nodes instead of sorting
and rebuilding the same tree for each depth. Table selection and encoded
bytes remain identical to zstd 1.5.7. -
Less work on incompressible blocks. After a failed Huffman attempt,
raw literals that already exceed the block's compression threshold cause
an early raw-block decision. This avoids copying a literals section and
encoding sequences that would be discarded. Short output buffers retain
their existing error behavior, and frames remain byte-identical to C. -
Simpler fast/dfast candidate checks. Bounded reads from the searchable
history reject stale indices and guard the comparison word with one range
check. The search order and minimum-match specializations are unchanged. -
Wider match comparisons. Fast, double-fast and lazy search share a
word-first, 128-byte scan in the selected SIMD backend. On AVX2 it tests
four vectors together, then resolves the first differing word. Short
matches and bounded tails keep their exact bytewise semantics. -
Less output initialization in cold compression.
compressand
compress_intoreserve the same capacity but initialize the output of
inputs larger than 128 KiB one block at a time, reusing unwritten room.
The RLE check uses bulk slice comparison. Frames remain byte-identical
to zstd 1.5.7; caller-owned slices keep their exact capacity semantics. -
Faster empty and tiny inputs at levels 11-22. The optimal parser's
position and match tables are sized per block on first use
(min(block + 3, ZSTD_OPT_SIZE)entries) instead of being filled to
4099 entries at every frame start. Frames are unchanged. -
Updated compression benchmarks
for the final working tree: 384 measurements, median speed relative to C
1.000×, with lower mean latency in 96 of 192 dataset/level pairs. All
frames remain byte-identical to zstd 1.5.7. Large text remains slower;
the aggregate does not establish an overall lead. -
Faster
Decompressor::decompressanddecompress_into. Raw and RLE
blocks are appended in one pass instead of zeroing room and copying over
it; frames that declare a content size are decoded into a vector cut at
their end, so neither slack nor spare capacity is kept (a declared frame's
vector now has exactly its length as capacity); capacity doubles, bounded
by the frame, and a declared size is reserved up front up to 32 bytes per
unread input byte. 1 MiB of random data decodes in 20.1 µs (was 43.8 µs;
zstd 1.5.7 20.6 µs), 64 KiB in 1.01 µs (was 1.82 µs; zstd 1.79 µs). Errors
are unchanged: a frame that overruns its declared size is re-decoded uncut
to report the sameContentSizeMismatch. -
New cold comparison against zstd 1.5.7 in
benchmarks/comparison
(its own workspace): mbrotli's eight datasets, including Alice in
Wonderland (testdata/alice29.txt), at levels -5, -1 and 1-22, with
byte-identity preflight, CSV export, and generated per-level pages,
per-dataset charts and run reports indocs/benchmarks/, laid out as
mbrotli's benchmark documentation. Seedocs/benchmarking.md.
v0.2.0
-
Breaking: parallel streaming sessions are rebuilt around a window
and a non-blockingprocess.ParallelCompressor::start(stream, window)
returns aParallelEncoderSessionand itsParallelTaskQueue;
into_session(stream, window)returns the owned
ParallelEncoderSessionOwned(withinto_compressor). Sessions have
process,flushandfinish, which report the new
ParallelEncoderStatus(NeedsInput,NeedsOutput,Pending,
Finished) in aParallelProgress, andwait, which blocks for one task
completion.windowbounds the segments queued, running or awaiting
delivery, and so the session's memory; a full window reportsPending.
Each full segment becomes a task on the queue, which executors drain with
for task in queue { task.run() }(recvblocks;next_taskdoes not).
With a declared size and without a flush, the frame is byte-identical to
prepare_slicefor any chunking, window and execution order; an input
that fits in one segment takes the batch's serial fallback with or without
a declared size. Finishing, failing or dropping the session closes the
queue, so executor loops end; dropping every queue handle fails the stream
with the newParallelEncodeError::TaskQueueClosedorTaskAbandoned
instead of hanging. Stream failures are the newStreamFailedwith the
cause as its source, repeated on every later call.push,drain_into,
finish_into,run_queuedandqueue()are removed. -
ParallelEncoderSessionandParallelEncoderSessionOwnedhave
process_uninit, which delivers into&mut [MaybeUninit<u8>]such as
Vec::spare_capacity_mutwithout zeroing it first. It produces the same
bytes, counts, statuses and errors asprocess, writes nothing past
produced, and may be mixed withprocesson one stream. The AFL
paralleltarget alternates the two. -
Parallel streams have
std::ioadapters.
ParallelCompressor::writer(sink, stream, window)returns a
ParallelEncoderWriterandParallelCompressor::reader(source, stream, window)aParallelEncoderReader, each beside theParallelTaskQueue
their segment tasks go to, asstartdoes. Both drive the parallel session
and produce its bytes, so with a declared size and without a flush the
output is byte-identical toprepare_slicehowever writes or source reads
are split. Unlikeprocess, they block inwaitwhile the window is full
or a part is pending, so the queue must run on other threads. The writer
keepsEncoderWriter's transactional guarantees: nothing is lost to short
writes or sink errors,Write::flushisOperation::Flush, andfinish
returns the adapter insideFinishErrorfor a retry.reader.into_parts()
returns read-ahead source bytes as the newEncoderReaderParts
(source,unread_input), whichEncoderReader::into_partsnow returns
too.ParallelEncodeErrornow converts intostd::io::Error, taking the
kind of its root cause throughStreamFailedandBatchFailedand never
InterruptedorWouldBlock. The AFLparalleltarget also drives both
adapters. -
Batch tasks report exactly once: a task dropped unrun is
TaskAbandoned { segment }and one that unwindsTaskPanicked { segment }
(the panic still propagates and its engine is discarded). A failed batch
returns the newParallelEncodeError::BatchFailedwith the first failing
task's cause as its source, andfinish_intoleaves its output untouched;
InvalidStatenow only reports a task leaked withmem::forget. Batch
frames are unchanged. -
A new
parallelCriterion bench (benches/parallel.rs) compresses one
16 MiB input in 1 MiB segments on four executor threads through the batch, the session and the
writer, beside zstd's single-threadedZSTD_compress2. Measured with
cargo bench --bench parallel -- --warm-up-time 1 --measurement-time 3(Apple M5 Pro): all three write the same frame, and
the session and writer run within noise of the batch (level 1: 9.99,
10.08 and 10.02 ms; level 3: 17.62, 17.27 and 17.12 ms). zstd serial takes
18.09 and 30.37 ms; the parallel frames are 0.05% and 0.36% larger
(6286456 vs 6283433 and 5712713 vs 5692172 bytes). -
mzstdandmzstd-ffiare now dual-licensed under MIT or
Apache-2.0, at your option. The MIT text moved fromLICENSEto
LICENSE-MIT, andLICENSE-APACHEwas added. -
The README and crate docs have one "Choose your API" table covering
both codecs, thestd::ioadapters andParallelCompressor
(prepare_slice,start). In the crate docs it renders for every feature
set; APIs a build leaves out link to the feature-selection section. -
The
CI Fuzzworkflow archives AFL's findings before uploading them:
actions/upload-artifactrejects AFL's file names, which contain:. -
Added
Backend, a public, host-validated SIMD level
(fearless_simd):Backend::available,Backend::name,
Display/Debug,CompressorBuilder::with_backend/
DecompressorBuilder::with_backend, andCompressor::backend/
Decompressor::backend. The default is detected once when a codec is
built; every backend produces identical output.compressionand
decompressionnow depend onfearless_simd, andcompressionon
branchesfor safe prefetch hints. -
Decompression is 2.5-9x faster, now at 0.93-1.18x of zstd 1.5.7's time
oncargo bench --bench decompress(i7-13700KF, AVX2, one core),
with identical output and the same verdicts on malformed input:- Huffman and sequence bitstreams use a 64-bit container refilled where
the reference refills. Four-stream literals decode interleaved with
the reference's sentinel fast loops, with a pair (X2) table when the
reference's timing model prefers it. - Sequence tables carry baselines and extra-bit counts per state;
predefined tables are kept between blocks; tables and buffers are
reused. - Sequences execute with wildcopies, including a byte-shuffle kernel for
overlapping short-offset matches on SSSE3, AVX2, AVX-512, NEON and
WASM SIMD. decompress_to_slicedecodes straight into the destination when every
frame declares a content size that fits.
- Huffman and sequence bitstreams use a 64-bit container refilled where
-
Compression is 1.3-3.4x faster. On
cargo bench --bench compressit now
runs at 0.88-1.07x of zstd 1.5.7's time on small inputs and at levels
16-19, 1.03-1.17x on 1 MiB of text, and 1.1-1.3x on binary input and
in streaming. Frames stay byte-identical to zstd 1.5.7 on every backend:- The fast and dfast finders are specialized per minimum match length,
read through fixed-width windows over the block's prefix, and mask
their table indices; extDict variants work the same way. - The greedy, lazy, lazy2 and btlazy2 finders compare row tags with one
u8x16/u8x32/u8x64compare (ZSTD_row_getMatchMask), prefetch
rows 8 positions ahead as the reference does, and are specialized per
search, depth, minimum match, row size and extDict mode. - The optimal parser (btopt, btultra, btultra2) and LDM are specialized
per backend, optimization level and extDict mode; one binary-tree walk
serves insertion and match collection; prices come from tables
rebuilt at each rescale with a vectorized weight form. - Match lengths are compared a word at a time, then with native vectors.
- Entropy coding follows the reference's BMI2 builds: packed Huffman
codes with its two-container loop, interleaved histograms, the fast FSE
spread, and 16-byte literal copies. - Per call: block states swap instead of copying, and tables are reused
whenever indices continue, asZSTD_reset_matchStatedoes.
- The fast and dfast finders are specialized per minimum match length,
-
XXH64 stays scalar: an AVX2 kernel measured 1.8x slower (no 64-bit
vector multiply); the scalar code matches the reference's throughput.