Repository navigation
v0.2.9
-
Session API in
mzstd-ffi. C callers can now compress and
decompress incrementally through reusable handles:mzstd_encoder_create,
_reset(with a pledged size orMZSTD_CONTENT_SIZE_UNKNOWN),
_process(MZSTD_CONTINUE,MZSTD_FLUSH,MZSTD_END) and_free, and
the matchingmzstd_decoder_*functions (MZSTD_DECODE_PROCESS,
MZSTD_DECODE_FINISH, awindow_log_maxof10..=31or0for 27).
Each process call takes in/out lengths and reportsMZSTD_NEEDS_INPUT,
MZSTD_NEEDS_OUTPUTorMZSTD_FINISHED. Encoder frames are
byte-identical toZSTD_compressStream2for the same inputs, output
capacities and operations; the decoder reads concatenated and skippable
frames asZSTD_decompressStreamdoes. Pointers, lengths, overlaps and
operation values are validated before use, and a panic poisons the handle
instead of unwinding into C. A newsessionCriterion benchmark times
both against zstd's streaming API with reused contexts. -
The
fastlevels no longer zero a block's room per dense block.
Promoting a dense full block to the fixed-size sequence cursor resized
the store's vectors to a full block and truncated them again every time,
writing 384 KiB of zeros per 128 KiB block. The store now keeps one
full-block destination, zeroed once, and the promoted block's entries
stay in it until the next block. Silesia nci at level -1 with a reused
context is 5-6% faster (it had fallen 4.5% behind the main branch),
cyclic Alice 2.4-2.9% at levels -5 to 2. Frames are unchanged. -
Benchmarks refreshed on AMD, against C and the main branch. The
published comparison is a new complete sweep of all 384 encoder and 384
decoder cases on a Ryzen 9 7950X, and its pages and charts carry a third
series, the main branch measured beside the working tree
(--main-csvinlevel_docs.py,plot.pyanddecoder_docs.py). A new
silesia.pytimes C, main and the working tree in random order on whole
Silesia files with reused contexts. The earlier Intel sweep and the
interim AMD tuning reports are removed. See the
benchmark results. -
Reused-context and literal-decoding performance without unsafe. The
row match finder views its kept tables a row further on at every context
reset: entries a previous input of similar content left in the
never-cleared tag table otherwise shared the new input's tags, and on
repeated Alice frames the greedy and lazy levels 4-10 took up to 6.5%
longer from finding and dismissing them. The reference salts its hash
instead (ZSTD_advanceHashSalt), at an XOR per hash; the view costs
nothing per position, so cold searches are unchanged. The
decoder runs its block loops with AVX2's features on AVX-512 hosts (the
compiler vectorized the Huffman streams' lookups into gathers there),
decodes large four-stream Huffman sections into a literal buffer of a
fixed size that lets the pair loop keep one output position per stream,
and reads offset bits with one shift. Thefastmatch path extends
backwards with the reference's byte loop, compares a match's first word
with one bounds test and keeps its sequence cursor's counts in
registers: 1.4-1.7% faster on Silesia at levels -1 and 1, still 7-8%
behind the reference there. Encoded bytes remain identical to
zstd 1.5.7. See the benchmark results. -
Alice performance follow-up without unsafe. Dense full
fastblocks
promote their sequence writer after 1024 sequences, lazy prefix parsing
uses relative positions, and sequence code/width maps share one lookup.
The decoder skips zero-width length reads, builds described FSE rows in
the selected feature context, and continues Huffman pairs into the tails
before exact stream-end validation. -
AMD performance tuning without unsafe. The encoder's block
pre-splitter runs in the selected SIMD feature context, uses exact 32-bit
fingerprint arithmetic and keeps its fixed histograms on the stack. The
decoder borrows compile-time predefined FSE tables and symbol rows, avoids
an enum-conversion table lookup, and interleaves three Huffman streams
while decoding the fourth separately. Encoded bytes remain identical to
zstd 1.5.7. -
Fixed a false
framed_roundtripfuzz crash. With one-byte chunks the
target writes a chunk per payload byte plus metadata, which exceeded the
4096-chunk decode limit of the framed targets. The limit is now 8192, and
the ClusterFuzzLite input is kept as a regression. The library is
unchanged. -
Formal verification against RFC 8878, ported from mbrotli. Charon and
Aeneas translate selected Rust items into Lean 4
(scripts/extract_lean.py), andverification/leanproves them correct
against a transcription of RFC 8878 itself, never against the C library:
the frame header (descriptor, window formula, dictionary ID, content size)
and the block header on both sides, with round trips from the encoder's
writers through the decoder's parsers;Number_of_Sequencesin its
shortest form and every form the RFC defines; the literals length and
match length codes, whose encoder picks the only code whose RFC range holds
each length and whose decoder tables read it back; the offset code tables,
the default distributions, the initial repeat offsets and the magic
numbers; the encoder's repeat offset update for every offset value; and the
FSENumber_of_BitsandBaselineof every state (Table 21). Each proof
also shows the translated code cannot panic or overflow on the inputs it
covers. The transcription checks itself against the RFC's worked examples;
RFC 8878's Table 18 misprints its last two rows, and the proofs follow the
RFC's text.ci-lean.ymlchecks that the translation matches the Rust
source and that the proofs build with only Lean's standard axioms. To make
the code translatable,FrameHeader::parsereads its fields by index,
Number_of_Sequencesis read and written by the newsequence_countand
write_sequence_count, and the encoder's frame header is written from a
FrameHeaderFieldsvalue; behaviour and output are unchanged. -
hotpath Cloud with Callgrind, ported from mbrotli. New opt-in
hotpath-callgrindandhotpath-cloudprofiling features, and the
optionalhotpathdependency is updated to 0.29.2. The new
profile_compressorexample drives the encoder at levels -5 to 22 on
repeated text, on a 17-byte input and on an optional corpus. The
hotpath benchmarkworkflow runs it on Alice for every push tomainand
pull request in merged mode: natively for timing and allocations, then
under Callgrind for instruction counts, in one report. It uploads the
report to hotpath Cloud (pull requests throughhotpath-relay.yml),
judged byhotpath/policy.toml. The published crate is unchanged apart from the
example. -
Continuous fuzzing with ClusterFuzzLite, ported from mbrotli.
fuzz/libfuzzerwraps all 14 AFL target bodies (11 stable, 3 framed) as
libFuzzer targets, which.clusterfuzzlite/build.shbuilds with
AddressSanitizer, debug assertions and overflow checks and seeds from the
AFL corpora and committed regressions. Pull requests that touch code get a
ten-minute run on the changed code (ci-cflite-pr.yml). A daily workflow
runs 5.5 hours of batch fuzzing, then corpus pruning and a coverage report
(ci-cflite-cron.yml).ci.ymllints the adapters. The published crate
is unchanged. The coverage build callscargo builddirectly, because the
image's cargo wrapper turnedcargo fuzz buildinto a build of the root
crate, so no targets were built. Pruning gets 10800 s instead of 600 s,
which timed out merging the first target's corpus.run_fuzzerssteps
declarelanguage: rust. -
Miri and AddressSanitizer workflows, ported from mbrotli and run on
everyv*tag or on demand.ci-miri.ymlrunsscripts/miri.sh, which
interprets only the tests that reachunsafein dependencies: the
fearless_simdvector kernels (opt_countmatch-length scans, the row
finder's tag masks, the decoder's shuffle wildcopy and sequence execution
around it) on the scalar, SSE2, SSE4.2 and AVX2 backends, plus AVX-512
when the runner supports it (MIRI_TARGET_CPU=nativein CI), and the
borrowed-bufuninitialized output of both codecs. Tests are named
exactly and each run checks its pass count.ci-sanitizer.ymlruns every
integration test withframed, then the same kernel modules natively,
under-Zsanitizer=addresswith LeakSanitizer. Tests that leaked their
inputs or&'staticdictionaries throughVec::leak/Box::leaknow
borrow locals or astatic OnceLock; the two tests that forget an
owning guard on purpose run in a separate step without leak detection.
Undercfg!(miri)theopt_countwindow
test keeps only lengths next to 16-byte boundaries and
compress_to_uninit_writes_the_one_shot_framecompresses 2 KB; native
runs check a superset of the previous window cases.