Skip to content

v0.2.9

Choose a tag to compare

@Mnwa Mnwa released this 30 Oct 10:31
· 12 commits to main since this release
c99fce3
  • Session API in mzstd-ffi. C callers can now compress and
    decompress incrementally through reusable handles: mzstd_encoder_create,
    _reset (with a pledged size or MZSTD_CONTENT_SIZE_UNKNOWN),
    _process (MZSTD_CONTINUE, MZSTD_FLUSH, MZSTD_END) and _free, and
    the matching mzstd_decoder_* functions (MZSTD_DECODE_PROCESS,
    MZSTD_DECODE_FINISH, a window_log_max of 10..=31 or 0 for 27).
    Each process call takes in/out lengths and reports MZSTD_NEEDS_INPUT,
    MZSTD_NEEDS_OUTPUT or MZSTD_FINISHED. Encoder frames are
    byte-identical to ZSTD_compressStream2 for the same inputs, output
    capacities and operations; the decoder reads concatenated and skippable
    frames as ZSTD_decompressStream does. Pointers, lengths, overlaps and
    operation values are validated before use, and a panic poisons the handle
    instead of unwinding into C. A new session Criterion benchmark times
    both against zstd's streaming API with reused contexts.

  • The fast levels no longer zero a block's room per dense block.
    Promoting a dense full block to the fixed-size sequence cursor resized
    the store's vectors to a full block and truncated them again every time,
    writing 384 KiB of zeros per 128 KiB block. The store now keeps one
    full-block destination, zeroed once, and the promoted block's entries
    stay in it until the next block. Silesia nci at level -1 with a reused
    context is 5-6% faster (it had fallen 4.5% behind the main branch),
    cyclic Alice 2.4-2.9% at levels -5 to 2. Frames are unchanged.

  • Benchmarks refreshed on AMD, against C and the main branch. The
    published comparison is a new complete sweep of all 384 encoder and 384
    decoder cases on a Ryzen 9 7950X, and its pages and charts carry a third
    series, the main branch measured beside the working tree
    (--main-csv in level_docs.py, plot.py and decoder_docs.py). A new
    silesia.py times C, main and the working tree in random order on whole
    Silesia files with reused contexts. The earlier Intel sweep and the
    interim AMD tuning reports are removed. See the
    benchmark results.

  • Reused-context and literal-decoding performance without unsafe. The
    row match finder views its kept tables a row further on at every context
    reset: entries a previous input of similar content left in the
    never-cleared tag table otherwise shared the new input's tags, and on
    repeated Alice frames the greedy and lazy levels 4-10 took up to 6.5%
    longer from finding and dismissing them. The reference salts its hash
    instead (ZSTD_advanceHashSalt), at an XOR per hash; the view costs
    nothing per position, so cold searches are unchanged. The
    decoder runs its block loops with AVX2's features on AVX-512 hosts (the
    compiler vectorized the Huffman streams' lookups into gathers there),
    decodes large four-stream Huffman sections into a literal buffer of a
    fixed size that lets the pair loop keep one output position per stream,
    and reads offset bits with one shift. The fast match path extends
    backwards with the reference's byte loop, compares a match's first word
    with one bounds test and keeps its sequence cursor's counts in
    registers: 1.4-1.7% faster on Silesia at levels -1 and 1, still 7-8%
    behind the reference there. Encoded bytes remain identical to
    zstd 1.5.7. See the benchmark results.

  • Alice performance follow-up without unsafe. Dense full fast blocks
    promote their sequence writer after 1024 sequences, lazy prefix parsing
    uses relative positions, and sequence code/width maps share one lookup.
    The decoder skips zero-width length reads, builds described FSE rows in
    the selected feature context, and continues Huffman pairs into the tails
    before exact stream-end validation.

  • AMD performance tuning without unsafe. The encoder's block
    pre-splitter runs in the selected SIMD feature context, uses exact 32-bit
    fingerprint arithmetic and keeps its fixed histograms on the stack. The
    decoder borrows compile-time predefined FSE tables and symbol rows, avoids
    an enum-conversion table lookup, and interleaves three Huffman streams
    while decoding the fourth separately. Encoded bytes remain identical to
    zstd 1.5.7.

  • Fixed a false framed_roundtrip fuzz crash. With one-byte chunks the
    target writes a chunk per payload byte plus metadata, which exceeded the
    4096-chunk decode limit of the framed targets. The limit is now 8192, and
    the ClusterFuzzLite input is kept as a regression. The library is
    unchanged.

  • Formal verification against RFC 8878, ported from mbrotli. Charon and
    Aeneas translate selected Rust items into Lean 4
    (scripts/extract_lean.py), and verification/lean proves them correct
    against a transcription of RFC 8878 itself, never against the C library:
    the frame header (descriptor, window formula, dictionary ID, content size)
    and the block header on both sides, with round trips from the encoder's
    writers through the decoder's parsers; Number_of_Sequences in its
    shortest form and every form the RFC defines; the literals length and
    match length codes, whose encoder picks the only code whose RFC range holds
    each length and whose decoder tables read it back; the offset code tables,
    the default distributions, the initial repeat offsets and the magic
    numbers; the encoder's repeat offset update for every offset value; and the
    FSE Number_of_Bits and Baseline of every state (Table 21). Each proof
    also shows the translated code cannot panic or overflow on the inputs it
    covers. The transcription checks itself against the RFC's worked examples;
    RFC 8878's Table 18 misprints its last two rows, and the proofs follow the
    RFC's text. ci-lean.yml checks that the translation matches the Rust
    source and that the proofs build with only Lean's standard axioms. To make
    the code translatable, FrameHeader::parse reads its fields by index,
    Number_of_Sequences is read and written by the new sequence_count and
    write_sequence_count, and the encoder's frame header is written from a
    FrameHeaderFields value; behaviour and output are unchanged.

  • hotpath Cloud with Callgrind, ported from mbrotli. New opt-in
    hotpath-callgrind and hotpath-cloud profiling features, and the
    optional hotpath dependency is updated to 0.29.2. The new
    profile_compressor example drives the encoder at levels -5 to 22 on
    repeated text, on a 17-byte input and on an optional corpus. The
    hotpath benchmark workflow runs it on Alice for every push to main and
    pull request in merged mode: natively for timing and allocations, then
    under Callgrind for instruction counts, in one report. It uploads the
    report to hotpath Cloud (pull requests through hotpath-relay.yml),
    judged by hotpath/policy.toml. The published crate is unchanged apart from the
    example.

  • Continuous fuzzing with ClusterFuzzLite, ported from mbrotli.
    fuzz/libfuzzer wraps all 14 AFL target bodies (11 stable, 3 framed) as
    libFuzzer targets, which .clusterfuzzlite/build.sh builds with
    AddressSanitizer, debug assertions and overflow checks and seeds from the
    AFL corpora and committed regressions. Pull requests that touch code get a
    ten-minute run on the changed code (ci-cflite-pr.yml). A daily workflow
    runs 5.5 hours of batch fuzzing, then corpus pruning and a coverage report
    (ci-cflite-cron.yml). ci.yml lints the adapters. The published crate
    is unchanged. The coverage build calls cargo build directly, because the
    image's cargo wrapper turned cargo fuzz build into a build of the root
    crate, so no targets were built. Pruning gets 10800 s instead of 600 s,
    which timed out merging the first target's corpus. run_fuzzers steps
    declare language: rust.

  • Miri and AddressSanitizer workflows, ported from mbrotli and run on
    every v* tag or on demand. ci-miri.yml runs scripts/miri.sh, which
    interprets only the tests that reach unsafe in dependencies: the
    fearless_simd vector kernels (opt_count match-length scans, the row
    finder's tag masks, the decoder's shuffle wildcopy and sequence execution
    around it) on the scalar, SSE2, SSE4.2 and AVX2 backends, plus AVX-512
    when the runner supports it (MIRI_TARGET_CPU=native in CI), and the
    borrowed-buf uninitialized output of both codecs. Tests are named
    exactly and each run checks its pass count. ci-sanitizer.yml runs every
    integration test with framed, then the same kernel modules natively,
    under -Zsanitizer=address with LeakSanitizer. Tests that leaked their
    inputs or &'static dictionaries through Vec::leak/Box::leak now
    borrow locals or a static OnceLock; the two tests that forget an
    owning guard on purpose run in a separate step without leak detection.
    Under cfg!(miri) the opt_count window
    test keeps only lengths next to 16-byte boundaries and
    compress_to_uninit_writes_the_one_shot_frame compresses 2 KB; native
    runs check a superset of the previous window cases.