Repository navigation
v0.2.0
-
Breaking: parallel streaming sessions are rebuilt around a window
and a non-blockingprocess.ParallelCompressor::start(stream, window)
returns aParallelEncoderSessionand itsParallelTaskQueue;
into_session(stream, window)returns the owned
ParallelEncoderSessionOwned(withinto_compressor). Sessions have
process,flushandfinish, which report the new
ParallelEncoderStatus(NeedsInput,NeedsOutput,Pending,
Finished) in aParallelProgress, andwait, which blocks for one task
completion.windowbounds the segments queued, running or awaiting
delivery, and so the session's memory; a full window reportsPending.
Each full segment becomes a task on the queue, which executors drain with
for task in queue { task.run() }(recvblocks;next_taskdoes not).
With a declared size and without a flush, the frame is byte-identical to
prepare_slicefor any chunking, window and execution order; an input
that fits in one segment takes the batch's serial fallback with or without
a declared size. Finishing, failing or dropping the session closes the
queue, so executor loops end; dropping every queue handle fails the stream
with the newParallelEncodeError::TaskQueueClosedorTaskAbandoned
instead of hanging. Stream failures are the newStreamFailedwith the
cause as its source, repeated on every later call.push,drain_into,
finish_into,run_queuedandqueue()are removed. -
ParallelEncoderSessionandParallelEncoderSessionOwnedhave
process_uninit, which delivers into&mut [MaybeUninit<u8>]such as
Vec::spare_capacity_mutwithout zeroing it first. It produces the same
bytes, counts, statuses and errors asprocess, writes nothing past
produced, and may be mixed withprocesson one stream. The AFL
paralleltarget alternates the two. -
Parallel streams have
std::ioadapters.
ParallelCompressor::writer(sink, stream, window)returns a
ParallelEncoderWriterandParallelCompressor::reader(source, stream, window)aParallelEncoderReader, each beside theParallelTaskQueue
their segment tasks go to, asstartdoes. Both drive the parallel session
and produce its bytes, so with a declared size and without a flush the
output is byte-identical toprepare_slicehowever writes or source reads
are split. Unlikeprocess, they block inwaitwhile the window is full
or a part is pending, so the queue must run on other threads. The writer
keepsEncoderWriter's transactional guarantees: nothing is lost to short
writes or sink errors,Write::flushisOperation::Flush, andfinish
returns the adapter insideFinishErrorfor a retry.reader.into_parts()
returns read-ahead source bytes as the newEncoderReaderParts
(source,unread_input), whichEncoderReader::into_partsnow returns
too.ParallelEncodeErrornow converts intostd::io::Error, taking the
kind of its root cause throughStreamFailedandBatchFailedand never
InterruptedorWouldBlock. The AFLparalleltarget also drives both
adapters. -
Batch tasks report exactly once: a task dropped unrun is
TaskAbandoned { segment }and one that unwindsTaskPanicked { segment }
(the panic still propagates and its engine is discarded). A failed batch
returns the newParallelEncodeError::BatchFailedwith the first failing
task's cause as its source, andfinish_intoleaves its output untouched;
InvalidStatenow only reports a task leaked withmem::forget. Batch
frames are unchanged. -
A new
parallelCriterion bench (benches/parallel.rs) compresses one
16 MiB input in 1 MiB segments on four executor threads through the batch, the session and the
writer, beside zstd's single-threadedZSTD_compress2. Measured with
cargo bench --bench parallel -- --warm-up-time 1 --measurement-time 3(Apple M5 Pro): all three write the same frame, and
the session and writer run within noise of the batch (level 1: 9.99,
10.08 and 10.02 ms; level 3: 17.62, 17.27 and 17.12 ms). zstd serial takes
18.09 and 30.37 ms; the parallel frames are 0.05% and 0.36% larger
(6286456 vs 6283433 and 5712713 vs 5692172 bytes). -
mzstdandmzstd-ffiare now dual-licensed under MIT or
Apache-2.0, at your option. The MIT text moved fromLICENSEto
LICENSE-MIT, andLICENSE-APACHEwas added. -
The README and crate docs have one "Choose your API" table covering
both codecs, thestd::ioadapters andParallelCompressor
(prepare_slice,start). In the crate docs it renders for every feature
set; APIs a build leaves out link to the feature-selection section. -
The
CI Fuzzworkflow archives AFL's findings before uploading them:
actions/upload-artifactrejects AFL's file names, which contain:. -
Added
Backend, a public, host-validated SIMD level
(fearless_simd):Backend::available,Backend::name,
Display/Debug,CompressorBuilder::with_backend/
DecompressorBuilder::with_backend, andCompressor::backend/
Decompressor::backend. The default is detected once when a codec is
built; every backend produces identical output.compressionand
decompressionnow depend onfearless_simd, andcompressionon
branchesfor safe prefetch hints. -
Decompression is 2.5-9x faster, now at 0.93-1.18x of zstd 1.5.7's time
oncargo bench --bench decompress(i7-13700KF, AVX2, one core),
with identical output and the same verdicts on malformed input:- Huffman and sequence bitstreams use a 64-bit container refilled where
the reference refills. Four-stream literals decode interleaved with
the reference's sentinel fast loops, with a pair (X2) table when the
reference's timing model prefers it. - Sequence tables carry baselines and extra-bit counts per state;
predefined tables are kept between blocks; tables and buffers are
reused. - Sequences execute with wildcopies, including a byte-shuffle kernel for
overlapping short-offset matches on SSSE3, AVX2, AVX-512, NEON and
WASM SIMD. decompress_to_slicedecodes straight into the destination when every
frame declares a content size that fits.
- Huffman and sequence bitstreams use a 64-bit container refilled where
-
Compression is 1.3-3.4x faster. On
cargo bench --bench compressit now
runs at 0.88-1.07x of zstd 1.5.7's time on small inputs and at levels
16-19, 1.03-1.17x on 1 MiB of text, and 1.1-1.3x on binary input and
in streaming. Frames stay byte-identical to zstd 1.5.7 on every backend:- The fast and dfast finders are specialized per minimum match length,
read through fixed-width windows over the block's prefix, and mask
their table indices; extDict variants work the same way. - The greedy, lazy, lazy2 and btlazy2 finders compare row tags with one
u8x16/u8x32/u8x64compare (ZSTD_row_getMatchMask), prefetch
rows 8 positions ahead as the reference does, and are specialized per
search, depth, minimum match, row size and extDict mode. - The optimal parser (btopt, btultra, btultra2) and LDM are specialized
per backend, optimization level and extDict mode; one binary-tree walk
serves insertion and match collection; prices come from tables
rebuilt at each rescale with a vectorized weight form. - Match lengths are compared a word at a time, then with native vectors.
- Entropy coding follows the reference's BMI2 builds: packed Huffman
codes with its two-container loop, interleaved histograms, the fast FSE
spread, and 16-byte literal copies. - Per call: block states swap instead of copying, and tables are reused
whenever indices continue, asZSTD_reset_matchStatedoes.
- The fast and dfast finders are specialized per minimum match length,
-
XXH64 stays scalar: an AVX2 kernel measured 1.8x slower (no 64-bit
vector multiply); the scalar code matches the reference's throughput.