Repository navigation
rapidgzip-rust v0.1.0
Initial release of the decoder-only rapidgzip-rust implementation.
Added
- Parallel decoding of ordinary gzip streams using rapidgzip's speculative
marker/window algorithm and zlib-rs as the inflate backend. - Correct verified decoding of concatenated multi-member gzip and BGZF,
including empty members, optional gzip header fields, and conventional BGZF
EOF members. - Specialized parallel paths for BGZF, stored DEFLATE streams, and archives
containing dense runs of independently decodable gzip members. - A push API over
ReadAtandWrite, plus an owned streaming
DecoderReaderimplementingRead + Sendfor consumers such as paraseq. - Elastic worker scheduling with affinity- and requested-budget-aware
bootstrapping, empirical upward/downward calibration, lazy OS-thread
creation, and persistent worker retirement. - Cloneable
DecoderHandleruntime control and lock-freeDecoderStats
telemetry covering active, busy, and spawned workers; decoder path;
backpressure; verified members; byte progress; and throughput. - Automatic decoder throttling when the final reader handoff is
consumer-bound, with excess workers retiring after their in-flight work. - Runtime-dispatched AVX2 and SSE4.1 acceleration on x86-64, baseline NEON on
AArch64, and scalar fallbacks. - The
rapidgzip-rustdecode/verify CLI, distributed in the
rapidgzip-rust-clipackage. - Multi-platform CI, an explicit Rust 1.87 MSRV check, safety documentation,
reproducible performance reports, and a guarded release script.
Correctness and resource guarantees
- Every accepted member must reach a real final DEFLATE block and match its
CRC32 and ISIZE footer. - Candidate member headers inside compressed data are accepted only after
independent inflation, footer authentication, and exact adjacency to the
preceding verified member. - Output limits, speculative work, queues, reader handoff, and dense-member
scanning are bounded. - Dropping a reader cancels and joins its pipeline; reaching EOF or calling
finishverifies the complete input.
Current scope
- Decoding only; compression is not implemented.
- No persisted indexes, decoded-output seeking, or non-positional compressed
input. - ISA-L-enabled C++ rapidgzip remains faster in the measured one-worker FASTQ
cell; multi-worker parity and the zlib-ng-backed C++ performance gate are
met on the published workloads.