Skip to content

rapidgzip-rust v0.1.0

Choose a tag to compare

@rob-p rob-p released this 01 Aug 01:43
· 81 commits to main since this release

Initial release of the decoder-only rapidgzip-rust implementation.

Added

  • Parallel decoding of ordinary gzip streams using rapidgzip's speculative
    marker/window algorithm and zlib-rs as the inflate backend.
  • Correct verified decoding of concatenated multi-member gzip and BGZF,
    including empty members, optional gzip header fields, and conventional BGZF
    EOF members.
  • Specialized parallel paths for BGZF, stored DEFLATE streams, and archives
    containing dense runs of independently decodable gzip members.
  • A push API over ReadAt and Write, plus an owned streaming
    DecoderReader implementing Read + Send for consumers such as paraseq.
  • Elastic worker scheduling with affinity- and requested-budget-aware
    bootstrapping, empirical upward/downward calibration, lazy OS-thread
    creation, and persistent worker retirement.
  • Cloneable DecoderHandle runtime control and lock-free DecoderStats
    telemetry covering active, busy, and spawned workers; decoder path;
    backpressure; verified members; byte progress; and throughput.
  • Automatic decoder throttling when the final reader handoff is
    consumer-bound, with excess workers retiring after their in-flight work.
  • Runtime-dispatched AVX2 and SSE4.1 acceleration on x86-64, baseline NEON on
    AArch64, and scalar fallbacks.
  • The rapidgzip-rust decode/verify CLI, distributed in the
    rapidgzip-rust-cli package.
  • Multi-platform CI, an explicit Rust 1.87 MSRV check, safety documentation,
    reproducible performance reports, and a guarded release script.

Correctness and resource guarantees

  • Every accepted member must reach a real final DEFLATE block and match its
    CRC32 and ISIZE footer.
  • Candidate member headers inside compressed data are accepted only after
    independent inflation, footer authentication, and exact adjacency to the
    preceding verified member.
  • Output limits, speculative work, queues, reader handoff, and dense-member
    scanning are bounded.
  • Dropping a reader cancels and joins its pipeline; reaching EOF or calling
    finish verifies the complete input.

Current scope

  • Decoding only; compression is not implemented.
  • No persisted indexes, decoded-output seeking, or non-positional compressed
    input.
  • ISA-L-enabled C++ rapidgzip remains faster in the measured one-worker FASTQ
    cell; multi-worker parity and the zlib-ng-backed C++ performance gate are
    met on the published workloads.