Skip to content

rapidgzip-rust v0.3.0

Choose a tag to compare

@rob-p rob-p released this 09 Aug 14:18
· 6 commits to main since this release

Added

  • Opt-in DecoderPool process-wide decode budgets with a validated bon
    builder, runtime resizing, stable max-min cross-decoder allocation, fair
    contended admission, and aggregate queue/thread/activity telemetry. Existing
    decoders retain private elastic scheduling unless a pool is attached.
  • Persistent runtime growth through DecoderHandle::request_workers and
    reader forwarding methods. DecoderStats now exposes the explicit request,
    desired concurrency before pool contention, and whether the shared pool is
    limiting progress.
  • A concurrent programmatic-reader benchmark driver for shared versus private
    scheduling through both ordinary Read and paraseq FASTQ parsing.
  • Optional cpu-accounting feature reporting lifetime worker and auxiliary
    thread CPU through DecoderStats::completed_worker_cpu_time,
    completed_auxiliary_cpu_time, and cpu_accounting_failures. Each thread
    reads its own CPU clock exactly twice, at registration and exit, so no timing
    call, atomic, or branch is added to decode task begin/end or the inflate
    loops. With the feature disabled the accounting compiles out and the fields
    are None. Threads still running are omitted until exit, making this
    component accounting rather than an exact decision-window counter.
  • Optional busy-time-accounting feature exposing a monotonic cumulative
    executing-region integral as DecoderStats::accounted_busy_time. This gives
    a scheduler an exact busy-time signal that does not depend on its own polling
    cadence; reconstructing the same quantity by sampling executing-worker counts
    was measured 17-313% off on sparse, bursty, and stored decoder paths.

Changed

  • Marker/window admission now uses immutable configured decoder and pool
    capacity rather than the transient application ceiling. A low early throttle
    bounds screen execution but no longer permanently latches a suitable stream
    to the sequential path, allowing reliable later growth.
  • Pool execution accounting covers specialized workers, sequential chunks,
    bounded format scans, coordinator bridges/fallbacks, and ordered checksums.
    Permits are released before bounded result and final reader handoffs so a
    backpressured decoder cannot strand capacity needed by another file.
  • Shared BGZF and dense-member workers amortize global permit traffic with a
    worker-local lease that is retained only across nonblocking handoffs and
    immediately available work. Shared positional readers sample an
    allowance-sized logical output backlog only when the pool is divided, so
    broad per-file headroom does not multiply persistent parser buffering or
    penalize a sole fast reader.