bam: optional htslib writer behind the htslib-bam feature (~17% faster at 16 threads) - #153
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
bam: optional htslib writer behind the htslib-bam feature (~17% faster at 16 threads)#153BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
BGZF compression parallelises perfectly, but the obvious way to exploit that
with the existing stack does not work here: swapping in
`noodles_bgzf::MultithreadedWriter` measured ~37% *slower* at every size up to
a 44 MB BAM. `noodles-bgzf` is already built with its `libdeflate` feature and
both its writers go through the same `deflate::encode`, so the multithreaded
one only adds per-block channel and ordering overhead. That attempt was dropped
rather than shipped.
htslib's `hts_set_threads` is a different mechanism: a genuinely parallel
deflate rather than a layer over one compressor. Measured on 400k reads,
`--outSAMtype BAM Unsorted --outBAMcompression 6`, five interleaved A/B pairs
at `--runThreadN 16`:
noodles 2.55 2.61 2.51 2.30 2.39 median 2.51 s
htslib 2.08 2.22 2.00 2.16 2.02 median 2.08 s
about 17% faster, and the gap widens with thread count (9% at 4, 7% at 8) as
compression takes a larger share.
Off by default, and never built on Windows. `rust-htslib` ships pre-built
bindings for Mac and Linux only, its README calls Windows `bindgen` untested,
and its own CI has no Windows job — while this project's matrix requires
`windows-x86_64`. So the dependency is declared under `cfg(not(windows))`,
`default-features = false` drops bzip2-sys and lzma-sys (CRAM only), and the
default build and published crate are untouched by its existence.
Only the unsorted-BAM-to-file path is taken over, which is where BAM writing
can dominate a run. Stdout and the sorted path keep the default writer; they
have their own constraints and nothing to gain here.
Both backends are handed the same rendered header and the same records go
through the same SAM rendering, so they cannot drift in field encoding.
`backends_agree_on_the_decoded_record_stream` checks the contract: BGZF block
boundaries may differ, since that is framing and htslib chooses differently,
but the decoded record stream may not. On the 400k-read run the decoded output
is identical:
noodles b2051d153f9510ec57260e6450434507
htslib b2051d153f9510ec57260e6450434507
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optional htslib BAM writer behind the
htslib-bamfeature.What changed
src/io/bam_htslib.rsadds a second unsorted-BAM-to-file backend usingrust-htslib, selected by thehtslib-bamfeature. htslib'shts_set_threadsgives a genuinely parallel deflate; worker count is--runThreadN - 1, so a single-threaded run stays serial.Only the unsorted-BAM-to-file path is taken over — the one where BAM writing can dominate a run. Stdout and the sorted path keep the default writer.
Why
BAM writing was measured as the dominant cost at high thread counts, and
noodles_bgzf::MultithreadedWriterwas tried first and rejected (~37% slower; numbers on #143 and #151). htslib is a different mechanism, not a layer over the same compressor.Five interleaved A/B pairs at 16 threads:
~17% faster, non-overlapping across the five pairs. Decoded records identical (
b2051d153f9510ec57260e6450434507).Platform gating
rust-htslibships pre-built bindings for Mac and Linux only; its README calls Windowsbindgenuntested.windows-x86_64.So the dependency is declared under
cfg(not(windows)), the feature is off by default, anddefault-features = falsedropsbzip2-sysandlzma-sys(CRAM only).cargo buildandcargo testwith no features never compile htslib. The default build and the published crate are unchanged.How the backends are kept honest
Both are handed the same rendered header, and every record goes through the same SAM rendering before htslib parses it, so they cannot drift in field encoding.
backends_agree_on_the_decoded_record_streamstates the contract: BGZF block boundaries may differ — that is framing, and htslib chooses them differently — but the decoded record stream may not.Verification
cargo test --release(default features): 561 lib + integration, greencargo test --release --features htslib-bam: greencargo clippy --all-targets -- -D warnings, with and without the feature: cleancargo fmt --check: cleanTrade-off on record
This pulls htslib (C) into a project that is otherwise pure Rust apart from libdeflater, on a crate published to crates.io. Optional and platform-gated limits the blast radius but does not remove the dependency-story change. If the dependency is unwanted, the measurement still stands on its own: it locates where the BAM write cost is.
🤖 Generated with Claude Code