v0.7.0

For those running the fgumi command line tools

v0.7.0 makes sort faster, hardens CODEC consensus, and brings dedup metrics to Picard/dupblaster parity.

Sort is up to 28% faster — and 2–3× faster than samtools sort

Whole-genome sort is the headline, at fgumi's default memory settings — no tuning required. On the benchmark fleet (--max-memory default 768M/thread), v0.6.0 → v0.7.0 template-coordinate WGS sort drops ~27–28%: 1kg-wgs-HG00096 goes 650 → 467s at 8 threads and 530 → 385s at 16. Against samtools sort on the same data at the same default settings, fgumi is ~2× on coordinate order and up to ~2.8× on template-coordinate, at equal-or-lower memory. The gain comes from three independent places: the sort engine itself (#846, #806, #814, #829); the read path, where the raw-BAM reader no longer zero-fills every record body before overwriting it (#797) and CRC verification is skipped on trusted piped input (#798, also 6–43% faster bwa mem | fgumi sort); and the multithreaded pipeline, where #860 restores ~15× decode parallelism at 16 threads — making consensus/group/filter/sort up to ~3.3× (worst-case 8.5×) faster on threaded runs. Sort output is byte-identical throughout.

CODEC consensus is hardened

Short-insert CODEC libraries get their duplex consensus fixed. The duplex consensus window is now clamped to the region both strands align (#767); the old window ran into the single-strand overhang that short inserts create when their strands dovetail — each read reads through past the other — and false-rejected those molecules as an indel disagreement between strands when neither CIGAR had an indel. Those molecules are now recovered, purely additively (nothing previously emitted changes). How much you recover scales with how short-insert / dovetail-heavy your library is — it is large on a strongly dovetail-enriched library and small on one with longer inserts, so re-baseline on your own data rather than assuming a fixed multiplier. --legacy-overlap-window restores the old fgbio-parity rejection (#861). Rejection reasons and codec --stats were also brought to fgbio parity (#754 now emits all 23 rows and fixes two miscounts; #755, #756), and a test-hardening pass fixed ten CODEC tests that could not fail (#768, #770, #774, #808).

dedup metrics now match Picard and dupblaster

fgumi dedup --metrics gained the library-complexity QC people expect from Picard MarkDuplicates and dupblaster: per-library rows plus an "All Reads" aggregate, read-level percent_duplication, a Lander–Waterman estimated_library_size (#799), a pair-vs-orphan duplicate breakdown (#815), a leading sample column, and an optional --duplication-ladder saturation curve — duplicate rate versus sequencing depth — off by default (#799, #841). Columns were added, not renamed or reordered, so name-keyed parsers need no change; positional ones do.

What else

• Consensus read-through bookkeeping. Dovetailed FR pairs (short inserts that read past their mate) now have their read-through trimmed in query space rather than reference space, across both the consensus callers and clip (#759, #760/#840, #853, #857). A correctness tidy-up we'd wanted for a while; it touches very few reads.
• Honest read accounting. Every read dropped by consensus is now counted and routed to --rejects — single-strand duplex rejections (#758), downsampled reads (#785), zero-length-after-trimming (#793). simplex --max-reads now caps per end rather than per group (#723, breaking — changes counts where it binds).
• Fixed pipeline OOMs. The multithreaded pipeline now honors --max-memory end to end (#764, #772, #775); previously a slow output sink could ignore the budget and balloon RSS (~31 GB in one repro), now a tight budget is respected.
• I/O and CLI safety. --check-crc / --no-check-crc is controllable and honored everywhere (#798, #805, #820, #842); a new retag command rewrites SAM tags (#824); output-collision detection now rejects any two outputs resolving to one path (#845); extract rejects mismatched-length paired FASTQs instead of silently losing records (#776); and pipelines fail on data left unflushed rather than truncating (#783).

---

What's Changed

• docs(cli): correct the TieRuleArg default and pin it with a test by @nh13 in #753
• fix(codec): emit the five codec stats rows fgbio reports by @nh13 in #754
• fix(codec): read out-of-range overlap boundaries the way fgbio does by @nh13 in #755
• fix(compare): qualify the content-mode diff count when pairing stopped early by @nh13 in #750
• fix(duplex): reconcile single-strand rejections with --stats and --rejects by @nh13 in #758
• fix(codec): route per-read rejections to the --rejects output by @nh13 in #756
• fix(consensus): clip overlapping reads by query distance, not reference distance by @nh13 in #759
• fix(codec): clamp the duplex window to the region both strands align by @nh13 in #767
• fix(codec): store the CODEC fixture's reverse read in reference orientation by @nh13 in #768
• test(codec): assert the bases the CODEC consensus tests are named for by @nh13 in #770
• test(codec): make the CODEC tests that cannot fail fail by @nh13 in #774
• fix(compare): treat spec-equivalent @hd SO/GO spellings as compatible by @nh13 in #784
• fix(consensus): count and route reads dropped by downsampling by @nh13 in #785
• refactor(pipeline): resolve clippy::pedantic cast-precision lints in base.rs by @nh13 in #790
• test(codec): assert the bases the codec_template consensus tests are named for by @nh13 in #808
• fix(extract)!: reject FASTQ inputs with different record counts by @nh13 in #776
• fix(pipeline)!: fail on data left unflushed at end of stream by @nh13 in #783
• perf(raw-bam): skip the record-read zero-fill via a spare-capacity read (#786) by @nh13 in #797
• perf(sort): read ahead deeply on the file the merge is blocked on by @nh13 in #806
• fix(pipeline): enforce the queue memory budget at the Read step by @nh13 in #764
• fix(pipeline): admit the gap-filler serial to Decode under memory-high by @nh13 in #787
• fix(pipeline): enforce the queue memory budget at the FASTQ Read step by @nh13 in #772
• fix(pipeline): separate the per-stage high-water mark from --max-memory by @nh13 in #775
• docs(pipeline): fix redundant explicit intra-doc links breaking the docs CI job by @nh13 in #822
• fix(duplex): count and route reads that trim to zero length by @nh13 in #793
• perf(bgzf): skip BGZF CRC verification on trusted stdin input by default (#786) by @nh13 in #798
• feat(dedup): per-library complexity metrics and duplication saturation ladder (#786) by @nh13 in #799
• fix(pipeline): saturate queue-byte debits so queue_bytes_in_flight cannot overflow by @nh13 in #811
• ci: run checks in the merge queue and cap job runtimes by @nh13 in #848
• test(pipeline): stabilize read-ahead backpressure assertion (#809) by @nh13 in #827
• fix(consensus)!: cap simplex --max-reads per end, not per group (#723) by @nh13 in #812
• test(pipeline): enforce backpressure fixture headroom centrally (#789) by @nh13 in #828
• ci: harden samtools install against apt mirror stalls by @nh13 in #825
• fix(clip): preserve non-UTF-8 string tag bytes when clipping by @nh13 in #833
• perf(zipper): probe tag membership with a 256×256 bitset by @nh13 in #830
• chore(deps): bump taiki-e/install-action from 2.85.4 to 2.85.12 in the all-actions group across 1 directory by @dependabot[bot] in #729
• chore(deps): bump the all-cargo-deps group across 1 directory with 9 updates by @dependabot[bot] in #781
• feat(bam-io): unify the single-threaded raw-BAM reader onto fgumi-bgzf (#800) by @nh13 in #842
• perf(filter): batch the interim progress counter out of the per-record path by @nh13 in #831
• fix(simulate): renumber grouped-reads MI ids into monotonic file order by @nh13 in #844
• perf: avoid three per-record heap allocations (group / downsample / overlapping consensus) by @nh13 in #832
• fix(clip): clip past-mate reads by query distance, not reference distance (#760) by @nh13 in #840
• feat(dedup): sample column and marginal duplication-ladder columns (#802) by @nh13 in #841
• fix(cli): reject colliding output paths across every command output by @nh13 in #845
• fix(pipeline): charge queue bytes before publishing on the processed queue by @nh13 in #850
• perf(sort): measure where the merge consumer's park time actually goes by @nh13 in #813
• perf(cigar): iterate raw CIGAR ops without a per-record Vec at four hot sites by @nh13 in #834
• feat(dedup): add Picard pair/orphan duplicate breakdown to --metrics (#804) by @nh13 in #815
• feat(bam-io): honor --check-crc on correct/group single-threaded paths (#800) by @nh13 in #805
• perf(zipper): make the single-thread path lightweight and CPU-efficient (#762) by @nh13 in #836
• perf(sort): give read-ahead depth to the file the merge is actually on by @nh13 in #814
• perf(extract): honor --check-crc for BGZF-compressed FASTQ input (#819) by @nh13 in #820
• perf(sort): name which floor the merge is on, and stop shouting the detail by @nh13 in #823
• feat(sort)!: put the performance diagnostics behind --sort-stats by @nh13 in #826
• perf(sort)!: cut the Phase 1 ingest thread's per-record cost by @nh13 in #829
• test(sort): bound the Phase 1 input path's cost with two harnesses by @nh13 in #837
• feat(sort): split the input reader's time between the disk and framing by @nh13 in #838
• perf(sort): overlap key extraction, reuse arena pages, and read with real queue depth by @nh13 in #846
• feat(retag): add retag command to rewrite SAM tags (copy/move/delete) by @nh13 in #824
• fix(raw-bam): capture overlapping selected aux tags independently by @nh13 in #851
• fix(consensus): trim read-through on dovetail FR pairs in the MC-tag path by @nh13 in #853
• refactor(sam): remove dead SamRecordClipper, port its tests to RawRecordClipper by @nh13 in #854
• docs(group): clarify the paired strategy never actually uses the index by @nh13 in #856
• perf(group): make parallel paired assigner dash-aware for asymmetric halves by @nh13 in #852
• refactor(sort): make --sort-stats run-scoped instead of a process-global static by @nh13 in #855
• fix(clip): clip dovetail FR overlaps and make the TLEN FR check crate-internal by @nh13 in #857
• fix(pipeline): bound Q3 decode admission by serial skew, not bytes by @nh13 in #860
• feat(codec): add --legacy-overlap-window for fgbio-parity output by @nh13 in #861
• chore: release v0.7.0 by @nh13 in #780

Full Changelog: v0.6.0...v0.7.0