v0.4.0

Added

• Best-of-N reference selection takes the ONT tiler to CoLoRd parity — the
  multi-reference tiling codec (SEQ_METHOD_TILE) now weighs several earlier-read
  references per tile and keeps the cheapest edit script, instead of blindly taking
  the furthest-reaching neighbour. At ONT coverage many earlier reads span the same
  region with independent error patterns, so the lowest-cost reference agrees with
  the query at more positions — the way CoLoRd picks its anchor, applied per tile.
  With a wider alignment band on top this is the dominant ONT sequence-ratio lever:
  on the ecoli_ont archive the sequence stream drops from 43.3 MB to 34.5 MB
  (1.150 → 0.915 bits/base), matching CoLoRd's ~34 MB, and the whole archive goes
  2.92× → 3.05× — losslessly (--verify). It is encoder-only (the tile block
  self-describes, so the decoder is unchanged and single-reference coding is
  byte-for-byte identical to before) and compute-heavy (~linear in the fan-out on
  the tiler's already-vectorised alignment), so it is gated to the top effort
  levels: the default keeps the single-reference cover and today's ONT speed,
  -l7/-l8 enable best-of-2/-4, and --max reaches the parity operating point.
  Nanopore long reads only. FQXV_TILE_BAND / FQXV_TILE_REFS override the band
  and fan-out for rebuild-free A/B measurement.

• Raw-LZMA sequence codec for ordinary-coverage long reads — a new per-block
  sequence method (SEQ_METHOD_LZMA) codes the block's ASCII bases with a
  large-window LZ, kept only when it beats the existing candidates. At ordinary
  long-read coverage — a real genome, not 300× of one organism — overlapping HiFi
  reads share long exact substrings that neither the within-read order-k model
  nor the consensus-edit overlap codec captures, but a large-window LZ finds
  directly. On the full PacBio Revio WGS run the archive goes from 9.72× to
  13.35× — from second-worst in the lossless field, below plain zstd19/xz9,
  to above both — losslessly. The overlap codec still wins on
  high-coverage/amplicon data, and the per-block keep_smaller gate picks the
  right one automatically. FQXV_SEQ_NO_LZMA disables the candidate for A/B
  measurement.

Changed

• --version reports git provenance on development builds. A binary built
  from a clean checkout of the release tag still prints just fqxv 0.4.0;
  anything else — commits past the tag, a dirty tree, an untagged branch —
  appends the git description (fqxv 0.4.0 (v0.4.0-7-gab12cd34-dirty)), so a
  bug report identifies the exact build. Source trees with no git (a crates.io
  tarball) print the plain version.

• Real progress bars for paired compress and decompress. Both paths showed a
  bare indeterminate spinner for the whole run — minutes of no information on a
  paired sample at a few cores. Compress picked its indicator on input count, so
  paired mates always fell through to the spinner even though the summed input
  length the bar needs was already computed; interleaving reads every input in
  lockstep, so one shared counter against that sum drives the same bar the
  single-input path already used. Decompress had no counter at all and now reports
  reads finished against the footer's read count (already read up front for the
  truncation check) — counting output records, not archive bytes consumed, which in
  --threads-sized batches runs ahead of the work and would pin at 100% while the
  last batch is still decoding. Both bars now carry an ETA. The reorder and stdin
  carve-outs still degrade to an indeterminate readout rather than report a
  percentage that would stall or lie, and --quiet and non-TTY output are
  unchanged (#189).

Performance

• Lower peak memory on Nanopore compression. The minimizer-occurrence record
  (Occ) that dominates the long-read overlap index is packed from 24 bytes to 16
  — the read index and the strand flag now share one u32 (read in the low 31
  bits, strand in the high bit) instead of a u32 beside a bool that padded the
  struct out to a third 8-byte word. That is a one-third cut of the index's largest
  allocation, on the order of a gigabyte on a large ONT block, so it directly eases
  the Nanopore memory pressure. Output is byte-identical — a hand-written Ord
  reproduces the previous (hash, read, pos, strand) sort order exactly — and that
  was verified both by the determinism round-trips and by an archive-level diff
  against the previous build.

• Nanopore compression is ~2.3x faster — the shared whole-file reference layout
  (#168) is now skipped on high-error Nanopore, where it never pays off. It helps
  low-error PacBio HiFi (a clean consensus stored once), but on noisy ONT the
  reference frame always costs more than it saves, so the whole-file gate (#184)
  rejects it every time — after building the whole-file reference and coding every
  block against it, a second full long-read assembly per block that profiling put at
  ~45% of ONT compress CPU, all discarded. Skipping it on Nanopore goes straight to
  the plain layout the gate would fall back to regardless, so the output is
  byte-identical — measured 7:20 → 3:11 on a 600 MB E. coli ONT file. HiFi is
  unaffected. Mirrors the Nanopore LZMA gate above.

• PacBio HiFi is smaller and ~2x faster to compress — the raw-sequence LZMA
  codec (SEQ_METHOD_LZMA, #197) now seeds its match finder on 12 bytes
  instead of 4. On raw ASCII a 4-byte seed keys only ~256 distinct DNA 4-grams, so
  the hash chains collided catastrophically and the depth cap truncated them —
  which both cost time (millions of pointer-chases through a capped chain) and
  lost matches beyond the cap. A 12-base seed keys ~16.7M grams, so a 12-mer
  recurs only ~a dozen times: chains are short, the cap never binds, and the
  finder sees every candidate. On a 40k-read Revio WGS subset the sequence stream
  drops 0.791 → 0.638 b/base and the whole archive 29.4 → 25.2 MB
  (−14%), while compress time falls ~2x (329s → 166s, below even the
  original 4-byte path). Validated lossless and on a second HiFi dataset. The
  match finder is tuned per call site, so the 2-bit packed-reference path keeps
  its 4-byte seed and byte-identical output; decode is seed-agnostic, so no format
  change. The match-extension inner loop is also AVX2-vectorized (CPU-feature
  gated, byte-identical, ~8% on top; FQXV_LZMA_NO_SIMD forces scalar).

• Nanopore compression is ~5.5x faster — the raw-LZMA sequence candidate
  (SEQ_METHOD_LZMA) is now skipped on high-error Nanopore, where it cannot win.
  LZMA pays off only where reads share long exact substrings — ordinary-coverage
  low-error PacBio HiFi (0.60 vs the overlap codec's 1.39 b/base); on Nanopore a
  base error every ~10 bases chops those matches short, so it reliably loses to
  the overlap/order-k codecs (2.2 vs 1.79 b/base) while costing the most encode
  time of any candidate. Because it runs on the serial block-0 probe as well as
  the fan-out blocks, coding-and-discarding it dominated the ONT wall-clock: on a
  600 MB E. coli ONT run compress drops 1362s → 246s with byte-identical
  output (LZMA was ~82% of the time and changed nothing). The gate keys off the
  detected platform, so PacBio still codes LZMA and keeps its win, and any
  platform where the loss can't be ruled out up front (Unknown) still codes it —
  the never-worse ratio guarantee is unchanged. keep_smaller continues to floor
  every block regardless.

• Long-read compression is ~40% faster where the shared reference wins
  outright. Making the whole-file gate exact meant coding both layouts for
  every block, which costs a second long-read assembly per block — on PacBio
  HiFi that roughly doubled compress time (225s → 461s on ecoli_hifi) to
  evaluate a candidate that never had a chance. The encoder now probes only the
  first block both ways and, when the reference wins by a wide enough margin
  to cover the reference frame three times over, skips the plain candidate for
  the remaining blocks. ecoli_hifi drops 461s → 275s with byte-identical
  output; ecoli_ont, where the plain layout genuinely wins, sees a negative
  margin and still runs the exact gate, so the ONT ratio is unchanged. The probe
  is integer arithmetic over block 0, so the decision stays thread-count
  invariant, and a shortcut block is still floored at order-k.

Fixed

• Higher effort no longer produces a larger archive. On a low-redundancy
  library (BGISEQ-500) --max and -l9 compressed worse than the default,
  because two effort knobs were applied unconditionally even where they cost more
  than they saved (#196). Both are now never-worse: (1) the level-8+ hashed
  high-order sequence tier is coded alongside plain order-k and the smaller kept,
  so enabling it can only help; (2) single-end --order any/--max codes the
  clustered and plain layouts and keeps the smaller, so reordering is used only
  when it pays — the same "code both, keep the smaller" rule the long-read shared
  reference (#192) and the overlap-vs-order-k choice already use. On the BGISEQ
  case --max drops from a 340 KB regression to matching the default; on a
  reorder-favourable NovaSeq set --max keeps its 2.4x win. Grouped (paired /
  10x) reorder is unchanged: it pays the permutation for spot reconstruction
  regardless, so the tradeoff differs.

• The long-read shared-reference gate now measures against the layout it falls
  back to. Adoption of the whole-file reference was gated on beating the plain
  order-k total, but the fallback path keeps the smaller of the per-block
  overlap codec and order-k — a stronger result than the bar being tested. A
  reference that lost to the per-block overlap codec could therefore still be
  adopted. Both layouts are now coded in the first pass and the smaller wins, so
  the never-worse property holds against the real alternative; the fallback
  reuses those streams instead of re-coding them. No archive changes on the
  benchmark corpus — ONT and HiFi output are byte-identical — so this closes a
  latent hole rather than changing current results (#184). Correction: this
  entry originally claimed coding both ways "costs no measurable wall time". That
  was wrong; it came from an A/B in which the build had not picked up the change,
  so two identical binaries were compared. Measured properly, coding the second
  candidate roughly doubles long-read compress time (ecoli_hifi 225s → 461s,
  six blocks). The correctness argument for the gate is unchanged, but the cost
  is real and is the price of an exact never-worse comparison.

• ONT seeding is chosen by index coverage, recovering 2.79 MB. Closed
  syncmers conserve anchors better than window minimizers at ONT error rates, but
  only once coverage is deep enough for the extra anchors to find partners. The
  long-read encoder builds two indexes on opposite sides of that crossover, and
  using syncmers for both put the ONT archive behind. Measured on ecoli_ont
  (block 1, 268 Mbase):

  index	syncmer	minimizer
  whole-file reference	1.280 b/base	1.416
  per-block overlap	1.517	1.243 b/base

  Each index now takes the scheme that suits its coverage, so both land on their
  better number instead of one giving up ~18%. ecoli_ont goes 216,381,249 →
  213,587,759 bytes (ratio 2.791 → 2.827), verified content-lossless. Both
  schemes share w and k, so anchor density and index cost are unchanged.
  PacBio is deliberately left unsplit — at <1% error minimizers are already
  near-optimal at either coverage — and HiFi output is byte-identical (#184).

• Interrupting a decompress no longer leaves a partial FASTQ at the
  destination. Ctrl-C during a decode published truncated output at its real
  name, and a short FASTQ is undetectable — BGZF blocks are self-contained so
  zcat exits 0 with no warning, and FASTQ carries no terminator or record
  count, so most of a run could sit there looking like a clean decode. Compress
  was already protected (sibling temp, rename on completion, registered with the
  signal handler); decompress created split mates and -o FILE directly at their
  final paths, so the handler had nothing to clean up and process::exit(130)
  never unwound to run Drop. Decompress outputs now route through the same
  AtomicOutput as compress, and the commit is deferred until after the footer
  read-count check, so a truncated archive also leaves the destination untouched
  instead of publishing a short file and then erroring. Recover commits too — its
  output is deliberately partial but still a finished artifact. Exit code is still
  130, and a full decode round-trips byte-identically (#188).

Security

• Decode-path allocation guards for the long-read overlap and reorder codecs.
  Several decoders sized allocations straight from untrusted on-disk length fields,
  before the backing bytes were known to exist — the #142 allocation-bomb class the
  tiler and the reorder reference decoder already guard against, ported to the
  siblings that had missed it. In fqxv-lroverlap's consensus overlap decoder
  (sequence methods SEQ_METHOD_OVERLAP / SEQ_METHOD_OVERLAP_REF) a ~10-byte
  crafted block could pre-reserve a read-count or contig-count vector from a raw
  header varint (a capacity-overflow abort under panic = "abort") or drive a
  multi-terabyte total_bases zero-fill; it now rejects an implausible
  total_bases up front against a shared MAX_BASES_PER_BYTE ceiling, leaves the
  count-driven vectors unreserved (each is bounded by the stream it reads from),
  and caps every insertion run at the read length. The reorder clustered / rescue /
  global block decoders bounded each input stream but not the aggregate
  reconstructed output, so a few kilobytes of MATCH-clone and CONTIG-copy ops
  could expand into unbounded memory; they now accumulate output bases against the
  same per-block ceiling the streams use. All of these are reachable from
  fqxv decompress on a crafted or corrupt archive (frame CRCs do not help — an
  attacker recomputes them). Decode-only and byte-identical on every valid archive;
  covered by a new regression test and the existing decode round-trips.