v0.5.0

Added

• fqxv decompress - reads from stdin, streaming the archive straight into the
  decoder (it only reads forward and stops at the terminator frame). This is how you
  read a remote archive — pipe a transfer tool in, so all the auth, retries, and
  resume stay in the tool built for it and fqxv gains no HTTP dependency:
  aws s3 cp s3://bucket/reads.fqxv - | fqxv decompress - -Z | bwa mem ref.fa -
  (or curl on a presigned URL, gsutil cat, …). A truncated stream still fails
  (premature EOF) rather than yielding a short file. --recover and --split need
  a seekable file (a stream can't be rewound) and refuse stdin with a clear message.
• Python: read archives over the network. The fqxv wheel gains a
  dependency-free fqxv.remote module (standard-library urllib). The streaming
  entry points — fqxv.open, decompress_to_* — now accept any file-like
  object, so a boto3/urllib/httpx response streams straight in
  (fqxv.open(s3.get_object(...)["Body"])); fqxv.remote.stream(url) /
  download(url, dest) wrap that for a URL. fqxv.remote.RemoteArchive adds
  column projection over HTTP byte-range requests: fetch just the footer index
  from the archive tail, then only the names (~1% of the file) or the sequence,
  CRC-verified per stream. It rests on IO-free primitives —
  parse_index_suffix, per-stream ranges/CRCs on Index, and
  decode_{names,sequences,qualities}_bytes — that a custom async client can drive
  directly for concurrent range fetches.

Performance

• Reorder compress: SIMD merge scan, malloc-free placement, incremental
  cast_vote. Three byte-identical, zero-ratio-cost speedups on the hot
  short-read reorder assembly/merge phases (--order any/shuffle/--max).
  (1) The overlap-merge successor scan now compares 16 bytes at a time
  (_mm_cmpeq_epi8 + movemask popcount, budget checked per block) behind a
  runtime is_x86_feature_detected!("sse2") dispatch with a byte-identical
  scalar fallback (no raised global baseline) — the count matches the scalar
  loop exactly when within budget and exceeds it exactly when the scalar loop
  would break, so best_key and the archive are unchanged. (2) place_on_contig
  and the rescue assembler's try_place/place no longer allocate a
  Vec<usize> of mismatch positions per candidate: candidates are scored with a
  count-only pass and only the winner's positions are materialized (into a
  caller-reused scratch buffer on the clustered path). (3) cast_vote updates
  the per-column plurality in O(1) by comparing the just-incremented base against
  the current winner (lowest-index tie rule preserved) instead of max_by_key
  over all four counts. About 2% faster single-thread (54.87 s → 53.83 s,
  1.02× ± 0.00, on 1M-read NovaSeq --order shuffle compress); within noise at
  --threads 16 (1.00× ± 0.04), where the wall clock is dominated by the serial
  assembly prelude. Byte-identical archives on the validation datasets
  (NovaSeq/GAIIx/MiSeq/MGI, both --order shuffle and --max), and
  thread-deterministic (--threads 1 == --threads 16). (#219)
• Skip the quality-quantizer probe on Nanopore. The per-block quantizer trial
  (MODE_SEQ_BINMIX_Q) only wins on skewed PacBio HiFi/Revio quality; on the flatter
  Nanopore distribution it never does, yet the bounded prefix probe still ran and cost
  ~5% of ONT compress for 0 bytes saved. encode_seq now takes a try_quantizer flag
  and the container passes false on Nanopore, skipping the histogram build and the
  probe entirely. Output is byte-identical (the kept stream was the baseline either
  way); only ONT compress speed changes. HiFi/Revio still trial and keep the quantizer.

Internal

• Optimize the workspace codec crates in dev/test builds. The [profile.dev]
package."*" override optimized external dependencies but not workspace members, so
  the codec crates compiled at opt-level 0 for tests — and the test suite runs real
  compression, so the long-read round-trip tests took 30–45 s each. Per-package
  opt-level = 2 overrides for the compute crates cut the CI test suite roughly 4–5×
  with no behavior change (SIMD is runtime-detected; determinism holds across opt-levels).
  The CLI stays at opt-level 0 to keep the edit-compile loop fast.

• Byte-identical speed in the long-read overlap search (chaining). The minimizer
  overlap search (fqxv-lroverlap) is the top self-cost of Nanopore compress, and a
  overlap search (fqxv-lroverlap) is the top self-cost of Nanopore compress, and a
  query's per-anchor chaining dominates it (a fresh profile put Chainer::chain at
  ~23% and find_overlaps at ~14% of ONT compress). Three output-preserving changes
  strip wasted work without moving a single archive byte:

  • The chainer's redundant re-sort is dropped. find_overlaps already sorts a
    query's whole anchor set by (target, strand, tpos, qpos), so each
    (target, strand) group it hands the chainer arrives already in (tpos, qpos)
    order — exactly what the chainer's own sort_unstable would produce. A new
    Chainer::chain_presorted entry point skips that re-sort (it still dedups),
    saving hundreds of sorts per read at high coverage.
  • The chaining DP's scratch buffers are reused. The per-anchor score,
    predecessor, order, and used arrays — plus the per-chain path — were
    heap-allocated on every group (hundreds of small allocations per read). They are
    now thread-local buffers, cleared and resized per group, never read stale — so
    the chain set stays a pure function of the input.
  • The hot anchor-bucket sort is a radix sort. Each anchor's group-and-position
    key (target, strand, tpos, qpos) packs losslessly into a u128 whose
    ascending order is exactly the tuple order, so the per-query sort (~7% of ONT
    compress) is now a deterministic LSD radix sort producing the identical total
    order rather than a comparison sort.
    About 15% faster single-thread ONT compress (ecoli_ont/DRR205413,
    559 s → 475 s), where chaining is the largest self-cost. At 16 threads long-read
    compress is block-bound — a handful of large blocks gate the wall clock, not the
    per-anchor work — so the gain is dataset-shaped: ~noise on ONT (few, large blocks)
    but about 10% faster on high-coverage HiFi (ecoli_hifi, --platform pacbio,
    254 s → 228 s at 16 threads), where the many smaller blocks keep the cores fed.
    Archives are byte-identical on ONT (default and --max) and HiFi, and
    thread-count invariant (--threads 1 == --threads 16). Proptests pin the radix
    order to the comparison sort's and the presorted chain path to the sorting path.
    The levers mirror minimap2's own presorted chaining and radix_sort_128x. (#151)

• Per-block quality context quantizer, trialled and kept only when it wins
  (HiFi/Revio). The long-read quality coder used to build its recent-quality
  context with one fixed quantization (q1>>1/q2>>3/q3>>4), which merges
  adjacent Phred values — costly on HiFi/Revio, where quality is packed at the top
  of the scale and neighbouring high-Q values need to be told apart. The encoder now
  also builds a per-block quantizer from the block's quality histogram (the fqzcomp
  qtab / CoLoRd platform-quantizer idea): full context resolution where quality
  actually varies, equal-population folding elsewhere. Both quantizers code the
  block and the smaller is kept, with the choice recorded in a self-describing
  header mode byte (MODE_SEQ_BINMIX_Q) and the small table transmitted so decode is
  unambiguous — so a block can only match or shrink (never-worse by construction).
  Quality-stream savings, measured against a clean baseline: PacBio HiFi (ecoli,
  Sequel II) −1.17%, Revio amplicon −3.25%, Revio WGS −0.25%; Nanopore
  0% and short reads 0% (byte-identical) — neither is touched. A bounded
  prefix probe decides whether the second full encode is worth coding, so the trial
  costs ≈ +5% compress on Nanopore (where it never wins) and is paid in full only on
  the skewed long-read data where it does. Lossless and thread-deterministic;
  archives round-trip byte-for-byte and are identical regardless of thread count.

• Anchor-restricted long-read tile coding (CoLoRd-style). The multi-reference
  ONT tiler used to re-align each tile with one banded DP over the whole
  read × reference window, re-deriving the exact-match stretches its own minimizer
  chain had already proven identical. It now walks that chain, emits each anchor as a
  free Match copy-run, and runs the aligner only on the short inter-anchor gaps and
  the two flanks — so alignment work scales with divergence, not read length. About
  2.5× faster single-thread ONT tiler compress, with an aggregate ratio
  improvement across the ONT corpus (most accessions smaller — e.g. DRR205413 −2.9%,
  DRR351396 −2.8%, DRR424350 −2.6%; worst real-data case ≈ +0.3%). Lossless and
  thread-deterministic; the edit-op stream keeps the same semantics, so the decoder
  is unchanged and archives still round-trip byte-for-byte. A tile whose chain is not
  recovered falls back to the whole-window DP; FQXV_TILE_NO_ANCHORGAP=1 forces the
  DP everywhere for A/B. (#226)

• Anchor-restricted coding for the shared-reference consensus codec (HiFi). The
  same lever as #226, now on the non-tiler path the high-coverage HiFi/PacBio blocks
  take: each read used to be re-aligned against its consensus with one banded DP over
  the whole read × consensus window (align_banded, the dominant HiFi compress
  self-cost). It now recovers the read↔consensus exact-k-mer chain, emits each anchor
  as a free Match copy-run, and aligns only the short inter-anchor gaps and two
  flanks — so alignment work scales with the read's divergence from its consensus,
  not its length. On low-error HiFi the read shares nearly all of its k-mers with the
  consensus, collapsing the DP area. About 1.23× faster at-scale (16-thread)
  full-file ecoli_hifi compress (313s → 254s, two trials), at an equal-or-slightly-better ratio
  (−0.05% archive on the full file; byte-identical on a single-block subset, where the
  anchor path emits the same equal-cost edit stream). Lossless and thread-deterministic
  — the edit-op stream keeps the same semantics, so the decoder is unchanged and
  archives still round-trip. A read whose chain is not recovered falls back to the
  whole-window DP;
  FQXV_LRO_NO_ANCHORGAP=1 forces the DP everywhere for A/B. The anchor_chain /
  anchorgap_build machinery is now shared by both coders (new anchorgap module).
  (#220)

• Reorder drops the redundant v2 single-contig sequence candidate per block. On
  the adaptive rescue path (--order any/--max default) each block coded both
  the v2 single-contig codec and the v3 literal-rescue codec and kept the smaller.
  v3 generalizes v2 — it attaches the reads v2 strands as literals and otherwise
  degenerates to the same coding — so it is the block-local floor on its own;
  coding v2 too was near-redundant. v2 is now coded only under --no-rescue (its
  intended fast single-contig path). Output is byte-identical on the validation
  datasets (NovaSeq/GAIIx/MiSeq/MGI, --order shuffle and --max) — v3 was never
  larger than v2 on any block — for one fewer per-block sequence encode
  (~1.08× faster single-thread --max on a 2M-read NovaSeq subset).

• Single-end reorder codes quality and names once, not twice. The never-worse
  gate on --order any/--order shuffle/--max used to code the whole file both
  clustered and plain and keep the smaller full archive — coding the quality and
  name streams a second time. The keep/skip decision is really "did clustering +
  the permutation beat plain order-k on the sequence", so the gate now codes each
  candidate's sequence + names + permutation, decides on that non-quality total,
  and only the winning layout codes quality — once, in the order it emits (the
  clustered encode is split into a prepare/finish pair; the plain candidate reuses
  its coded names+sequence via a new write_plain_layout path). Output is
  byte-identical to the previous build on the validation datasets
  (NovaSeq/GAIIx/MiSeq/MGI, both --order shuffle and --max) and
  thread-deterministic; it removes a redundant whole-file quality/name encode
  (a small single-thread win, since quality coding is cheap next to the reorder
  assembly and the still-required plain sequence coding).

• Lower peak memory on Nanopore compression — the whole-input buffer is gone.
  compress_auto routed every long-read input to a buffered path that held the
  entire file in memory to build the whole-file shared reference (#168). #211
  disabled that reference on high-error Nanopore, leaving the buffer as dead
  weight there. Explicit-Nanopore single-end input now takes the streaming path
  instead (one block in flight, not the whole file): heap-profiling put the buffer
  at ~25% of peak, and on a 600 MB ONT file peak RSS drops 3.72 GB → 2.91 GB
  (−22%). The archive is byte-identical — streaming cuts blocks at the same
  MAX_BLOCK_SEQ_BYTES budget and, with no shared reference, each Nanopore block
  codes with the same plain per-block codec either way (verified by cmp and the
  determinism round-trips). HiFi still buffers (its shared reference pays off); an
  auto-detected ONT set without --platform also still buffers, since the
  streaming path detects platform from read names (issue #225).

Changed

• Long-read compress is faster at identical output. Two changes cut compress
  time without moving any ratio or byte of the archive: the overlap-consensus
  candidate that Nanopore always discarded is no longer built (#223, ~42% faster
  ONT default), and the banded-DP traceback is de-packed to one byte per cell
  (#222, ~25% faster ONT / ~13% faster HiFi). Combined, default-mode ONT compress
  is ~2.3× faster than v0.4.0. A full benchmark rerun on this build reproduced
  every ratio and per-stream size byte-for-byte.

Added

• Python bindings expose estimate() and verify() (#218).