v0.19.0

Added

• A waveform bundle declares where its per-base sequence comes from
  (#312). waveform_model.preprocessing.reference_source is accepted, and a
  value this runtime does not assemble is refused at load, naming both what the
  bundle asked for and what escpod does.

  The bundle already carried motif_reference: "fasta" and a focus_rule
  saying to "find CCAGGC in the REFERENCE". Both are true, and both are about
  locating the anchor in reference coordinates, which the FASTA is fine for.
  Neither says where the bases come from — and every corpus built so far
  takes those from each read's MD tag, via pysam's
  get_reference_sequence(). Reading "the REFERENCE" the obvious way and
  slicing the FASTA is what this runtime did until #306, and it validated
  cleanly: 87 of 256 corpus chunks bit-identical against 256 of 256, with
  nothing raised on the other 169.

  It is not a difference that can be resolved upstream. The N in all 47
  records of the shipped panel is the 3'-terminal base of the 5' adapter's RNA
  tail CUGGN, ordered degenerate on purpose; over 1,055,660 reads no
  letter covers even 55% of that position, so substituting the modal base
  would be silently wrong for 45% of reads — strictly worse than an N, which
  is at least visibly unknown. The ambiguity is a permanent property of the
  input, so the runtime has to be right about its source permanently, and a
  declaration it can refuse is the only thing that makes that checkable.

  Absent means md, so every bundle already published stays readable —
  WaveformPreprocessing is deny_unknown_fields, which is also why the key
  has to be accepted here before escapepod-models can emit it
  (rnabioco/escapepod-models#109), the same sequencing
  barcode_crf_fdx4_rna004's signal.anchor used.

  A fixture now carries the ambiguity: trna_reference_ambiguous.fa is the
  fixture panel with the code put back where the real one has it. The existing
  reference has the ambiguity removed — 0 codes against the panel's 47 —
  so every fixture read took the resolved path and no golden here could ever
  have seen the difference, the same blind spot #306's own feature_set had.
  The new test pins that the ambiguity changes nothing end to end, and, in the
  same pass, that putting the same N into the MD-derived sequence does
  move the chunks — so the equality is evidence about the source rather than
  two paths that never differ.

• escpod classify runs a waveform_model charging bundle (#306).
  A third bundle variant, beside gbm and feature_model: it reads a signal
  window rather than a column vector — normalised current plus its k-mer
  residual, the sequence k-mer context scattered along the signal axis, and 12
  per-base dwell/level rows — and emits a single BCE logit. On the feature
  network's own training rows and test reads it is worth +0.0050 AUROC
  (n=1,387,667; 3 seeds x 2 geometries, 6/6 positive, paired sd 0.00008), and
  0.988 recall against 0.960 at the fnn's own FPR.

  The chunk assembly moved down into escapepod-signal
  (escapepod_signal::chunk) rather than being written a third time here. It
  was already implemented twice — leech's Python dataset and leech-core's Rust
  pipeline — and the failure mode is on the record: escapepod-classify
  reproduced a superseded feature definition for two months and its counted
  golden missed it, because all 19 fixture reads took the other branch. The
  module is generic in the way that matters: the anchor is a base index, the
  window is (left, right) samples, and the channels are a list the caller
  supplies, so two models that read the same twelve rows in a different order
  are two Vecs rather than two code paths.

  Nothing about those rows is hard-coded here. The bundle ships
  waveform_model.channels.{signal,features}.order — asked of the corpus
  builder at build time rather than transcribed — and the runtime resolves it,
  refusing a name it cannot compute or a length that disagrees with the count
  beside it. This is the one rule no shape check can catch: permute the rows
  and the tensor still has exactly the dimensions the graph wants, every read
  still scores, and the answers are wrong.

  Two further rules are cross-checked rather than assumed, because each fails
  silently. preprocessing.motif/motif_offset must agree with the anchor
  block — this variant anchors at motif +2, one base earlier than the
  feature-grid variants' +3, and inheriting the other offset places every
  window off-anchor and validates cleanly. And the graph's single logit is the
  logit of whichever class the bundle names: leech assigned its class
  integers at merge time and gave charged 0, so P(charged) is
  1 - sigmoid(logit) here, and reading it the obvious way inverts every call
  without erroring.

  The refinement refusal at load is narrowed rather than removed: the column
  variants still cannot reproduce a banded-DP pass, and are still refused for
  it; the windowed variant reproduces it from its declared parameters.

  The reference each read is scored against comes from its own MD tag, not
  from the FASTA. That is how the training corpus builds it (pysam's
  get_reference_sequence()), and the two disagree wherever the FASTA carries
  an ambiguity code — every reference in the shipped tRNA panel holds exactly
  one N, where the alignment recorded a concrete base. It is not a one-base
  difference: levels are looked up per 9-mer, so one unknown base makes nine
  consecutive k-mers unknown and leaves a run of zero levels the corpus does not
  have, which the banded DP then walks a different path through for the rest of
  the read. Slicing the FASTA instead cost 169 of 256 chunks their
  bit-exactness, with boundaries moving a sample throughout the read and the
  feature window 96 bases downstream wrong — while erroring on nothing. A read
  without an MD tag is refused (no_md_tag) rather than fallen back on, for
  that reason. The assembly is now bit-identical to the corpus on 256/256
  chunks, end to end within 3.3e-6 — the graph's own residual.

  It runs through tract, statically linked, like every other ONNX graph
  escpod runs — so it is in the default build and works from a stock release
  binary. That is true only from charging_tcn_rna004@v0.1.1 onward, and the
  reason is worth recording, because it is the second time this model family
  has hit it.

  The first export, @v0.1.0, could not go through tract at all: not for want
  of an op, but because tract 0.23's shape inference cannot close it. Measured
  five ways, and the failures fall into two independent causes, either of
  which alone is enough:

  • dynamo writes a value_info entry for all 667 intermediates with the batch
    axis as the symbol batch. A consumer that pins the batch — which every
    ONNX loader here does — cannot unify that, and tract dies at the first
    convolution with Sym(batch) vs Val(1). Every other graph escpod loads
    carries zero value_info; the legacy TorchScript exporter never wrote any.
  • adaptive_avg_pool1d(390 -> 11), which dynamo open-codes into a rank-8
    GatherND because the output size does not divide the input.

  It is not nn.MultiheadAttention, which this changelog, the module doc
  and escapepod-models#96 all named until somebody read the graph. The
  offending GatherND consumes relu_17, the last block of signal_tcn; its
  (11, 37) bool mask is a bin mask and its (11,) divisor is
  [36, 36, 37, 36, 37, …], that pool's bin widths. cross_attn exports as
  plain Mul/MatMul/Softmax/MatMul/Gemm, with no mask and no gather.
  #306's original suspect was right and its retraction was not.

  Neither onnx-simplifier (which folds away every Shape node) nor
  onnxruntime's own optimiser makes it loadable, so it could not be papered over
  at load time the way fnn::hoist_conv_padding papers over padded
  convolutions. This ran through onnxruntime via ort for exactly as long as
  that was true — which, ort being load-dynamic, meant a libonnxruntime.so
  on ORT_DYLIB_PATH and no way to run such a bundle from a static-musl release
  at all.

  The fix belonged in the export, and is the one this family already needed once:
  escapepod-models retracted "tract cannot run Resize" in July after finding
  the same shape-inference cause, an export fix costing one line and no retrain,
  and a 6x tract speedup for free. @v0.1.1 is that re-export (leech 0.10.0) —
  the pool written as one MatMul against a constant segment-mean matrix,
  value_info stripped, same weights, no retrain, evaluation bit-identical,
  479 -> 319 nodes and GatherND 2 -> 0. With it go the ort dependency, the
  classify-waveform feature, and the per-rayon-worker session pool that ort
  needed because Session::run takes &mut self. Re-measured from this side
  with escapepod-demux/examples/tract_dynamo_probe.rs, kept so the claim can
  be re-run against a later tract or a later export (its counts are tract's own
  after parsing, hence larger than the ONNX node counts):

  v0.1.0   669 nodes   analysis fails at node_GatherND_329 / node_index
  v0.1.1   471 nodes   optimized to 655, runs, output [1, 1]
  

  So an unloadable graph is now a bundle problem with a named fix and a
  build-time gate on it (escapepod-models#96 and #97), rather than a runtime
  gap: @v0.1.0 fails at load with tract's own analysis error and the file
  named. The lesson generalises, and is why that gate exists: onnxruntime loaded
  the broken graph perfectly, so the export's own torch round-trip was green
  throughout. "It exports and agrees with torch" is a weaker claim than "a
  runtime can load it".

  The swap is not free and the cost is worth stating: on the same harness and
  the same 256 chunks, tract is 6.27 ms/chunk against onnxruntime's 4.4,
  single-threaded — about 1.4x. It is paid back by static linking (the variant
  is reachable from a release binary at all, which it was not) and recovered in
  practice by rayon, which this pipeline already fans out across. Graph parity
  is unaffected: max |dlogit| 3.3e-6 over the corpus's own tensors, median
  4.8e-7, against an export whose own residual vs torch is 1.3e-5.

  The bundle's shipped Platt calibration is carried, not applied: the
  operating point beside it is stated on the uncalibrated probability the graph
  emits, so calibrating silently would move the scale out from under the very
  threshold it ships with. escpod says so at load.

• demux --model is repeatable, so several barcode axes are called in one
  pass over the POD5. A dual-indexed library carries a 3' index and a 5' one;
  reading them with two runs decompresses every read twice.

  escpod demux reads.pod5 --model ldx=ldx32/ --model fdx=fdx4/ --annotate
  escpod annotate --design samplesheet.csv reads.pod5   # ldx,fdx -> library,condition

  With more than one model each is given as NAME=PATH, and NAME is that
  axis's sidecar annotation column — which is exactly what annotate --design
  keys a multi-column samplesheet on, a sidecar feature that until now nothing
  could produce in one command. The name is the operator's, not the bundle's:
  it names the axis in your experimental design, and the same model can serve
  different axes in different runs. NAME=PATH is resolved against the
  filesystem first, so a bundle directory that itself contains = is not
  misread.

  What is shared is the expensive prefix — one POD5 sweep, one VBZ decode, one
  adapter detection — and not the encoders, which are per-model and are most of
  the cost. Expect meaningfully less than two separate runs, not close to one.
  Measured on 5,000 FDX Run1 reads, the fused ldx+fdx run is byte-identical
  to the two separate runs, read for read, on both axes.

  Constraints, each an error rather than a surprise:

  • Several models require --annotate/--classifications, not -d.
    Writing one POD5 per barcode is single-axis; split afterwards with
    demux split --sidecar --annotation ldx.
  • Only CRF bundles combine. The fingerprint heads (DTW-SVM, GBM) do not
    declare their own segmentation geometry — it is a compiled-in default
    shared by every such model — so a second one would silently be
    fingerprinted with the first one's parameters. Checked before any model is
    opened, so the refusal costs no ONNX load.
  • Every axis that needs a boundary detector must pin the same one.
    Detection runs once per block and its answer goes to every head; a
    different detector means a different adapter_end and a different amount
    of signal decoded per read. Compared on method, weights (by sha256, so two
    bundles shipping their own byte-identical copy agree) and input geometry.
    The GPU path runs several axes too. Each encoder worker holds one session
    per axis and runs its sub-block through all of them, and the pool is divided
    by the axis count so the number of live ORT sessions per device is exactly
    what a single-axis run uses — DEVICE_ROW_BUDGET is per device, and four
    unshared workers on one 24 GB A30 previously exhausted it and killed the run.

  Measured on an A30, 40 k reads, ldx+fdx, warm cache, arms interleaved (never
  ascending — page-cache warming has faked a 1.5× here before), two reps:
  fused 14.6 s against 8.4 s + 8.4 s = 16.8 s for the two separate runs, so
  1.15×. That is the shape to expect and not a disappointment: the shared
  prefix is ~2.2 s of a ~8.4 s run and the encoders are the rest, so fusing
  saves the prefix once rather than halving anything. It should matter more on
  a cold BeeGFS mount, where the POD5 sweep is a much larger share — not
  measured.

  Parity is exact where it must be. On 5,000 reads the fused GPU run's ldx
  and fdx columns are byte-identical, read for read, to the two single-axis
  GPU runs; fusing changes neither axis. Fused CPU output is likewise
  identical to the two separate CPU runs. CPU against GPU differs on 3 reads in
  5,000 (99.94%), which is this pipeline's existing tract-versus-onnxruntime
  variance and is present with or without fusing.

  A single-model run is unchanged in every respect: the sidecar column is still
  barcode, the classifications CSV header is still
  read_id,barcode,confidence[,crf_*], and the per-read output is byte-identical.
  Only a multi-axis run prefixes its columns (ldx, ldx_confidence,
  ldx_crf_margin, …).

• One .p5s for a whole directory. A run that produced fifty POD5s
  produced one set of barcode calls, not fifty — so escpod annotate and
  escpod demux --annotate now write a single collection sidecar when
  they are pointed at a directory: run1/pod5/ gets run1/pod5.p5s. It is
  the same Arrow table as a per-file sidecar with one extra index column
  (member_idx) and a member table in the schema metadata under
  escapepod:members, so pyarrow reads the whole run in one call with no
  join:

  import pyarrow.ipc as ipc
  table = ipc.open_file("run1/pod5.p5s").read_all()

  Read UUIDs are unique across files, which is what lets one set of columns
  cover every member. The path rule is unchanged — append .p5s to the path
  you named — so a collection sits beside the directory and can never
  collide with a member's own *.pod5.p5s inside it. Declared as
  escapepod:p5s_version 3; an escpod that predates it refuses the file by
  name ("is a collection sidecar covering a directory of POD5 files") rather
  than reporting a version number or a missing column.

  A collection is bound to N files, so the file-level identity gate moves down
  a level: a POD5 gets rows from a collection only when its footer UUID and
  byte size match a member entry. A file that appeared in the directory after
  the collection was written is told it has no sidecar rather than inheriting
  a neighbour's labels.

  Nothing downstream had to learn the new shape. view --include, filter --annotation, demux split --sidecar and the Python Reader all ask a
  POD5 for its columns, and that lookup now consults the file's own .p5s
  first and then the collection beside its directory, merging the two per
  column with the file's own sidecar winning — so an index-only sidecar from
  escpod index coexists with an annotated collection and neither hides the
  other.

Changed

• ChargingBundle's offsets/columns/span_mode moved behind
  feature_space(), and recipe()/select_columns() are now fallible. A
  windowed bundle has no columns at all, and three fields that would have to be
  empty for it cannot distinguish "no feature space" from "a feature space with
  nothing in it".

• A directory argument to annotate / demux --annotate no longer also
  writes per-file sidecars. The labels go to the directory's collection and
  nowhere else; writing them into every member as well would be fifty copies
  of one result to keep in step. Naming files individually is unchanged — with
  no directory there is nothing for a collection to sit beside, so each file
  gets its own sidecar as before. Per-file index and signal-geometry caches
  remain escpod index's job, and the two shapes coexist.

• escpod signal classify is escpod classify again. The command moved
  under a signal group in 0.11.0 so the word classify could not be
  confused with escpod demux classify, which assigns a barcode from a DTW/GBM
  adapter fingerprint. That traded a small ambiguity for a bigger irregularity:
  every other tool in this binary is a single word, so the only read-level
  model runner became the one command you had to know a namespace to find — and
  the namespace held exactly one subcommand. The two were never actually
  ambiguous in use, either: demux classify is a stage of the demux workflow
  that consumes a fingerprint CSV, while this takes a POD5 and an aligned BAM
  and writes cl onto the BAM.

  escpod signal classify keeps working: like the top-level spelling it
  replaced, it is now the hidden deprecated alias, warning
  warn: `escpod signal classify` is deprecated; use `escpod classify`. and
  forwarding to the same runner. The end-to-end test that pinned the previous
  alias is pointed the other way, so the two invocations still have to produce
  byte-identical calls rather than merely both exiting zero.

  The cl encoding, flags, output and bundle contract are unchanged. The only
  other user-visible difference is the @PG record on the output BAM, which
  now records escpod classify --model … as the command line.

• The demux summary no longer claims to have written files it did not. A
  sidecar-only run now ends N reads across M label(s) rather than M barcode file(s), and a multi-axis run reports N reads x A axes instead of a total
  that counts every read once per axis.

Fixed

• Charging bundles from current escapepod-models are loadable again
  (#314). MetaFile accepts an optional top-level basecaller block.

  escapepod-models#106 started emitting one in every charging bundle. The
  schema is deny_unknown_fields and had no such field, so every charging
  bundle built from that builder was refused at load by every escpod — on
  main, not just in a release, with unknown field \basecaller`` and a list
  of the fifteen keys it did know. The refusal worked exactly as designed; the
  field was simply missing. Absent still means "not declared", so the seven
  bundles already published keep loading.

  It is a named block rather than free-form provenance, and that is the
  judgment call. The charging feature set is mean + z-scored k-mer residual
  and the expected level is predicted from the read's own basecall — taking it
  from the reference instead costs 0.110 AUROC — so a charging model
  substantially detects how the basecaller fails at the aminoacyl adduct, and
  swapping the basecaller changes what its dominant feature means. Measured
  rather than argued (escapepod-models#108): the same reads called two ways,
  through one model and one shared label vector, lose ~0.0097 AUROC across two
  flow cells, lose 3.0-3.2 pp of TPR and gain ~0.4 pp of FPR at the shipped
  set point — so no threshold recovers it — and flip 3.9% of per-read
  calls, while the aggregate charged fraction moves 0.04 pp. The one
  statistic anyone would check when changing basecaller reads "no change" while
  one read in 26 answers differently, which is what a free-form note does not
  catch and a readable field does.

  Carried, not enforced. escpod cannot know what called the BAM it is
  handed without reading @PG, so it states the declaration at load —
  bundle was trained on basecalls from rna004_sup@v6.0.0 (dorado 2.1.1+…),
  or a line saying the bundle does not declare one — and leaves the comparison
  to the caller. ChargingBundle::basecaller exposes it. Refusing on mismatch
  could follow now that the identity is readable. The precedent is on the
  record: the 20260825 RLA QC scored v6 data with a v5.3.0-trained bundle, and
  every model shipped before #106 states the requirement ("aligned BAM with
  mv/ns/ts tags") and never the identity, so nothing could have caught it.

  The block itself stays closed, like every other declaration here — a key
  inside it this runtime does not implement is refused rather than dropped.
  That is the same strictness that caused this bug, kept deliberately: the
  alternative is a rule silently ignored. What the incident actually argues for
  is sequencing, which is now the rule on both sides — a new key is accepted
  here, released, and only then emitted (escapepod-models#113 gates emission on
  the pinned escpod version).

• ESCAPEPOD_CRF_GPU_TRACE reports reader decode as CPU time, not wall
  time. The shard's decode figure was an Instant around a par_iter, which
  measures the calling thread's slice of an N-thread rayon stage and counts
  the time it sits blocked on a page fault or a join. It read as a large,
  stable share of the run and pointed at decode as a bottleneck it was not.
  The clock is now CLOCK_THREAD_CPUTIME_ID, sampled inside the decode
  closure where the work actually happens, and the two syscalls per read that
  costs sit behind the same ESCAPEPOD_CRF_GPU_TRACE=1 that already gates the
  trace — an ordinary run pays nothing. Diagnostic output only; no classified
  read changes.

• A CRF bundle's signal.anchor is read instead of dropped, so a read-end
  model is no longer windowed onto the 3' adapter. SignalSpec parsed only
  chunk and stride, and carried no deny_unknown_fields — so
  "anchor": "read_end" and "window": "[read_end - chunk, read_end]", both
  shipped in barcode_crf_fdx4_rna004@v0.1.1, were discarded at parse time and
  the window was taken as [adapter_end - chunk, adapter_end].

  This matters because the two anchors sit at opposite ends of the molecule.
  RNA004 translocates 3'->5', so a 3'-adapter index (ldx, nbc) is found by
  windowing back from the boundary detector's adapter_end, while a 5' index
  (fdx) goes through the pore last and its adapter is simply where the signal
  stops. Running the second against the first decodes the far end of the read
  and returns exactly the output shape it should — a confident wrong answer, no
  error, nothing downstream that can detect it. It is the consumer-side half of
  the defect escapepod-models#100 fixed at the source.

  Measured, because "runs and is wrong" needs a number. Every molecule in the
  FDX Run1 pool carries both a 5' fdx index and a 3' ldx index, and ldx code
  is nested inside fdx library (escapepod-models config/fdx4_libraries.yaml),
  so the shipped barcode_crf_ldx32_rna004 call is an independent per-read
  truth label — different adapter, different end of the molecule, different
  model. Over 5,000 reads (4,113 with an in-pool ldx call), the same fdx4
  weights on the same reads:

  window	agrees with ldx truth	precision when called	unclassified
  [read_end - 3500, read_end] (fixed)	92.3%	92.4%	2
  [adapter_end - 3500, adapter_end] (before)	20.4%	25.9%	865

  Four codes, so 20.4% is chance. The fixed path lands on the bundle's own
  published numbers (balanced_recall@0.97 0.918, balanced_precision@0.97
  0.944); the old one called 79% of reads anyway, at a median edit-distance
  margin of 16 — the panel's maximum — so no per-read confidence field
  distinguishes the two. The two windows agree on 18.4% of reads, and the
  damage is not uniform noise: fdx03 collapsed from 950 reads to 21, which in
  an experiment deletes a library rather than degrading it.

  There was a second, independent instance of the same bug: a CNN detector's
  signal_decode_bound truncates each read to its leading max_obs_trace
  (16 000) samples, so "the read end" would have been sample 16 000 rather than
  the read's last sample even with the window fixed. A read-end head now
  overrides that bound and decodes the whole read.

  Bundles that omit anchor are unchanged: absent means adapter_end, which is
  what every bundle predating the key was built with.

  Three refusals come with it, because each of these ran and was silently wrong
  rather than erroring:

  • an unknown key under signal, or an unknown anchor value. That block is
    a set of rules the model was built with, so one this runtime does not
    implement is refused rather than ignored — the doctrine already applied to
    the charging bundle. The top level stays open for provenance.
  • a signal.window string that contradicts signal.anchor. The prose and
    the machine-readable key describe one rule; when they disagree escpod
    refuses instead of picking a side. This is fdx4@v0.1.0 exactly.
  • a read-end bundle that also pins a boundary detector under boundary.
    escpod honours that block at runtime (it refuses --method llr against a
    cnn pin), so believing it would window the wrong end. Provenance belongs
    under built_beside, which is ignored.

  --method, --cnn-model, --boundary-margin and --clamp-max-shift are
  likewise refused against a read-end bundle rather than accepted and ignored;
  a read-end run builds no detector at all.

  demux basecall had the same truncation defect independently — it decodes an
  adapter_end-sized prefix per read — and now decodes the whole read for a
  read-end bundle, using the boundaries CSV only to select which reads to
  basecall. demux --info no longer describes an adapter_end window, a
  boundary margin or a window clamp for a model that has none, and no longer
  prints a suggested command line containing the --method it would refuse.

---

Which artifact

Artifact	Linkage	--gpu
…-x86_64-unknown-linux-musl.tar.gz	static (musl)	no
…-aarch64-unknown-linux-musl.tar.gz	static (musl)	no
…-x86_64-unknown-linux-gnu-gpu.tar.gz	dynamic, glibc ≥ 2.28	yes
…-x86_64-apple-darwin.tar.gz	dynamic	no
…-aarch64-apple-darwin.tar.gz	dynamic	no

The musl builds are the portable default and the right thing for
an unattended installer to fetch. The GPU paths cannot be static —
they dlopen their runtimes — so they ship in the single dynamically
linked …-linux-gnu-gpu artifact instead, built against glibc 2.28
(RHEL/Rocky/Alma 8+, Ubuntu 20.04+).

What …-linux-gnu-gpu needs at run time

Only when --gpu is actually requested — otherwise it behaves
exactly like the musl one. It expects a CUDA 12 runtime and
cuDNN 9, with an NVIDIA driver ≥ 535 (the DTW kernels target the
CUDA 12.2 driver API), and a CUDA-enabled libonnxruntime matching
the ort 2.0.0-rc.13 it links.

Rather than assembling that by hand, see
GPU acceleration,
which covers the pixi environment that supplies it and how to
confirm the CUDA execution provider actually loaded.