Releases: rnabioco/leech
Release list
Release v0.13.1
Fixed
data prepare --signal-context-baseswith--seq-encoding signal_kmer(the default) could scatter a chunk's k-mer sequence identity at the wrong signal positions, andleech predict's Rust path inherited the same bug. Whenever a read's requested base-defined window was wider than--signal-len,seq_to_sig_mapwas built from the pre-crop request rather than the windowplace_windowactually placed the signal in — a shift of up to hundreds of samples, invisible to the Python/Rust backend-parity suite because both prepare backends shared the identical bug. Fixed in the Python extractor and the Rustdata preparepath directly; predict's Rust path relies onescapepod_signal::chunk::cut_chunk's internalSignalKmerarm, which carried the same mismatch in the pinned upstream crate (rnabioco/escapepod-rs#388) — fixed there inescapepod-signalv0.27.1, which this pins to, so predict inherits the fix with no code change of its own. ((#343))leech data merge's base-to-signal-map gather is now block-streamed like the rest of the merge, instead of loading one whole input file's CSR values array at a time._load_s2s_csr's per-filenp.loadbounded the merge's transient memory to the largest single input file'sseq_to_sig_values, which scales with--signal-context-bases's window width; the newiter_npz_csr_value_blocks(leech.chunking.serialization) reads it in row-aligned byte blocks the same way the fixed-width members already are. A benchmark confirmed the merge itself scales linearly with context width at fixed corpus size, so this is a preventive fix rather than a response to a known incident. Also adds the first test coverage for the genuinely ragged (signals/dwells/features) object-array merge fallback, which no fixture had ever exercised. ((#344))encode_signal_kmer's pure-Python fallback (used by any install without theleech[rust]extra) no longer crashes withIndexErroron a chunk whose k-mer context runs off the edge of a read. The pinnedescapepod-signalcrate treats asequence_intsindex past the end of the array as "nothing to encode at this k-mer position," matching an explicit upstream test (missing_context_is_skipped_not_padded); the Python fallback indexed unconditionally instead and raised. Invisible until now because every CI job and dev environment has the Rust extension built, so the fallback was never exercised — new tests forceHAS_RUST=Falseto close that gap for good. ((#347))
What's Changed
Other Changes
- fix(prepare): key seq_to_sig_map off the placed window, not the pre-crop request by @jayhesselberth in #346
- fix(splitting): block-stream the CSR base-to-signal map gather in merge by @jayhesselberth in #345
- fix(features): encode_signal_kmer fallback skips missing context instead of raising by @jayhesselberth in #348
- fix(predict): bump escapepod-signal to v0.27.1, closing #343's predict-path gap by @jayhesselberth in #349
Full Changelog: v0.13.0...v0.13.1
Release v0.13.0
Added
leech predictnow runs a model trained with--signal-context-basesthrough the fast Rust extraction path instead of always falling back to Python.rust/src/inference_pipeline/inference.rsmirrors the per-chunkChunkSpecresolutiondata prepare's Rust path already used (issue #278): for a base-defined window it resolves the sample interval via a base-to-signal map lookup shared withtraining.rsand cuts with a spec cloned just for that chunk. Predict's Rust and Python paths now produce byte-identical signal windows for such a model, held equal by a new backend-parity test. ((#342))
What's Changed
Other Changes
- feat(predict): implement signal-context-bases on Rust inference path by @jayhesselberth in #342
Full Changelog: v0.12.3...v0.13.0
Release v0.12.3
Fixed
--pod5can now point at a directory of POD5 files (a MinKNOW run) on the Rust extraction path, not just a single file.rust/src/pod5_cache.rsnow resolves reads throughescapepod_signal::cached_dataset(escapepod-rs#384) instead ofcached_reader, with no file/directory branch — a single-file input costs nothing extra, since every file a dataset touches is opened through the same per-file reader cache either way.--backend auto/--backend rustboth work on directory input now;--backend pythonalready did. ((#339))
What's Changed
Other Changes
- fix(pod5): accept a directory of POD5 files on the Rust extraction path by @jayhesselberth in #340
Full Changelog: v0.12.2...v0.12.3
Release v0.12.2
Fixed
--resumecan now recover a run an external kill (SLURM walltime, OOM,scancel) cut off mid-loop, not only a clean exit.model_last.ptwas written exactly once, after the epoch loop exited, so an interrupted run left--resumenothing to find and every retry started over from a fresh random seed -- the whole reason a Snakemakerestart-timesretry existed was defeated by the checkpoint design it depended on. A new rolling checkpoint,model_resume.pt, is written atomically at the end of every epoch and carries the full resumable state (history, early-stopping patience, the adversarial head, theClipGradbuffer, per-rank RNG state, the run seed) plus a recipe and corpus fingerprint that--resumenow refuses to cross rather than silently warm-starting a different run. It is deleted on a clean finish and deliberately left undeclared in the Snakemake rules, since a declared output would be deleted by Snakemake itself the moment a walltime kill "fails" the job -- exactly when the retry needs it most.train.smkandcompare_models.smknow pass--resume {output_dir}/model_resume.ptunconditionally. ((#330))
What's Changed
Other Changes
- fix(train): interrupt-safe --resume via a rolling model_resume.pt by @jayhesselberth in #334
Full Changelog: v0.12.1...v0.12.2
Release v0.12.1
Fixed
data prepareagainst a POD5 pre-filtered to a subset of the BAM's reads
no longer aborts the whole run. A read the POD5 does not carry was counted
as a per-read failure on both backends, so the expected BAM/POD5 mismatch
produced byescpod bam-filter(and the pipeline's own Filter stage) crossed
MAX_FAILED_READ_FRACTIONand raised, discarding a corpus whose every present
read had extracted correctly. Absent reads are now their own expected-exclusion
bucket, reported asreads_missing_from_pod5in the stats and in the "Read
yield" log line, and the failed fraction is measured over the reads actually
attempted. Issue #265's zero-tolerance policy is unchanged for genuine
failures:leech_core.extract_training_chunksnow returns
(chunks, n_missing_from_pod5)so the Rust dispatcher's "reads went in, no
chunks came out" check can still fire on reads that were found in the POD5. ((#325))leech model train --model-config best_params.jsonno longer crashes onleech model optimize's own output.best_params.jsonalways recordsselection_metricfor provenance, buttrain's model-config loader passed it straight through to the model constructor, raisingTypeError: model got unexpected keyword argument(s): selection_metricon the very first run of the documented optimize-then-train workflow.selection_metricis now dropped before model construction, the same waycheckpoint_metricalready was. ((#326))leech data mergeno longer silently collapses a same-body contrast to one class. Pairwise relabeling matched chunks against each input file's originallabelsvalues rather than which-iargument supplied the file, so two files sharing an internal label (e.g. two preparations of the same tRNA body, contrasted by attached ligand rather than body identity) resolved every chunk to the same class — with the log still printing a correct-looking0/1assignment._parse_and_validate_inputsandmerge_and_split_chunks/merge_and_kfold_split_chunksnow track group membership by file provenance (relabel_by_file) instead of re-deriving it from label values; the remaining value-based path (process_comparison_spec's TSV workflow) now raises if the two groups' label sets overlap instead of collapsing silently. ((#327))
What's Changed
Other Changes
- fix(train): drop selection_metric from --model-config before model construction by @jayhesselberth in #326
- fix(merge): track chunk provenance instead of re-deriving group membership from labels by @jayhesselberth in #327
- fix(prepare): don't count reads absent from a pre-filtered POD5 as failures by @jayhesselberth in #328
- build: move both escapepod pins to v0.26.0 by @github-actions[bot] in #320
- chore: release v0.12.1 by @jayhesselberth in #329
New Contributors
- @github-actions[bot] made their first contribution in #320
Full Changelog: v0.12.0...v0.12.1
Release v0.12.0
Added
-
LeechDatasetwarns when a signal crop reaches outside the stored chunk.
left_context/right_contextcrop[focus - left_context, focus + right_context); when a corpus was prepared with a narrower
signal_contextthan that window, the shortfall was zero-padded with
nothing logged. In2026-aa-trna-models, production corpora stored
signal_context [225, 225]and trained withright_context 300, so the
last 75 samples of every chunk were zero for two production retrains before
anyone noticed.LeechDatasetnow logs one warning per dataset the first
time this happens, naming the requested window, the stored window, and the
number of padded samples;strict_window(leech model train --strict-window) raises instead. The same silent zero-pad also existed in
the plainer symmetric case (a--signal-lenwider than the stored chunk with
noleft_context/right_contextset at all), which now warns/raises through
the same path. Detection lives in one shared_warn_or_raise_paddingused by
the asymmetric (_note_crop_padding) and symmetric (_note_plain_pad)
cases, called from both the row (_prepare_signal) and block
(_prepare_signals_block) fill paths, so pre-loaded and streamed corpora are
both covered.strict_windowis also recorded in the savedconfig.jsonso
a checkpoint's guard setting can be audited later. ((#255)) -
leech data prepare --mask-seq-left-of-focus/--mask-seq-right-of-focus
blanks sequence-branch bases on one side of the focus base. With
seq_encoding: signal_kmer(the default),sequence_with_kmer_context
begins with acceptor-stem bases 5' of a 3'-end motif and identifies a tRNA's
body outright, so every model trained so far had tRNA identity leaked to it
through the sequence branch regardless of what the signal/feature branches
learned. The new flags writeNover sequence-branch characters strictly to
one side of the focus base — in bothsequence(base_onehot) and
sequence_with_kmer_context(signal_kmer) — reusing the existing
"non-ACGT maps to -1 and is skipped" convention (sequence_to_int,
encode_signal_kmer) rather than a new mechanism; the focus base's own
character is never masked. Masking is baked into the corpus atdata prepare
time (ChunkConfig.mask_seq_side, applied once inLeechRead.get_chunkfrom
the exact local geometry, not reverse-engineered from stored arrays later),
recorded inprepare_config.json, carried into the trained model's
config.jsonbymodel train, and auto-applied bypredictto live chunks
so inference sees the same masked geometry the model was trained on. Not
implemented in the Rust extraction path —data preparefalls back to the
Python workers andpredict --backend rustraises;--backend autofalls
back with a warning, matching the existingrecover_softclip_signalgating.
Noleech model trainflag: the mask is a property of the corpus, not a
training-time choice. Fixes #256. ((#256)) -
leech data prepare --signal-context-bases L,R: a base-defined signal
window. Mutually exclusive with--signal-context. Instead of a fixed
number of samples on each side of the focus base, the window is cut at the
base-to-signal map positions of offsets-Land+R(inclusive), so a fast
and a slow read cover the same bases of context rather than the same
samples — a sample-defined window can't reach a base at+24on a slow
read (~36 samples/base) without also reaching+25on a fast one
(~24 samples/base), which for the charging assay is where the LDX barcode
starts.The resolved sample interval is padded (right-aligned zero-fill, the
conservative default) or centre-cropped to a fixed--signal-len, which
defaults to(L + R + 1) * 36samples/base (sized off a conservative
slow-read rate so a typical read pads rather than crops) and can be
overridden explicitly. A focus base near either edge of a read gets a
narrower window rather than a dropped chunk, matching the one existing drop
rule (a focus base with no signal boundaries at all).Implemented in both
data preparebackends (PythonLeechRead.get_chunkand
the Rust training pipeline), held to identical output by
tests/test_backend_parity.py. The resolved window is logged, recorded in
prepare_config.json, and carried into the trained model'sconfig.jsonso
predictre-derives the same window and the ONNX contract sidecar states it
(the Rust predict pipeline does not yet implement it and falls back to the
Python predict path, as it already does for a few other options only the
training-side Rust pipeline supports). ((#278)) -
--loss noise_corrected_bce: a label-noise-aware loss for enrichments with a known, per-group impurity rate.losses.NoiseCorrectedBCEWithLogitsLossapplies a forward correction (Patrini et al. 2017) using a per-sample label-flip probability looked up from each chunk'ssource_groupvia--label-noise-rate group=rate[,group=rate,...](e.g.--label-noise-rate gold=0.09,enzymatic=0.17); groups not named get rate 0 and the loss reduces to plain BCE exactly. Rates are measured upstream (leech does not estimate them) and are recorded verbatim inconfig.json;predictis unaffected. Reduces by element count like the other losses, so it decomposes correctly under--gpus N. ((#279)) -
Checkpoint selection and loss shaping for the low-FPR operating regime.
--checkpoint-metric(and grid search's--selection-metric) now also
accepttpr_at_fpr:<f>(TPR at a fixed FPR) andcallable_at_precision:<p>
(fraction of validation reads callable at a precision floor) alongside the
existingauto/val_acc/val_f1/val_auc-- binary tasks only, computed
once on the gathered predictions under--gpus Nlike the others, and
reported back in a run'ssummary.json/eval testoutput when selected, so
two runs using the same threshold can be compared after the fact.
leech.metrics.tpr_at_fpr/callable_at_precisionimplement the two metrics
directly from(labels, probs).FocalBCEWithLogitsLossgains an optional
--focal-neg-gamma, making the focal loss asymmetric: a higher gamma on
negatives than positives down-weights easy negatives harder, concentrating
gradient on the hard negatives that set FPR at a given threshold. Leaving it
unset (or set equal to--focal-gamma) reproduces the current loss
bit-for-bit. ((#280)) -
Time-stretch augmentation for speed invariance.
--augment-time-stretch MIN,MAX(leech model train/optimize) resamples each training chunk's signal window by a per-sample factor drawn uniformly from[MIN, MAX], about the focus position, cropped/padded back tosignal_len. The CSR base-to-signal map is scaled by the same factor sosignal_kmerstays aligned, and thedwell/dwell_meanfeature channels scale with it (dwell_logshifts bylog(factor);dwell_ratio,dwell_stdand level channels are unchanged). Implemented on the batchedLeechDataset.__getitems__fetch path (one(B,)factor draw per batch, numpy gather rather than a torchindex_select); off by default, recorded inconfig.json, and never applied to validation. Targets the ~7-point sensitivity gap the charging model shows on translocation speeds it never saw during training. ((#281)) -
A per-chunk
junction_indel/junction_mappedfield, plus a predict-time
abstention rule that uses it.data preparenow records, for every chunk
whose motif was found through aReferenceMotifSearcher, the CIGAR-measured
disruption at the motif's mapped span (junction_indel = mapped_len - len(motif), 0 when exact) and whether that measurement was possible at all
(junction_mapped) — the value the motif searcher already computed to decide
whether to keep or reject a motif, previously discarded once that decision was
made. Both prepare backends emit it identically, since the measurement is made
once in Python and both backends consume the same motif search result.
--sample-weight-field FIELDgeneralizes--balance-groups's inverse-
frequency sampling to any chunk metadata field (--balance-groupsis now a
named instance of the same mechanism), so--sample-weight-field junction_indelover-samples chunks whose motif junction is disrupted.
leech predict --abstain-on-junction-indelenforces--min-marginonly on
calls whose junction is disrupted, writingunc/BELOW_THRESHOLD_LABELfor
those below threshold while an intact junction bypasses the margin check
entirely;junction_indel/junction_mappedare always recorded (BAMji/jm
tags, orjunction_indel/junction_mappedTSV columns) so the rule can be
re-applied offline.eval testreports accuracy/F1 (and, for binary models,
the full metric set) stratified byjunction_indel == 0vs!= 0whenever the
test corpus carries the field. The field is not fed to any model as an input
channel — it is a sampling/abstention signal only. ((#282)) -
Symmetric (non-causal) TCN padding and recorded per-channel feature standardisation, both opt-in.
TemporalBlock/TCNgain acausal: bool = trueparam (components.py);causal = falsesplits the same total padding evenly across both sides instead of putting it all on the left, doubling the receptive field on either side of a position for the fixed-window classifiers this project trains (parameter shapes are unchanged, so a checkpoint loads either way, but the trained values aren't a meaningful initialization for the other mode). It reaches every TCN-family TOML config (tcn_dwell*.toml) as a[params]entry, settable via--model-configor a future[[variants]]pin, same asnorm_type.--standardize-featurescomputes per-channel feature mean/std once over the training corpus (LeechDataset.__init__, reusing the tensor thefeature_noiseaugmentation already stacks) and f...
Release v0.11.1
Changed
-
escapepod pin bumped v0.21.0 -> v0.24.3 (
rust/Cargo.tomltag and the
escapepodPyPI floor, moved together as always). Picks up upstream's DP
speedups todp_step_with_dwell_penalty— the exact function leech's own
refinement preset (RefineSettings::move_table_refinement, which resolves
toRefineAlgo::DwellPenalty) drives on every base of every chunk during
data prepareandpredict. Upstream measured roughly 8-11x on the DP
loop itself (criterion) and +25.5% / +8.7% end-to-end wall clock across the
contributing releases (v0.24.1, v0.24.3).Not bit-identical to v0.21.0 output. Two of the upstream changes
(v0.24.1's cumsum-prefix-sum reordering, v0.24.3's halvedmax_check) are
documented, deliberate correctness tradeoffs forRefineAlgo::DwellPenalty
— escapepod's own real-data A/B saw up to 1.76% of reads shiftp_charged
by some amount and 0.025% flip a discrete call. Adata preparerun
against this release will not bit-match chunks prepared under leech
<=0.11.0. leech's own Python/Rust backend parity is unaffected, since both
backends call the same upstream function with the same settings. (#250,
#251)
What's Changed
Other Changes
- build(deps): bump escapepod pin to v0.24.3 by @jayhesselberth in #251
- chore: release v0.11.1 by @jayhesselberth in #252
Full Changelog: v0.11.0...v0.11.1
Release v0.11.0
Added
-
Single-node multi-GPU training:
leech model train --gpus N. N
data-parallel ranks on one node, opt-in, train only —eval testmeasured
input-bound (53-79% GPU with every DataLoader worker pegged), so DDP buys it
nothing and would only add risk to the path that produces the scores.--batch-sizestays the GLOBAL batch and is split across ranks. The
PyTorch convention is the opposite — per-rank, so the effective batch grows
with the GPU count — and it is wrong here, because a leech run is one arm of
a paired comparison: the same command line has to mean the same recipe at any
--gpus, or every number already measured needs re-measuring. Splitting
leaves the optimizer-step count, the LR schedule,ClipGrad's quantile
buffer and thegrad_accum_splitarithmetic exactly where they are.Measured on the production charging corpus (6.72M chunks, 36.5 GiB npz,
2 epochs, one 4xA30 node):--gpusjob wall speedup peak RSS 1 24:31 1.00x 40.6 GiB 2 16:40 1.47x 88.3 GiB 4 10:53 2.25x 157.6 GiB 2.25x rather than 4x because ~3 min of per-rank corpus load and the final
eval do not shard. Memory is ~1:1 with the corpus per rank, so--gpusand
the job'smem_mbhave to move together or the second rank is OOM-killed
during its load;pipeline/workflow/rules/train.smkscales both off one
train_gpuskey.Four things that would each have been silent:
WeightedRandomSamplerhanded
to N ranks gives every rank the same oversampled draw, so
DistributedWeightedSamplershards one global multinomial and the union of
the shards is exactly the single-GPU epoch; validation shards without padding
and gathers before it scores, so the metrics are equal to the single-GPU
numbers rather than close to them; the checkpoint keeps the single-GPU
state_dictkeys, since amodule.prefix either fails to load downstream or
loads nothing understrict=Falseand exports an untrained graph; and the
auxiliary heads are wrapped individually, because they sit in the optimizer
but outside the model and a DDP around the model alone never allreduces them. -
Daily check that the two escapepod pins agree
(.github/workflows/escapepod-sync.yml).rust/Cargo.tomltags the
escapepod-signalcrate andpyproject.tomlfloors theescapepodPyPI
package; they are one upstream released in lockstep, and leech drives both.
Dependabot bumps the tag-pinned crate on its own schedule (#236) and cannot
know the Python package has to follow, so merging one of its PRs leaves main
skewed with nothing red. The check compares them daily and opens the matching
bump rather than failing dependabot's PR, which would leave a human to do the
second half by hand. -
Dependabot PRs merge themselves once CI has passed
(.github/workflows/dependabot-auto-merge.yml). Not GitHub's native
auto-merge, which needsallow_auto_mergeand required status checks on
mainto mean anything — without required checks it merges immediately
rather than waiting for CI. Triggering onworkflow_runmakes "CI finished
and it was green" the entry condition instead.escapepod-signaland any
major bump stay with a person: the first because its PyPI twin has to move in
the same commit, the second because a green suite says the tests still run,
not that the semantics held.
Fixed
-
Weighted
CrossEntropyLossis normalized by the global summed weight under
DDP. Every other loss in the trainer reduces by element count, which equal
shards make exact;CrossEntropyLoss(weight=...)divides by the summed weight
of the samples it sees, so each rank normalized by its own shard's class
composition and the averaged gradient was the single-GPU one only when the
shards happened to draw alike. 8% off on a fixture whose shards differ, with
nothing raised. -
DataLoader workers fork inside spawned ranks.
multiprocessing.spawn.prepareforces a spawned child's default start method
to match how it was created, so every DataLoader a rank built pickled the
dataset's stacked tensors through/dev/shminstead of COW-sharing them —
voiding the invariant those contiguous buffers exist for. It surfaces nowhere
near the cause and does not look like memory: shm pages are charged to the
cgroup but not to RSS, so the run died at 176 GiB RSS against a 244 GiB
allocation, in the fourth rank's validation loader, asNo space left on device. Fixing it took the same run to 157.6 GiB. -
Spawned ranks configure their logging.
setup_loggingruns in the click
entry point, which a rank never reaches, so theleechlogger had no handler
and every INFO line was dropped — including rank 0's, where the effective
batch, the sampler statistics and the encoding-fallback warning are reported.
The run worked and said nothing about itself.
Changed
- escapepod moved to v0.21.0 on both backends — the
escapepod-signal
crate and theescapepodPyPI package together, since a skew between them
lets the two prepare backends compute different dwells and different
level-derived features from the same read (the divergence behind #193).
Validated withtests/test_backend_parity.pyagainst aleech_coreactually
built on v0.21.0; a compile only says the API still exists.
Internal
tests/test_workflows.pyfails when a GitHub Actions workflow does not
parse. Nothing else in the repo reads those files, and a scheduled workflow
that never fires is indistinguishable from one with nothing to report.
What's Changed
Other Changes
- test: stop setup_logging's propagate=False from blinding caplog by @jayhesselberth in #234
- feat(training): single-node data parallelism via
leech model train --gpus Nby @jayhesselberth in #242 - ci: merge dependabot PRs once CI has passed on them by @jayhesselberth in #249
- chore: release v0.11.0 by @jayhesselberth in #248
Full Changelog: v0.10.0...v0.11.0
Release v0.10.0
Fixed
-
Exported ONNX graphs load in a runtime that is not PyTorch.
charging_tcn_rna004@v0.1.0shipped from this exporter with a graph no
releasedescpodbinary could load — tract parses it and then gives up
during shape analysis (rnabioco/escapepod-models#96). onnxruntime loads it
fine, which is whyverify_onnxhad nothing to say and the failure surfaced
at integration rather than at build time: "it exports and round-trips" is a
weaker claim than "a runtime can load it", and only the second one ships.Two independent causes, both measured against tract 0.23.5 through the load
pathescapepod_classifyactually uses:adaptive_avg_pool1dwith an output size that does not divide the
input (390 -> 11 here). The dynamo exporter open-codes it as
Unsqueeze -> Transpose -> GatherND -> Transpose -> Where: a rank-8 gather
over an all-constant index and mask, which tract refuses pinned, unpinned,
and withvalue_infocleared. No post-hoc rewrite helps.
models.components.AdaptiveAvgPool1dnow writes the same arithmetic as one
matmul against a constant[L_in, L_out]segment-mean matrix, using
PyTorch's own bin rule (upsampling included —ResNetDwellpools 4 up to
11), in float32 outside autocast so the accumulation matches the aten op
under AMP. Agreement with the aten op is 2.4e-07 over a grid of lengths and
output sizes. One implementation, so the registry layer,resnet_dwell,
transformer_dwelland thetests/reference_*oracles move together and
the config-vs-reference parity tests stay bit-exact.SignalCNN's
AdaptiveAvgPool1d(1)is untouched: 1 divides everything and exports as
GlobalAveragePool.value_info. Dynamo writes one entry per intermediate — 667 for this
model — carrying the batch axis as the symbolbatch, because that is
whatdynamic_axesasked for. A consumer that pins the batch then cannot
unify, and tract fails at the first convolution.strip_value_infodrops
them, andexport_onnxalways calls it. Nothing needs them: every runtime
re-infers,onnx.checkeris satisfied, and every graph escpod loads today
has zero. Initializers are untouched, external data references included.
Measured on the shipped
TCNDwellResidualLNweights, no retrain: 479 -> 319
nodes,GatherND2 -> 0,Gather76 -> 0; tract loads, optimizes and runs at
batch 1 and 32 (was: five distinct failures), max |dlogit| 5.72e-06 against
torch over 256 real chunks with 0 decision disagreements; onnxruntime vs torch
1.335e-05 over 4096 real chunks (shipped graph: 1.4305e-05).
Changed
- Numerical output moves by ~2.4e-07 for the architectures that use a
non-dividing adaptive pool (ResNetDwell,TransformerDwell, the TCN family
and any config using theAdaptiveAvgPool1dregistry layer). Same weights,
same decisions — existing checkpoints load and predict as before, but exact
float equality with a v0.9.0 run does not hold.
Documentation
- The exporter's case for the dynamo path is re-measured rather than inherited,
since the pool no longer emits an aten adaptive pool and that could have
retired the reason. It did not:dynamo=Falsestill refuses both the aten
pool and leech's replacement, becausetorch.jit.traceturns.shape[-1]
into a Tensor and takes the dynamic-length fallback. A test pins it.
What's Changed
Other Changes
- fix(export): make the ONNX loadable by a non-Python runtime by @jayhesselberth in #233
Full Changelog: v0.9.0...v0.10.0
Release v0.9.0
Added
-
leech.crf.evaluate: decode a corpus, match it to references, report per
group. The generic half of CRF evaluation — what a panel is (which classes
exist, which share a flowcell) stays with whatever defines the panel; what
arrives here is a reference set, a grouping and a corpus.Three rules it holds rather than leaving to callers, because each produces a
plausible-looking wrong number:- Match what the model emits.
emitted_referencesapplies
target[state_len:]once. Scoring against full-length targets forces
state_lenleading deletions into every alignment, which inflates every
distance and compresses the margin — an aligner puts those deletions where
they help most, discounting wrong references more than the right one. - Report per group. When classes are crossed with batch, one pooled table
measures batch.balanced_recalltakes the grouping as an argument (only
the caller knows whether their classes are confounded) and raises when
no group has reads, because a null headline serializes fine and ships. - Balanced, not raw. A pooled accuracy over unbalanced classes is
dominated by the deepest class.
lev_vs_refsscores one decode against the whole reference set at once, which
is the shape of every evaluation loop; scoring R references one at a time is R
DP tables per read. Its vectorisation recovers the serial insertion term
exactly (j + cummin(tmp[k] - k)), and that identity is asserted against the
scalar implementation on random strings rather than assumed. edlib is used
where importable, with the pure-Python fallback kept under its own name so a
test compares the two rather than comparing edlib with itself.Validated end to end on the production ldx corpus: 16 references at emitted
length 44 from 48, 4000 held-out reads decoded, and per-flowcell reporting
that correctly finds 8 classes in each — the pilot's code-flowcell confound,
which is exactly why pooling would be wrong. - Match what the model emits.
-
ONNX export, for the classifier arms and the CRF encoder (#217).
leech model export --format onnxbeside the existing--format torch
(unchanged default), andleech.crf.export.export_crf_onnx.torch.export
makes a model loadable by anything with PyTorch and by nothing else; a runtime
consuming ONNX — which is what escapepod-rs runs — could not load a leech
model at all.Both use the dynamo exporter at opset 18.
dynamo=False, the obvious
first attempt, fails on these architectures with anadaptive_avg_pool1d
error that reads like a model problem and is an exporter limitation; that is
documented where someone will hit it, and a regression test pins it.Each export writes a contract beside the graph, carrying the two things a
consumer needs and cannot recover from it: which input is which (including
that thesignal_kmersequence input is built in the dataset, not the model,
and thatleech-coreships that encoder), and what the output means — a
single BCE logit, not a two-class softmax. The CRF's contract additionally
carries standardisation, which is in neither the config nor the checkpoint,
and its emitted references (target[state_len:]), computed from the
state_lenthe encoder declares.Verified across the serialization boundary rather than in process:
onnxruntime against torch, 3.58e-07 for the CRF encoder against a float32 eps
of 1.19e-07.New
onnxextra (onnx,onnxruntime,onnxscript). CI installs it so the
round-trip tests run rather than skip. -
leech model train-crf, the CLI for the CTC-CRF trainer. Sits beside
model trainrather than in a group of its own: it is the same workflow step,
a different task. EveryCrfTrainConfigfield is exposed, and a test asserts
each option actually reaches the config — a click option that silently does
not is how a sweep ends up running the default every time.The summary reports the emission rule (
target_len -> emits target_len - state_len) because widening the window to get a longer decode is the mistake
it prevents, and prints a second table only when the run discarded steps or
saw non-finite gradients, since a discarded step is otherwise invisible. -
leech.crf.training: a CTC-CRF trainer.CrfTrainerruns the schedule
and writesmodel.ptplus amodel.jsonsidecar. The sidecar is not optional:
the standardisation constants live in neither the architecture config nor the
checkpoint, so weights alone cannot be used correctly.Separate from
leech.training.Trainer, which is classification-locked through
pos_weight,num_out, BCE/focal/CE and AUROC/F1 checkpointing — a sequence
task shares none of it, and forcing it through would put the production
classifier path at risk. The decisions are split out as plain functions
(compute_standardisation,apply_quality_gate,resolve_split,
encode_targets,select_checkpoint) so they are testable without a GPU; the
loop is mechanical, the decisions are where runs go wrong quietly.Verified against production data, not just fixtures: standardisation over the
391,174 x 3000 ldx corpus reproduces the shipped model's recorded constants
(61.8216743766 / 9.5716818880) with a delta of 0.000e+00 on both, and the
train/test counts match whatplan_corpusderives independently from the
manifest (283,296 / 107,878). A two-epoch GPU run trains at ~40 s/epoch with
loss falling 0.4446 -> 0.0639. -
leech.crf.corpus: cut a CRF training corpus from a manifest.
plan_corpusdecides which reads and in which split, touching no POD5;
build_corpusextracts their signal, streaming it to a memory-mappable
<out>_X.npybeside a<out>_meta.npz. The signal is never held in RAM as a
whole — an 80-plex corpus is tens of gigabytes, and the memmap is what makes
the size a disk question instead of an allocation that fails.The two stages are separate because everything subtle is in the plan, and a
corpus planned wrongly still trains and still reports a number. Four rules,
each pinned by a test: a cap only caps if every class can reach it
(per_group="auto"is the rarest class's trainable depth, with the test
fraction reserved first); the split is carved before capping and ranked per
class globally across batches, since per-(batch, class)ranking multiplies
the cap by the batch count whenever classes are crossed with batch; batches
are interleaved rather than concatenated, or the whole held-out set comes from
whichever batch sorts first and the headline number measures batch; and
sharding happens after planning, so every shard keeps its share of one global
split. Extracting nothing, or less than half the plan, is a hard error with a
different message for each — the causes differ, and a 0-row corpus otherwise
exits cleanly and reaches a GPU job.Validated against the production ldx manifest: 1,139,602 rows plan to 391,174
reads atchunk=3000, the same count escapepod-models' extractor reports for
that input, with the training pool balanced exactly across all 16 groups and
held-out reads drawn from both flowcells.load_corpus/load_corpus_metaread both this layout and the legacy
single-.npzone, so corpora written before the split layout keep loading.
Changed
-
The signal-level k-mer encoding comes from escapepod-signal rather than
being held here (escapepod-rs#271 / #272; requires escapepod 0.16.0).
rust/src/encoding.rsandsequence_to_intare now calls into
escapepod_signal::seq_encoding.leech held the only copy of this rule, inside a
crate-type = ["cdylib"]
Python extension module — so a native runtime for a leechsignal_kmermodel
could not link it and had to transcribe it, which is a second definition that
diverges silently. It is also the natural pair toescapepod_signal::mapping,
which produces the base-to-signal map the encoding consumes: the producing
half was already upstream and the consuming half was not.This is a delegation, so the only acceptable outcome is identity: 198 parity
tests pass unchanged, includingtest_backend_parity.py, which compares every
array in the npz between the Rust and Python backends against a Python
reference this change does not touch.The k-mer context slice delegates too, via
sequence_bases_with_context
(escapepod-rs#274, escapepod 0.16.1). It could not at first: leech needs the
window as bases, since the corpus serializessequence_with_kmer_context
as a string, where upstream only offered ints. Upstream now exposes both forms
over one windowing rule, withsequence_to_int(bases) == intspinned by a
test there — so the three halves of the signal-level k-mer path (the map, the
window, the encoding) all live inescapepod-signaland none is duplicated
here.That third one is the highest-stakes of the three: it is where
beforeand
afterare not interchangeable, and swapping them displaces every k-mer
silently because the encoder only sees the total width. It is also the most
directly checkable —sequence_with_kmer_contextis one of the fields
test_backend_parity.pycompares array-by-array between backends.Only
rust/Cargo.toml's git tag moves to v0.16.1; theescapepodPython pin
stays>=0.16.0, because the new function is in the Rust crate and not in the
Python bindings.
Fixed
-
signal_kmerdegraded tobase_onehoton the strength of one chunk, and
the checkpoint did not record that it had (#230). Three separable defects
that combined into a run which finishes, looks fine, and trained on a
different model input than was asked for.The
--seq-encodingdefault issignal_kmer, so a corpus written by one
leech version and read by another that finds no base-to-signal maps warns and
carries on. That reached production through a projec...