Skip to content

Releases: rnabioco/leech

Release v0.13.1

Choose a tag to compare

@github-actions github-actions released this 17 Sep 14:24

Fixed

  • data prepare --signal-context-bases with --seq-encoding signal_kmer (the default) could scatter a chunk's k-mer sequence identity at the wrong signal positions, and leech predict's Rust path inherited the same bug. Whenever a read's requested base-defined window was wider than --signal-len, seq_to_sig_map was built from the pre-crop request rather than the window place_window actually placed the signal in — a shift of up to hundreds of samples, invisible to the Python/Rust backend-parity suite because both prepare backends shared the identical bug. Fixed in the Python extractor and the Rust data prepare path directly; predict's Rust path relies on escapepod_signal::chunk::cut_chunk's internal SignalKmer arm, which carried the same mismatch in the pinned upstream crate (rnabioco/escapepod-rs#388) — fixed there in escapepod-signal v0.27.1, which this pins to, so predict inherits the fix with no code change of its own. ((#343))
  • leech data merge's base-to-signal-map gather is now block-streamed like the rest of the merge, instead of loading one whole input file's CSR values array at a time. _load_s2s_csr's per-file np.load bounded the merge's transient memory to the largest single input file's seq_to_sig_values, which scales with --signal-context-bases's window width; the new iter_npz_csr_value_blocks (leech.chunking.serialization) reads it in row-aligned byte blocks the same way the fixed-width members already are. A benchmark confirmed the merge itself scales linearly with context width at fixed corpus size, so this is a preventive fix rather than a response to a known incident. Also adds the first test coverage for the genuinely ragged (signals/dwells/features) object-array merge fallback, which no fixture had ever exercised. ((#344))
  • encode_signal_kmer's pure-Python fallback (used by any install without the leech[rust] extra) no longer crashes with IndexError on a chunk whose k-mer context runs off the edge of a read. The pinned escapepod-signal crate treats a sequence_ints index past the end of the array as "nothing to encode at this k-mer position," matching an explicit upstream test (missing_context_is_skipped_not_padded); the Python fallback indexed unconditionally instead and raised. Invisible until now because every CI job and dev environment has the Rust extension built, so the fallback was never exercised — new tests force HAS_RUST=False to close that gap for good. ((#347))

What's Changed

Other Changes

  • fix(prepare): key seq_to_sig_map off the placed window, not the pre-crop request by @jayhesselberth in #346
  • fix(splitting): block-stream the CSR base-to-signal map gather in merge by @jayhesselberth in #345
  • fix(features): encode_signal_kmer fallback skips missing context instead of raising by @jayhesselberth in #348
  • fix(predict): bump escapepod-signal to v0.27.1, closing #343's predict-path gap by @jayhesselberth in #349

Full Changelog: v0.13.0...v0.13.1

Release v0.13.0

Choose a tag to compare

@github-actions github-actions released this 17 Sep 02:30

Added

  • leech predict now runs a model trained with --signal-context-bases through the fast Rust extraction path instead of always falling back to Python. rust/src/inference_pipeline/inference.rs mirrors the per-chunk ChunkSpec resolution data prepare's Rust path already used (issue #278): for a base-defined window it resolves the sample interval via a base-to-signal map lookup shared with training.rs and cuts with a spec cloned just for that chunk. Predict's Rust and Python paths now produce byte-identical signal windows for such a model, held equal by a new backend-parity test. ((#342))

What's Changed

Other Changes

  • feat(predict): implement signal-context-bases on Rust inference path by @jayhesselberth in #342

Full Changelog: v0.12.3...v0.13.0

Release v0.12.3

Choose a tag to compare

@github-actions github-actions released this 16 Sep 19:25

Fixed

  • --pod5 can now point at a directory of POD5 files (a MinKNOW run) on the Rust extraction path, not just a single file. rust/src/pod5_cache.rs now resolves reads through escapepod_signal::cached_dataset (escapepod-rs#384) instead of cached_reader, with no file/directory branch — a single-file input costs nothing extra, since every file a dataset touches is opened through the same per-file reader cache either way. --backend auto/--backend rust both work on directory input now; --backend python already did. ((#339))

What's Changed

Other Changes

  • fix(pod5): accept a directory of POD5 files on the Rust extraction path by @jayhesselberth in #340

Full Changelog: v0.12.2...v0.12.3

Release v0.12.2

Choose a tag to compare

@github-actions github-actions released this 16 Sep 11:56

Fixed

  • --resume can now recover a run an external kill (SLURM walltime, OOM, scancel) cut off mid-loop, not only a clean exit. model_last.pt was written exactly once, after the epoch loop exited, so an interrupted run left --resume nothing to find and every retry started over from a fresh random seed -- the whole reason a Snakemake restart-times retry existed was defeated by the checkpoint design it depended on. A new rolling checkpoint, model_resume.pt, is written atomically at the end of every epoch and carries the full resumable state (history, early-stopping patience, the adversarial head, the ClipGrad buffer, per-rank RNG state, the run seed) plus a recipe and corpus fingerprint that --resume now refuses to cross rather than silently warm-starting a different run. It is deleted on a clean finish and deliberately left undeclared in the Snakemake rules, since a declared output would be deleted by Snakemake itself the moment a walltime kill "fails" the job -- exactly when the retry needs it most. train.smk and compare_models.smk now pass --resume {output_dir}/model_resume.pt unconditionally. ((#330))

What's Changed

Other Changes

Full Changelog: v0.12.1...v0.12.2

Release v0.12.1

Choose a tag to compare

@github-actions github-actions released this 15 Sep 14:10
0afaf84

Fixed

  • data prepare against a POD5 pre-filtered to a subset of the BAM's reads
    no longer aborts the whole run.
    A read the POD5 does not carry was counted
    as a per-read failure on both backends, so the expected BAM/POD5 mismatch
    produced by escpod bam-filter (and the pipeline's own Filter stage) crossed
    MAX_FAILED_READ_FRACTION and raised, discarding a corpus whose every present
    read had extracted correctly. Absent reads are now their own expected-exclusion
    bucket, reported as reads_missing_from_pod5 in the stats and in the "Read
    yield" log line, and the failed fraction is measured over the reads actually
    attempted. Issue #265's zero-tolerance policy is unchanged for genuine
    failures: leech_core.extract_training_chunks now returns
    (chunks, n_missing_from_pod5) so the Rust dispatcher's "reads went in, no
    chunks came out" check can still fire on reads that were found in the POD5. ((#325))
  • leech model train --model-config best_params.json no longer crashes on leech model optimize's own output. best_params.json always records selection_metric for provenance, but train's model-config loader passed it straight through to the model constructor, raising TypeError: model got unexpected keyword argument(s): selection_metric on the very first run of the documented optimize-then-train workflow. selection_metric is now dropped before model construction, the same way checkpoint_metric already was. ((#326))
  • leech data merge no longer silently collapses a same-body contrast to one class. Pairwise relabeling matched chunks against each input file's original labels values rather than which -i argument supplied the file, so two files sharing an internal label (e.g. two preparations of the same tRNA body, contrasted by attached ligand rather than body identity) resolved every chunk to the same class — with the log still printing a correct-looking 0/1 assignment. _parse_and_validate_inputs and merge_and_split_chunks/merge_and_kfold_split_chunks now track group membership by file provenance (relabel_by_file) instead of re-deriving it from label values; the remaining value-based path (process_comparison_spec's TSV workflow) now raises if the two groups' label sets overlap instead of collapsing silently. ((#327))

What's Changed

Other Changes

  • fix(train): drop selection_metric from --model-config before model construction by @jayhesselberth in #326
  • fix(merge): track chunk provenance instead of re-deriving group membership from labels by @jayhesselberth in #327
  • fix(prepare): don't count reads absent from a pre-filtered POD5 as failures by @jayhesselberth in #328
  • build: move both escapepod pins to v0.26.0 by @github-actions[bot] in #320
  • chore: release v0.12.1 by @jayhesselberth in #329

New Contributors

  • @github-actions[bot] made their first contribution in #320

Full Changelog: v0.12.0...v0.12.1

Release v0.12.0

Choose a tag to compare

@github-actions github-actions released this 14 Sep 04:15

Added

  • LeechDataset warns when a signal crop reaches outside the stored chunk.
    left_context/right_context crop [focus - left_context, focus + right_context); when a corpus was prepared with a narrower
    signal_context than that window, the shortfall was zero-padded with
    nothing logged. In 2026-aa-trna-models, production corpora stored
    signal_context [225, 225] and trained with right_context 300, so the
    last 75 samples of every chunk were zero for two production retrains before
    anyone noticed. LeechDataset now logs one warning per dataset the first
    time this happens, naming the requested window, the stored window, and the
    number of padded samples; strict_window (leech model train --strict-window) raises instead. The same silent zero-pad also existed in
    the plainer symmetric case (a --signal-len wider than the stored chunk with
    no left_context/right_context set at all), which now warns/raises through
    the same path. Detection lives in one shared _warn_or_raise_padding used by
    the asymmetric (_note_crop_padding) and symmetric (_note_plain_pad)
    cases, called from both the row (_prepare_signal) and block
    (_prepare_signals_block) fill paths, so pre-loaded and streamed corpora are
    both covered. strict_window is also recorded in the saved config.json so
    a checkpoint's guard setting can be audited later. ((#255))

  • leech data prepare --mask-seq-left-of-focus/--mask-seq-right-of-focus
    blanks sequence-branch bases on one side of the focus base.
    With
    seq_encoding: signal_kmer (the default), sequence_with_kmer_context
    begins with acceptor-stem bases 5' of a 3'-end motif and identifies a tRNA's
    body outright, so every model trained so far had tRNA identity leaked to it
    through the sequence branch regardless of what the signal/feature branches
    learned. The new flags write N over sequence-branch characters strictly to
    one side of the focus base — in both sequence (base_onehot) and
    sequence_with_kmer_context (signal_kmer) — reusing the existing
    "non-ACGT maps to -1 and is skipped" convention (sequence_to_int,
    encode_signal_kmer) rather than a new mechanism; the focus base's own
    character is never masked. Masking is baked into the corpus at data prepare
    time (ChunkConfig.mask_seq_side, applied once in LeechRead.get_chunk from
    the exact local geometry, not reverse-engineered from stored arrays later),
    recorded in prepare_config.json, carried into the trained model's
    config.json by model train, and auto-applied by predict to live chunks
    so inference sees the same masked geometry the model was trained on. Not
    implemented in the Rust extraction path — data prepare falls back to the
    Python workers and predict --backend rust raises; --backend auto falls
    back with a warning, matching the existing recover_softclip_signal gating.
    No leech model train flag: the mask is a property of the corpus, not a
    training-time choice. Fixes #256. ((#256))

  • leech data prepare --signal-context-bases L,R: a base-defined signal
    window.
    Mutually exclusive with --signal-context. Instead of a fixed
    number of samples on each side of the focus base, the window is cut at the
    base-to-signal map positions of offsets -L and +R (inclusive), so a fast
    and a slow read cover the same bases of context rather than the same
    samples — a sample-defined window can't reach a base at +24 on a slow
    read (~36 samples/base) without also reaching +25 on a fast one
    (~24 samples/base), which for the charging assay is where the LDX barcode
    starts.

    The resolved sample interval is padded (right-aligned zero-fill, the
    conservative default) or centre-cropped to a fixed --signal-len, which
    defaults to (L + R + 1) * 36 samples/base (sized off a conservative
    slow-read rate so a typical read pads rather than crops) and can be
    overridden explicitly. A focus base near either edge of a read gets a
    narrower window rather than a dropped chunk, matching the one existing drop
    rule (a focus base with no signal boundaries at all).

    Implemented in both data prepare backends (Python LeechRead.get_chunk and
    the Rust training pipeline), held to identical output by
    tests/test_backend_parity.py. The resolved window is logged, recorded in
    prepare_config.json, and carried into the trained model's config.json so
    predict re-derives the same window and the ONNX contract sidecar states it
    (the Rust predict pipeline does not yet implement it and falls back to the
    Python predict path, as it already does for a few other options only the
    training-side Rust pipeline supports). ((#278))

  • --loss noise_corrected_bce: a label-noise-aware loss for enrichments with a known, per-group impurity rate. losses.NoiseCorrectedBCEWithLogitsLoss applies a forward correction (Patrini et al. 2017) using a per-sample label-flip probability looked up from each chunk's source_group via --label-noise-rate group=rate[,group=rate,...] (e.g. --label-noise-rate gold=0.09,enzymatic=0.17); groups not named get rate 0 and the loss reduces to plain BCE exactly. Rates are measured upstream (leech does not estimate them) and are recorded verbatim in config.json; predict is unaffected. Reduces by element count like the other losses, so it decomposes correctly under --gpus N. ((#279))

  • Checkpoint selection and loss shaping for the low-FPR operating regime.
    --checkpoint-metric (and grid search's --selection-metric) now also
    accept tpr_at_fpr:<f> (TPR at a fixed FPR) and callable_at_precision:<p>
    (fraction of validation reads callable at a precision floor) alongside the
    existing auto/val_acc/val_f1/val_auc -- binary tasks only, computed
    once on the gathered predictions under --gpus N like the others, and
    reported back in a run's summary.json/eval test output when selected, so
    two runs using the same threshold can be compared after the fact.
    leech.metrics.tpr_at_fpr/callable_at_precision implement the two metrics
    directly from (labels, probs). FocalBCEWithLogitsLoss gains an optional
    --focal-neg-gamma, making the focal loss asymmetric: a higher gamma on
    negatives than positives down-weights easy negatives harder, concentrating
    gradient on the hard negatives that set FPR at a given threshold. Leaving it
    unset (or set equal to --focal-gamma) reproduces the current loss
    bit-for-bit. ((#280))

  • Time-stretch augmentation for speed invariance. --augment-time-stretch MIN,MAX (leech model train/optimize) resamples each training chunk's signal window by a per-sample factor drawn uniformly from [MIN, MAX], about the focus position, cropped/padded back to signal_len. The CSR base-to-signal map is scaled by the same factor so signal_kmer stays aligned, and the dwell/dwell_mean feature channels scale with it (dwell_log shifts by log(factor); dwell_ratio, dwell_std and level channels are unchanged). Implemented on the batched LeechDataset.__getitems__ fetch path (one (B,) factor draw per batch, numpy gather rather than a torch index_select); off by default, recorded in config.json, and never applied to validation. Targets the ~7-point sensitivity gap the charging model shows on translocation speeds it never saw during training. ((#281))

  • A per-chunk junction_indel/junction_mapped field, plus a predict-time
    abstention rule that uses it.
    data prepare now records, for every chunk
    whose motif was found through a ReferenceMotifSearcher, the CIGAR-measured
    disruption at the motif's mapped span (junction_indel = mapped_len - len(motif), 0 when exact) and whether that measurement was possible at all
    (junction_mapped) — the value the motif searcher already computed to decide
    whether to keep or reject a motif, previously discarded once that decision was
    made. Both prepare backends emit it identically, since the measurement is made
    once in Python and both backends consume the same motif search result.
    --sample-weight-field FIELD generalizes --balance-groups's inverse-
    frequency sampling to any chunk metadata field (--balance-groups is now a
    named instance of the same mechanism), so --sample-weight-field junction_indel over-samples chunks whose motif junction is disrupted.
    leech predict --abstain-on-junction-indel enforces --min-margin only on
    calls whose junction is disrupted, writing unc/BELOW_THRESHOLD_LABEL for
    those below threshold while an intact junction bypasses the margin check
    entirely; junction_indel/junction_mapped are always recorded (BAM ji/jm
    tags, or junction_indel/junction_mapped TSV columns) so the rule can be
    re-applied offline. eval test reports accuracy/F1 (and, for binary models,
    the full metric set) stratified by junction_indel == 0 vs != 0 whenever the
    test corpus carries the field. The field is not fed to any model as an input
    channel — it is a sampling/abstention signal only. ((#282))

  • Symmetric (non-causal) TCN padding and recorded per-channel feature standardisation, both opt-in. TemporalBlock/TCN gain a causal: bool = true param (components.py); causal = false splits the same total padding evenly across both sides instead of putting it all on the left, doubling the receptive field on either side of a position for the fixed-window classifiers this project trains (parameter shapes are unchanged, so a checkpoint loads either way, but the trained values aren't a meaningful initialization for the other mode). It reaches every TCN-family TOML config (tcn_dwell*.toml) as a [params] entry, settable via --model-config or a future [[variants]] pin, same as norm_type.

    --standardize-features computes per-channel feature mean/std once over the training corpus (LeechDataset.__init__, reusing the tensor the feature_noise augmentation already stacks) and f...

Read more

Release v0.11.1

Choose a tag to compare

@github-actions github-actions released this 12 Sep 13:29
83ecc13

Changed

  • escapepod pin bumped v0.21.0 -> v0.24.3 (rust/Cargo.toml tag and the
    escapepod PyPI floor, moved together as always). Picks up upstream's DP
    speedups to dp_step_with_dwell_penalty — the exact function leech's own
    refinement preset (RefineSettings::move_table_refinement, which resolves
    to RefineAlgo::DwellPenalty) drives on every base of every chunk during
    data prepare and predict. Upstream measured roughly 8-11x on the DP
    loop itself (criterion) and +25.5% / +8.7% end-to-end wall clock across the
    contributing releases (v0.24.1, v0.24.3).

    Not bit-identical to v0.21.0 output. Two of the upstream changes
    (v0.24.1's cumsum-prefix-sum reordering, v0.24.3's halved max_check) are
    documented, deliberate correctness tradeoffs for RefineAlgo::DwellPenalty
    — escapepod's own real-data A/B saw up to 1.76% of reads shift p_charged
    by some amount and 0.025% flip a discrete call. A data prepare run
    against this release will not bit-match chunks prepared under leech
    <=0.11.0. leech's own Python/Rust backend parity is unaffected, since both
    backends call the same upstream function with the same settings. (#250,
    #251)

What's Changed

Other Changes

Full Changelog: v0.11.0...v0.11.1

Release v0.11.0

Choose a tag to compare

@github-actions github-actions released this 07 Sep 13:44
f67d3a1

Added

  • Single-node multi-GPU training: leech model train --gpus N. N
    data-parallel ranks on one node, opt-in, train only — eval test measured
    input-bound (53-79% GPU with every DataLoader worker pegged), so DDP buys it
    nothing and would only add risk to the path that produces the scores.

    --batch-size stays the GLOBAL batch and is split across ranks. The
    PyTorch convention is the opposite — per-rank, so the effective batch grows
    with the GPU count — and it is wrong here, because a leech run is one arm of
    a paired comparison: the same command line has to mean the same recipe at any
    --gpus, or every number already measured needs re-measuring. Splitting
    leaves the optimizer-step count, the LR schedule, ClipGrad's quantile
    buffer and the grad_accum_split arithmetic exactly where they are.

    Measured on the production charging corpus (6.72M chunks, 36.5 GiB npz,
    2 epochs, one 4xA30 node):

    --gpus job wall speedup peak RSS
    1 24:31 1.00x 40.6 GiB
    2 16:40 1.47x 88.3 GiB
    4 10:53 2.25x 157.6 GiB

    2.25x rather than 4x because ~3 min of per-rank corpus load and the final
    eval do not shard. Memory is ~1:1 with the corpus per rank, so --gpus and
    the job's mem_mb have to move together
    or the second rank is OOM-killed
    during its load; pipeline/workflow/rules/train.smk scales both off one
    train_gpus key.

    Four things that would each have been silent: WeightedRandomSampler handed
    to N ranks gives every rank the same oversampled draw, so
    DistributedWeightedSampler shards one global multinomial and the union of
    the shards is exactly the single-GPU epoch; validation shards without padding
    and gathers before it scores, so the metrics are equal to the single-GPU
    numbers rather than close to them; the checkpoint keeps the single-GPU
    state_dict keys, since a module. prefix either fails to load downstream or
    loads nothing under strict=False and exports an untrained graph; and the
    auxiliary heads are wrapped individually, because they sit in the optimizer
    but outside the model and a DDP around the model alone never allreduces them.

  • Daily check that the two escapepod pins agree
    (.github/workflows/escapepod-sync.yml). rust/Cargo.toml tags the
    escapepod-signal crate and pyproject.toml floors the escapepod PyPI
    package; they are one upstream released in lockstep, and leech drives both.
    Dependabot bumps the tag-pinned crate on its own schedule (#236) and cannot
    know the Python package has to follow, so merging one of its PRs leaves main
    skewed with nothing red. The check compares them daily and opens the matching
    bump rather than failing dependabot's PR, which would leave a human to do the
    second half by hand.

  • Dependabot PRs merge themselves once CI has passed
    (.github/workflows/dependabot-auto-merge.yml). Not GitHub's native
    auto-merge, which needs allow_auto_merge and required status checks on
    main to mean anything — without required checks it merges immediately
    rather than waiting for CI. Triggering on workflow_run makes "CI finished
    and it was green" the entry condition instead. escapepod-signal and any
    major bump stay with a person: the first because its PyPI twin has to move in
    the same commit, the second because a green suite says the tests still run,
    not that the semantics held.

Fixed

  • Weighted CrossEntropyLoss is normalized by the global summed weight under
    DDP.
    Every other loss in the trainer reduces by element count, which equal
    shards make exact; CrossEntropyLoss(weight=...) divides by the summed weight
    of the samples it sees, so each rank normalized by its own shard's class
    composition and the averaged gradient was the single-GPU one only when the
    shards happened to draw alike. 8% off on a fixture whose shards differ, with
    nothing raised.

  • DataLoader workers fork inside spawned ranks.
    multiprocessing.spawn.prepare forces a spawned child's default start method
    to match how it was created, so every DataLoader a rank built pickled the
    dataset's stacked tensors through /dev/shm instead of COW-sharing them —
    voiding the invariant those contiguous buffers exist for. It surfaces nowhere
    near the cause and does not look like memory: shm pages are charged to the
    cgroup but not to RSS, so the run died at 176 GiB RSS against a 244 GiB
    allocation, in the fourth rank's validation loader, as No space left on device. Fixing it took the same run to 157.6 GiB.

  • Spawned ranks configure their logging. setup_logging runs in the click
    entry point, which a rank never reaches, so the leech logger had no handler
    and every INFO line was dropped — including rank 0's, where the effective
    batch, the sampler statistics and the encoding-fallback warning are reported.
    The run worked and said nothing about itself.

Changed

  • escapepod moved to v0.21.0 on both backends — the escapepod-signal
    crate and the escapepod PyPI package together, since a skew between them
    lets the two prepare backends compute different dwells and different
    level-derived features from the same read (the divergence behind #193).
    Validated with tests/test_backend_parity.py against a leech_core actually
    built on v0.21.0; a compile only says the API still exists.

Internal

  • tests/test_workflows.py fails when a GitHub Actions workflow does not
    parse. Nothing else in the repo reads those files, and a scheduled workflow
    that never fires is indistinguishable from one with nothing to report.

What's Changed

Other Changes

Full Changelog: v0.10.0...v0.11.0

Release v0.10.0

Choose a tag to compare

@github-actions github-actions released this 30 Aug 23:41

Fixed

  • Exported ONNX graphs load in a runtime that is not PyTorch.
    charging_tcn_rna004@v0.1.0 shipped from this exporter with a graph no
    released escpod binary could load — tract parses it and then gives up
    during shape analysis (rnabioco/escapepod-models#96). onnxruntime loads it
    fine, which is why verify_onnx had nothing to say and the failure surfaced
    at integration rather than at build time: "it exports and round-trips" is a
    weaker claim than "a runtime can load it", and only the second one ships.

    Two independent causes, both measured against tract 0.23.5 through the load
    path escapepod_classify actually uses:

    • adaptive_avg_pool1d with an output size that does not divide the
      input
      (390 -> 11 here). The dynamo exporter open-codes it as
      Unsqueeze -> Transpose -> GatherND -> Transpose -> Where: a rank-8 gather
      over an all-constant index and mask, which tract refuses pinned, unpinned,
      and with value_info cleared. No post-hoc rewrite helps.
      models.components.AdaptiveAvgPool1d now writes the same arithmetic as one
      matmul against a constant [L_in, L_out] segment-mean matrix, using
      PyTorch's own bin rule (upsampling included — ResNetDwell pools 4 up to
      11), in float32 outside autocast so the accumulation matches the aten op
      under AMP. Agreement with the aten op is 2.4e-07 over a grid of lengths and
      output sizes. One implementation, so the registry layer, resnet_dwell,
      transformer_dwell and the tests/reference_* oracles move together and
      the config-vs-reference parity tests stay bit-exact. SignalCNN's
      AdaptiveAvgPool1d(1) is untouched: 1 divides everything and exports as
      GlobalAveragePool.
    • value_info. Dynamo writes one entry per intermediate — 667 for this
      model — carrying the batch axis as the symbol batch, because that is
      what dynamic_axes asked for. A consumer that pins the batch then cannot
      unify, and tract fails at the first convolution. strip_value_info drops
      them, and export_onnx always calls it. Nothing needs them: every runtime
      re-infers, onnx.checker is satisfied, and every graph escpod loads today
      has zero. Initializers are untouched, external data references included.

    Measured on the shipped TCNDwellResidualLN weights, no retrain: 479 -> 319
    nodes, GatherND 2 -> 0, Gather 76 -> 0; tract loads, optimizes and runs at
    batch 1 and 32 (was: five distinct failures), max |dlogit| 5.72e-06 against
    torch over 256 real chunks with 0 decision disagreements; onnxruntime vs torch
    1.335e-05 over 4096 real chunks (shipped graph: 1.4305e-05).

Changed

  • Numerical output moves by ~2.4e-07 for the architectures that use a
    non-dividing adaptive pool (ResNetDwell, TransformerDwell, the TCN family
    and any config using the AdaptiveAvgPool1d registry layer). Same weights,
    same decisions — existing checkpoints load and predict as before, but exact
    float equality with a v0.9.0 run does not hold.

Documentation

  • The exporter's case for the dynamo path is re-measured rather than inherited,
    since the pool no longer emits an aten adaptive pool and that could have
    retired the reason. It did not: dynamo=False still refuses both the aten
    pool and leech's replacement, because torch.jit.trace turns .shape[-1]
    into a Tensor and takes the dynamic-length fallback. A test pins it.

What's Changed

Other Changes

Full Changelog: v0.9.0...v0.10.0

Release v0.9.0

Choose a tag to compare

@github-actions github-actions released this 26 Aug 12:01

Added

  • leech.crf.evaluate: decode a corpus, match it to references, report per
    group.
    The generic half of CRF evaluation — what a panel is (which classes
    exist, which share a flowcell) stays with whatever defines the panel; what
    arrives here is a reference set, a grouping and a corpus.

    Three rules it holds rather than leaving to callers, because each produces a
    plausible-looking wrong number:

    • Match what the model emits. emitted_references applies
      target[state_len:] once. Scoring against full-length targets forces
      state_len leading deletions into every alignment, which inflates every
      distance and compresses the margin — an aligner puts those deletions where
      they help most, discounting wrong references more than the right one.
    • Report per group. When classes are crossed with batch, one pooled table
      measures batch. balanced_recall takes the grouping as an argument (only
      the caller knows whether their classes are confounded) and raises when
      no group has reads, because a null headline serializes fine and ships.
    • Balanced, not raw. A pooled accuracy over unbalanced classes is
      dominated by the deepest class.

    lev_vs_refs scores one decode against the whole reference set at once, which
    is the shape of every evaluation loop; scoring R references one at a time is R
    DP tables per read. Its vectorisation recovers the serial insertion term
    exactly (j + cummin(tmp[k] - k)), and that identity is asserted against the
    scalar implementation on random strings rather than assumed. edlib is used
    where importable, with the pure-Python fallback kept under its own name so a
    test compares the two rather than comparing edlib with itself.

    Validated end to end on the production ldx corpus: 16 references at emitted
    length 44 from 48, 4000 held-out reads decoded, and per-flowcell reporting
    that correctly finds 8 classes in each — the pilot's code-flowcell confound,
    which is exactly why pooling would be wrong.

  • ONNX export, for the classifier arms and the CRF encoder (#217).
    leech model export --format onnx beside the existing --format torch
    (unchanged default), and leech.crf.export.export_crf_onnx. torch.export
    makes a model loadable by anything with PyTorch and by nothing else; a runtime
    consuming ONNX — which is what escapepod-rs runs — could not load a leech
    model at all.

    Both use the dynamo exporter at opset 18. dynamo=False, the obvious
    first attempt, fails on these architectures with an adaptive_avg_pool1d
    error that reads like a model problem and is an exporter limitation; that is
    documented where someone will hit it, and a regression test pins it.

    Each export writes a contract beside the graph, carrying the two things a
    consumer needs and cannot recover from it: which input is which (including
    that the signal_kmer sequence input is built in the dataset, not the model,
    and that leech-core ships that encoder), and what the output means — a
    single BCE logit, not a two-class softmax. The CRF's contract additionally
    carries standardisation, which is in neither the config nor the checkpoint,
    and its emitted references (target[state_len:]), computed from the
    state_len the encoder declares.

    Verified across the serialization boundary rather than in process:
    onnxruntime against torch, 3.58e-07 for the CRF encoder against a float32 eps
    of 1.19e-07.

    New onnx extra (onnx, onnxruntime, onnxscript). CI installs it so the
    round-trip tests run rather than skip.

  • leech model train-crf, the CLI for the CTC-CRF trainer. Sits beside
    model train rather than in a group of its own: it is the same workflow step,
    a different task. Every CrfTrainConfig field is exposed, and a test asserts
    each option actually reaches the config — a click option that silently does
    not is how a sweep ends up running the default every time.

    The summary reports the emission rule (target_len -> emits target_len - state_len) because widening the window to get a longer decode is the mistake
    it prevents, and prints a second table only when the run discarded steps or
    saw non-finite gradients, since a discarded step is otherwise invisible.

  • leech.crf.training: a CTC-CRF trainer. CrfTrainer runs the schedule
    and writes model.pt plus a model.json sidecar. The sidecar is not optional:
    the standardisation constants live in neither the architecture config nor the
    checkpoint, so weights alone cannot be used correctly.

    Separate from leech.training.Trainer, which is classification-locked through
    pos_weight, num_out, BCE/focal/CE and AUROC/F1 checkpointing — a sequence
    task shares none of it, and forcing it through would put the production
    classifier path at risk. The decisions are split out as plain functions
    (compute_standardisation, apply_quality_gate, resolve_split,
    encode_targets, select_checkpoint) so they are testable without a GPU; the
    loop is mechanical, the decisions are where runs go wrong quietly.

    Verified against production data, not just fixtures: standardisation over the
    391,174 x 3000 ldx corpus reproduces the shipped model's recorded constants
    (61.8216743766 / 9.5716818880) with a delta of 0.000e+00 on both, and the
    train/test counts match what plan_corpus derives independently from the
    manifest (283,296 / 107,878). A two-epoch GPU run trains at ~40 s/epoch with
    loss falling 0.4446 -> 0.0639.

  • leech.crf.corpus: cut a CRF training corpus from a manifest.
    plan_corpus decides which reads and in which split, touching no POD5;
    build_corpus extracts their signal, streaming it to a memory-mappable
    <out>_X.npy beside a <out>_meta.npz. The signal is never held in RAM as a
    whole — an 80-plex corpus is tens of gigabytes, and the memmap is what makes
    the size a disk question instead of an allocation that fails.

    The two stages are separate because everything subtle is in the plan, and a
    corpus planned wrongly still trains and still reports a number. Four rules,
    each pinned by a test: a cap only caps if every class can reach it
    (per_group="auto" is the rarest class's trainable depth, with the test
    fraction reserved first); the split is carved before capping and ranked per
    class globally across batches, since per-(batch, class) ranking multiplies
    the cap by the batch count whenever classes are crossed with batch; batches
    are interleaved rather than concatenated, or the whole held-out set comes from
    whichever batch sorts first and the headline number measures batch; and
    sharding happens after planning, so every shard keeps its share of one global
    split. Extracting nothing, or less than half the plan, is a hard error with a
    different message for each — the causes differ, and a 0-row corpus otherwise
    exits cleanly and reaches a GPU job.

    Validated against the production ldx manifest: 1,139,602 rows plan to 391,174
    reads at chunk=3000, the same count escapepod-models' extractor reports for
    that input, with the training pool balanced exactly across all 16 groups and
    held-out reads drawn from both flowcells.

    load_corpus / load_corpus_meta read both this layout and the legacy
    single-.npz one, so corpora written before the split layout keep loading.

Changed

  • The signal-level k-mer encoding comes from escapepod-signal rather than
    being held here (escapepod-rs#271 / #272; requires escapepod 0.16.0).
    rust/src/encoding.rs and sequence_to_int are now calls into
    escapepod_signal::seq_encoding.

    leech held the only copy of this rule, inside a crate-type = ["cdylib"]
    Python extension module — so a native runtime for a leech signal_kmer model
    could not link it and had to transcribe it, which is a second definition that
    diverges silently. It is also the natural pair to escapepod_signal::mapping,
    which produces the base-to-signal map the encoding consumes: the producing
    half was already upstream and the consuming half was not.

    This is a delegation, so the only acceptable outcome is identity: 198 parity
    tests pass unchanged, including test_backend_parity.py, which compares every
    array in the npz between the Rust and Python backends against a Python
    reference this change does not touch.

    The k-mer context slice delegates too, via sequence_bases_with_context
    (escapepod-rs#274, escapepod 0.16.1). It could not at first: leech needs the
    window as bases, since the corpus serializes sequence_with_kmer_context
    as a string, where upstream only offered ints. Upstream now exposes both forms
    over one windowing rule, with sequence_to_int(bases) == ints pinned by a
    test there — so the three halves of the signal-level k-mer path (the map, the
    window, the encoding) all live in escapepod-signal and none is duplicated
    here.

    That third one is the highest-stakes of the three: it is where before and
    after are not interchangeable, and swapping them displaces every k-mer
    silently because the encoder only sees the total width. It is also the most
    directly checkable — sequence_with_kmer_context is one of the fields
    test_backend_parity.py compares array-by-array between backends.

    Only rust/Cargo.toml's git tag moves to v0.16.1; the escapepod Python pin
    stays >=0.16.0, because the new function is in the Rust crate and not in the
    Python bindings.

Fixed

  • signal_kmer degraded to base_onehot on the strength of one chunk, and
    the checkpoint did not record that it had
    (#230). Three separable defects
    that combined into a run which finishes, looks fine, and trained on a
    different model input than was asked for.

    The --seq-encoding default is signal_kmer, so a corpus written by one
    leech version and read by another that finds no base-to-signal maps warns and
    carries on. That reached production through a projec...

Read more