feat(crf): a CTC-CRF trainer

[jayhesselberth]

CrfTrainer runs the schedule over a corpus (#218) and writes model.pt beside
a model.json sidecar. The sidecar is not optional — standardisation lives
in neither the architecture config nor the checkpoint, so weights alone cannot
be used correctly.

Why a separate trainer

leech.training.Trainer is classification-locked all the way down: pos_weight,
num_out, BCE/focal/CE, AUROC/F1 checkpointing, and a wrapper that forwards
three input branches. A sequence task shares none of that, and threading one
through would put the production classifier path at risk to save a few hundred
lines.

The decisions are functions, not loop internals

The loop is mechanical; the decisions are where runs go wrong without failing.
Each is a plain function, tested without a GPU:

function	the silent failure it prevents
compute_standardisation	float32 summation over billions of samples loses its tail; streaming in float64 blocks also means a corpus larger than RAM costs nothing to summarise
apply_quality_gate	an unscored read cannot pass a gate, so partial coverage trains on a small non-random subset — one corpus went 56% → 13.5% that way. Refuses rather than gates
resolve_split	a read-level split when classes are crossed with batch reads optimistically; held-out batch wins, then the corpus's own split, then a seed
encode_targets	12,288 dict lookups per step (1.34 ms of a 39 ms step) vs 0.06 ms for a prebuilt array
select_checkpoint	a run that passed through 0.0047, diverged, and recovered to 0.0072 ships weights it already beat by 53%

select_tol is deliberately loose (25%). Training loss does not rank models
at this scale — one seed reached 0.0045 where another reached 0.0072 on the same
split and measured 0.2pp worse on held-out recall — so this is a divergence
detector, not a ranking.

Verified against production data

Standardisation over the real 391,174 × 3000 ldx corpus:

computed      : mean=61.8216743766  std=9.5716818880
shipped model : mean=61.8216743766  stdev=9.5716818880
delta         : 0.000e+00 on both

Train/test counts (283,296 / 107,878) match what plan_corpus derives
independently from the manifest in #218. Two epochs on an A30: ~40 s/epoch, loss
0.4446 → 0.0639.

The gate was checked to be live rather than a no-op — the corpus was extracted
at exactly (66, 5) so everything passes at defaults, while (70, 8) keeps 236,119
and (75, 10) keeps 167,197. That is the point of gating at training time: a
sweep costs a run, not a re-extraction.

The instrumentation earned itself immediately

Epoch 1 of the GPU run reported 3 skipped steps and 3 non-finite gradients at
|g|max 50.45, settling to 0 and 2.22 by epoch 2. That is normal GradScaler
warm-up — but it is undetectable without the counters, because on non-finite
gradients scaler.step() silently does nothing and update() halves the scale.
The scale dropping is the signal.

Testing

• 34 trainer tests, including an end-to-end CPU run and a check that the loss
  actually falls
• Full suite 1442 passed, 44 skipped
• ruff format --check, ruff check, ty check, zensical build clean

Not in this PR

The CLI surface (leech model train-crf vs a leech crf group — worth settling
before it ships), the ONNX export, and metrics/eval. Nor the paired 3-seed
retrain against the shipped barcode_crf_ldx16: at ~20 min per 32-epoch run
that is now cheap enough to be worth doing properly as its own exercise.

🤖 Generated with Claude Code