Skip to content

v3.20.0 — the bound compresses instead of truncating

Choose a tag to compare

@mikessh mikessh released this 26 Sep 19:04
· 84 commits to master since this release

Centring and scaling is a per-column affine map: it cannot change a downstream model that standardises its own input, and it cannot change a rank correlation at all. So of everything a scale reference does, exactly two things can move a result — which columns it scales, and what happens at the bound.

np.clip(z, -8, 8) was many-to-one. Every sample past the bound collapsed onto one value, the ordering was gone, and nothing in the matrix said so. Scoring one cohort against two shipped references, the share of finite entries sitting at the bound:

block reference A reference B
rsig:div 71.41 % 2.50 %
vsig:len 49.84 % 3.15 %
rsig:contrast 5.35 % 19.28 %
all columns 8.74 % 3.44 %

Reference B is better on every block but one — and that block was what a transfer model rested on: rsig:contrast alone went AUC 0.7216 → 0.4108 on a held-out cohort. Choosing a reference was choosing which columns to truncate.

Changed. The bound is identity inside and a log1p tail outside. The scaled order of a column is its raw order; every value with |z| <= clip is bit-identical to 3.19.0; and the map is invertible, so unapply() recovers natural units without recomputing a cohort. Measured on a real 10-sample four-locus matrix, the squash merged 13 adjacent raw pairs (largest relative gap 9.253e-16, about four float64 epsilons) against the clip's 714 pairs and 0.4948. squash="hard" reproduces an archived matrix; clip=None removes the bound.

Added. Saturation reported four ways — ScaleReference.saturation(frame), report(frame), a cohort warning above 2 % of a column naming the columns, and a stderr line from mir signature every run. Plus compare_references(), so picking a reference is a table (columns scaled, median n_obs, frac out of bound, median centre shift) rather than a preference, and on_unscaled="hole" to mark the columns a reference could not scale.

Also. signature() no longer completes a partial cstar dict with a fake unreachable level — vdjtools 3.18.0 gives the hole directly (floor raised to vdjtools>=3.18.0), so standard is 689 columns now. And the pool benchmark was measuring against a baseline it cannot beat: four workers are 0.79x one in-process pass because the junction kernel threads over every core while each worker takes one. It gates the pool against one worker's kernel (2.65x) now.

Full notes in CHANGELOG.md.