Skip to content

v4.3.0 — embedding diversity, and a Rao that stops being a presence mask

Choose a tag to compare

@mikessh mikessh released this 28 Sep 10:42
· 67 commits to master since this release

The geometry half emits embedding diversity, not only Rao — and stops emitting a presence mask
under a diversity name. Requires vdjtools >= 4.3.0.

Fixed: Rao of a sub-floor locus was 0.0, and that is not a measurement

rsig:div:<L>:rao came out as exactly 0.0 when a locus held a single clonotype. The arithmetic is
right — a one-point set has no dispersion — and the column is then not a diversity read-out. It is
mask:<L>:present in different units, sitting at the bottom of a distribution whose other values
are around 16.

On 885 bulk blood RNA-seq samples over seven loci it affected TRD in 118 samples (13.3%), TRG in 8,
IGH in 1 and IGK in 1. Against the non-zero distribution (1st percentile 16.43, median 16.63) a
single-clonotype locus lands about 29 robust deviations below the 1st percentile after
standardising, with nothing in the output saying the value was categorically different.

Both consequences were found by a check rather than by a failure:

  • A Cox screen over 261 channels returned a hazard ratio of 739 per standard deviation at
    p = 7×10⁻¹⁵⁵
    for rsig:div:IGH:rao against overall survival, on 838 patients and 305 events —
    off one patient. The Spearman correlation between that channel and survival is −0.023:
    there is no monotone association at all, and the fit was reading a single extreme point.
  • A downstream baseline moved by 0.077 ROC-AUC in one direction and 0.054 in the other, from
    four values in one cohort.

The whole div/disp family is now a hole below the same floor band_shares has always used,
with mask carrying the reason. The geometry is still computed there: Φ is measurable from
three clonotypes when its dispersion is not.

Added: the embedding-diversity family

Rao is the metric-space analogue of Simpson diversity, and it was the only such read-out this half
gave up — everything else it knew was inside a rotated component, which has no name a domain reader
can use. The reason to embed a repertoire is that a counting index cannot see that two clonotypes
are one substitution apart.

Per locus, all from the chunked pass that already computes Φ: Simpson per germline component
(q_v, q_j, q_c, which sum to rao because the strides are literal column offsets — "how
much of this is V-driven" needs no attribution model), where the diversity lives as a composition
(q_frac_*), evenness, eff_dim (the richness analogue: directions of receptor space occupied —
1,000 clones in one convergent cluster occupy few, 1,000 unrelated clones occupy many),
eff_dim_pr, per-compartment diversity, and displacement channels including class-switch geometry
for IGH.

Only ever within one locus: each locus has its own prototype panel, so a distance between
Φ(TRA) and Φ(TRB) is arithmetic without a meaning.

Cost, measured at the shipped K = 256. Per locus added to the Φ pass: 3.3 ms at 50
clonotypes, 3.9 ms at 200, 25.3 ms at 700, 34.3 ms at 2,500, 61.9 ms at 10,000. A realistic
seven-locus blood sample pays roughly 125 ms, about doubling rsig's own per-sample cost and
adding under a tenth to a full signature run. Nearly all of it is one eigendecomposition: Σλ is
the trace of the covariance and Σλ² its squared Frobenius norm, so eff_dim_pr needs no spectrum
at all, and eff_dim takes eigenvalues on the smaller of the p×p covariance and the n×n Gram.

Added: mask on the geometry half, named=, and --min-clonotypes

rsig:mask:<L>:present / :estimable, under the same names and convention as the statistics half.
mask is a feature block, not diagnostics: which loci a donor resolved is biology, and adding
the presence flags moved an external ROC-AUC from 0.6243 to 0.6676 on the cohort where it was
measured. rsig(..., named=True) returns the reportable raw blocks, matching vsig.
mir signature --min-clonotypes N exposes the estimability floor — a flag the command's own help
text had named for several releases without the parser defining it.

No corpus is invalidated

A channel is pass-through: the rotation is indexed by raw_columns and nothing else, and
corpus.apply fills every registered channel from the sample. Verified against the published
4.1.0 naive artifact downloaded fresh from its release — all 18 new TRB channels finite, no
channel column missing. No refit, no new download.


Full notes in CHANGELOG.md.
Pair with vdjtools 4.3.0 for the statistics half; run both against the same corpus name and join
on sample_id.