Skip to content

v4.5.0 — every per-item loop, audited and measured

Choose a tag to compare

@mikessh mikessh released this 29 Sep 01:10
· 225 commits to master since this release

Every per-item loop in the package, audited and fixed where it mattered. All 84 modules read
twice for one pattern: Python iterating over data where a batched native call, a polars
expression, or a grouped computation does the same work. Equality or byte-identity is asserted
against the previous output in every case.

site what it was measured
infer_nt codon reconstruction Python per candidate, GIL held 0.853 -> 0.086 ms/key TRB (9.89x), 94x TRA
generate(engine="native") Python per sequence 149x-577x; IGH productive_only 4,029 -> 2,325,649 seq/s
infer._align_init Python per read, in the native path too 14.4x-36.6x, bitwise-identical
io.convert x9 readers a Python dict per input row 5.32x on 42,877 rows, all 10 fixtures byte-identical
sc.resolve_chains / sc.pair_chains a frame slice per cell 517x / 540x at 20,000 cells
model.score._pgen_nt_many a pool task per sequence one native.pgen_nt_batch call
pgen CLI, sc.paired_pgen a native call per row one batched call per kind / per chain
dynamics.expansion._exact_p betabinom.pmf per clonotype 10.3x-307.5x, max rel diff 1.08e-14
model.data.write_prepared a Python string per clonotype 4.6x, byte-identical
biomarker.cooccurrence._dense a dict lookup per row one replace_strict

Two native entry points that did not exist. native.pgen_nt_batch was the one native Pgen with
no batch, so every caller with more than one sequence either looped in Python or wrapped that loop
in a ThreadPoolExecutor dispatching one task per sequence -- the only per-item pool in the
package. It is gone. _core.translate_junctions translates a whole junction column the way the
legacy converters do, including the bidirectional out-of-frame walk polars cannot express: 0
mismatches
against this repo's own translation on 4,009 sequences at every frame offset.

generate(engine="native") is 244x on TRB and 577x on IGH productive_only, seeded from
(seed, row) so the output never depends on the thread count. The default engine stays
"reference" -- the two are different random streams and every shipped artifact drawn from this
function has its stream frozen. They agree in distribution: TRB V-usage correlation 0.99766 and
total variation 0.0156 against a 200,000-draw native sample, tightening as 1/sqrt(n_ref).

_align_init is what the second pass caught, and the largest thing here after the codon DP. EM
seeds gene usage from a per-read alignment vote before its first iteration -- nV + nJ germline
character walks per read -- and it runs before every fit, native path included: 436.9 -> 13.98 ms
for 20,000 human TRB reads. The votes still accumulate in read order, so the seeded tables are
bitwise-identical, and splitting a tied vote across germline-identical paralogs survives unchanged.

Two honest negatives, recorded so nobody re-derives them. model.analyze's group-wise
entropies were converted to aggregations and reverted: bit-identical output at 0.6x the speed.
A first call profiles entropy_table at 386 ms with 90% in polars group_by; the steady state is
8.7 ms and the 386 ms was query compilation. And collapse._rep removes 54,972 Python calls per
IGH model load
and moves the wall clock 0.90x-1.09x, because a model load is polars query
overhead rather than those calls -- kept as better code, not as a speedup.

Two loops must stay loops: io.read_vidjil parses JSON and sc.write_airr_cell emits YAML, so
the per-object Python is the work. Both are marked as such.

4.4.0 built every wheel and published none

Its own pre-publish thread-scaling guard demanded 2.5x at 4 threads; a 4-vCPU runner measures
2.25x, which is real parallelism at 56% efficiency. The guard is two bars now -- 1.8x at 4 threads
everywhere, 2.5x only where available_cores reports 8 or more -- and 4.4.0's library changes ship
here. 4.5.0 supersedes it; 4.4.0 is not on PyPI.

Full detail, with every measurement and the cohort shape it rests on, in
CHANGELOG.md.