Repository navigation
v4.5.0 — every per-item loop, audited and measured
Every per-item loop in the package, audited and fixed where it mattered. All 84 modules read
twice for one pattern: Python iterating over data where a batched native call, a polars
expression, or a grouped computation does the same work. Equality or byte-identity is asserted
against the previous output in every case.
| site | what it was | measured |
|---|---|---|
infer_nt codon reconstruction |
Python per candidate, GIL held | 0.853 -> 0.086 ms/key TRB (9.89x), 94x TRA |
generate(engine="native") |
Python per sequence | 149x-577x; IGH productive_only 4,029 -> 2,325,649 seq/s |
infer._align_init |
Python per read, in the native path too | 14.4x-36.6x, bitwise-identical |
io.convert x9 readers |
a Python dict per input row | 5.32x on 42,877 rows, all 10 fixtures byte-identical |
sc.resolve_chains / sc.pair_chains |
a frame slice per cell | 517x / 540x at 20,000 cells |
model.score._pgen_nt_many |
a pool task per sequence | one native.pgen_nt_batch call |
pgen CLI, sc.paired_pgen |
a native call per row | one batched call per kind / per chain |
dynamics.expansion._exact_p |
betabinom.pmf per clonotype |
10.3x-307.5x, max rel diff 1.08e-14 |
model.data.write_prepared |
a Python string per clonotype | 4.6x, byte-identical |
biomarker.cooccurrence._dense |
a dict lookup per row | one replace_strict |
Two native entry points that did not exist. native.pgen_nt_batch was the one native Pgen with
no batch, so every caller with more than one sequence either looped in Python or wrapped that loop
in a ThreadPoolExecutor dispatching one task per sequence -- the only per-item pool in the
package. It is gone. _core.translate_junctions translates a whole junction column the way the
legacy converters do, including the bidirectional out-of-frame walk polars cannot express: 0
mismatches against this repo's own translation on 4,009 sequences at every frame offset.
generate(engine="native") is 244x on TRB and 577x on IGH productive_only, seeded from
(seed, row) so the output never depends on the thread count. The default engine stays
"reference" -- the two are different random streams and every shipped artifact drawn from this
function has its stream frozen. They agree in distribution: TRB V-usage correlation 0.99766 and
total variation 0.0156 against a 200,000-draw native sample, tightening as 1/sqrt(n_ref).
_align_init is what the second pass caught, and the largest thing here after the codon DP. EM
seeds gene usage from a per-read alignment vote before its first iteration -- nV + nJ germline
character walks per read -- and it runs before every fit, native path included: 436.9 -> 13.98 ms
for 20,000 human TRB reads. The votes still accumulate in read order, so the seeded tables are
bitwise-identical, and splitting a tied vote across germline-identical paralogs survives unchanged.
Two honest negatives, recorded so nobody re-derives them. model.analyze's group-wise
entropies were converted to aggregations and reverted: bit-identical output at 0.6x the speed.
A first call profiles entropy_table at 386 ms with 90% in polars group_by; the steady state is
8.7 ms and the 386 ms was query compilation. And collapse._rep removes 54,972 Python calls per
IGH model load and moves the wall clock 0.90x-1.09x, because a model load is polars query
overhead rather than those calls -- kept as better code, not as a speedup.
Two loops must stay loops: io.read_vidjil parses JSON and sc.write_airr_cell emits YAML, so
the per-object Python is the work. Both are marked as such.
4.4.0 built every wheel and published none
Its own pre-publish thread-scaling guard demanded 2.5x at 4 threads; a 4-vCPU runner measures
2.25x, which is real parallelism at 56% efficiency. The guard is two bars now -- 1.8x at 4 threads
everywhere, 2.5x only where available_cores reports 8 or more -- and 4.4.0's library changes ship
here. 4.5.0 supersedes it; 4.4.0 is not on PyPI.
Full detail, with every measurement and the cohort shape it rests on, in
CHANGELOG.md.