v2.2.0 — the counter handing out the work, and the error rate at every depth
Throughput
A sampling profile of assemble put 21% of all CPU samples, across every thread, on one instruction: the atomic fetch_add in parallel_for that claimed one item at a time. When an item is one read's tag scan, sixteen cores serialise on one cache line. Items are claimed in batches now, sized so each worker takes ~8 turns, collapsing to 1 when there are few items — the uneven case (one bucket per item) the counter exists for.
Two serial blocks went with it, both read-only 3L-binary-search scans of the barcode table that tally integers: the distance-1 census in estimate_umi_error, which is checkout's per-sample statistics tail, and refine's residual-FDR scan, 0.53 s of a 2.17 s run on one core after everything around it had been parallelised.
| stage | 1 thread | 16 threads | was, at 16 |
|---|---|---|---|
checkout |
213,880 | 1,548,835 | 1,056,472 |
refine |
617,802 | 1,554,156 | 1,012,368 |
assemble |
554,106 | 2,470,928 | 2,051,937 |
reads/s. checkout's gap between end-to-end and matching throughput fell from 20% to 9%.
The barcode error rate, measured at every depth
New table <sample>.umi_errors.tsv, one row per exact parent depth. A parent carrying c reads offered c·L barcode bases to be miscalled, so the same eps falls out two ways:
distinct children u(c) = 3L (1 - exp(-c eps / 3)) saturates at 3L
reads in children r(c) = c L eps no ceiling
Where they part company is where the library's barcode neighbourhood filled up, measured rather than predicted. On 1e-3 injected: distance-1 excess 9.73e-04, children 9.89e-04 = Q30, all depths 9.98e-04.
New fields error_at_depth, error_phred, error_from_children, error_depth, and two migec plot panels.
Neither estimator is saturation-free — both are bounded by the merges correction actually made. As a fraction of an injected truth:
| occupancy | 0.2% | 2.3% | 9.8% | 33% | 100% |
|---|---|---|---|---|---|
| distance-1 excess | 0.97 | 0.96 | 0.76 | 0.45 | 0.001 |
| from the children | 0.99 | 0.95 | 0.88 | 0.62 | 0.00 |
At 100% both are zero, because correct_umis refuses to merge on a full barcode space and is right to. The saturated flag is what says the answer is a floor.
A sparse spectrum is points, never a line
mig_size_spectrum drew reads in them with lines over a table at exact sizes, where past the head almost every size holds one molecule — so reads == size and the line drew the y = x diagonal as the figure's most prominent feature, a tautology that reads as a second mode.
Compatibility
No breaking change: every 2.1.0 command, flag and output file behaves as it did. The additions are one table, four JSON fields and two plot panels.
Full notes in CHANGELOG.md; the method is written up in docs/umi_errors.rst.