Why 40% of every worm was being thrown away, and what fixed it

[lhqing]

Resolves discussion #354. Landed as #356 → PR #357, measured in #358 → PR #359.

Roughly 40% of every worm in our first big Smart-seq3 run was being thrown in the bin by the aligner. It turned out not to be a data-quality problem, a sequencing problem, or a bug in our code. It was a measuring-tape problem. Here is the whole story, in plain language.

1. The symptom

We sequenced 784 C. elegans worms. When the alignment started coming back, fewer than 42% of reads were landing cleanly on the genome. The biggest single category wasn't "couldn't find where this goes" — it was a bucket STAR labels unmapped: too short, and it held about 38% of everything.

That is a strange thing to see, because three other numbers said the data was fine:

• The reads that did align, aligned beautifully — 0.32% mismatch per base. These were not degraded or error-riddled sequences.
• Repetitive junk wasn't eating them — only 0.03% were rejected for matching too many places.
• The loss was concentrated in that one bucket.

So we had reads that were clean, unique, and being discarded anyway. Something was wrong with the test, not the reads.

2. What "too short" actually means

This is the crux, and the name is genuinely misleading.

STAR has a rule: at least 66% of a read must match the genome, or the read is rejected. Note what that percentage is taken over — the whole read, including any part that isn't biological sequence at all.

An analogy. Imagine an exam with 150 questions, and you need 66% to pass. Unknown to you, 75 of those questions aren't real questions — they're printer's gibberish. You answer all 75 real questions perfectly. Your score: 75 out of 150 = 50%. You fail, having got everything right.

You can't fix that by studying harder. The only fix is to strike the gibberish off the paper before grading, so you're scored out of 75 instead of 150.

That is exactly what was happening. STAR was finding the worm DNA in these reads, aligning it correctly, and then failing them on a percentage computed partly over sequence that was never worm DNA in the first place.

3. Where the gibberish came from

Smart-seq3 uses an enzyme called Tn5 that chops DNA and staples a short synthetic tag onto each cut end. Tn5 cuts more or less at random, so you get fragments of all lengths — and whenever a fragment is shorter than the read length, the sequencer runs off the end of the real biology and just keeps going, reading straight into that synthetic tag and the sequencing machinery behind it.

A 150-letter read might be 75 letters of worm followed by 75 letters of adapter. Real biology, then gibberish. Nothing in our pipeline was removing it.

There's a nice irony here. We already knew about this sequence — it's in the chemistry's own recognition notes, where it's specifically flagged as something we must not use to identify Smart-seq3, because it shows up in 6.5–79.5% of this chemistry's reads and screening on it would reject the very libraries we're trying to detect. The exact abundance that makes it useless for recognising the chemistry is what makes it essential to process it. Two different questions, and we'd only answered one.

4. The fix, and one design argument worth writing down

Mechanically the fix is small: tell the aligner to cut everything from that sequence onward.

The interesting part was where that instruction belongs, and this thread and the issue reached opposite conclusions. #354 argued trimming is a per-run choice — you might trim, you might not, and choosing to trim shouldn't change the identity of the dataset.

We ended up going the other way, and wrote down why (ADR-0048). The dividing line we settled on is terminality:

• A trim is a choice. "Drop low-quality tails", "cut anything under 30 letters" — reasonable people disagree, different analyses want different answers. That's a per-run setting.
• A read-through is not a choice. Past the Tn5 tag there is no biology. Not low-quality biology — none. No analysis could rationally want it kept. It's a fact about how the molecule was built, like the barcode's position.

So it's recorded as part of the chemistry description, written down exactly once, and every pipeline works out its own aligner flag from it. That means it costs a recompile: any dataset already processed gets a fresh output directory and its old alignments aren't reused. We took that price deliberately — the reads genuinely are being processed differently now, and pretending otherwise is how you end up with two incompatible matrices that claim to be the same thing.

5. Does it actually work?

We didn't want to take the reasoning on trust, so we ran a controlled test — and deliberately on someone else's data, four published mouse cells (GSE207085), not our worms. Same input file, same aligner, same everything, one flag different:

	uniquely mapped	too short	mismatch rate
before	42.4%	54.4%	0.57%
after	63.6%	22.8%	0.54%

+21 points of usable data. Well over half the discarded pile recovered.

Three things make this convincing rather than just encouraging:

• We set a way to be wrong in advance. The prediction was that too short would shrink and unique mapping would rise. If the bucket had emptied without unique mapping going up, the diagnosis was wrong and we'd have said so. It went up in all four cells.
• The mismatch rate didn't budge. We're not lowering a bar to let rubbish through — the recovered reads align just as cleanly as the ones that always worked.
• It's dose-dependent. The cell that had the most adapter removed gained the most; the one with least gained least. That's the mechanism showing itself, not a coincidence.

And a bonus we didn't plan: the untouched arm of that experiment independently reproduced our own numbers. Different lab, different species, different tissue — unclipped, it sat at 42.4% uniquely mapped against our worm plate's 42.0%. Our run wasn't botched. This is just what unclipped Smart-seq3 looks like. The published literature agrees (Deserranno et al., BMC Genomics 2026, report 34% for a Smart-seq3 variant at stock settings).

6. What this doesn't fix

Being straight about the limits:

• Not every discarded read comes back. Of the reads freed from the bin, about 7 in 10 become uniquely mapped. Roughly 13% turn out to have been almost entirely adapter — strip it and there's a stub too short to place. Those were never recoverable; clipping just reveals that.
• Two of three suspects are still at large. ~40% of every aging_SS3 worm is discarded as "unmapped: too short", and the recipe has no trimming step #354 named Tn5 read-through, poly-A tails, and leftover oligo-dT. Only the first is fixed. Whatever the other two cost is still unmeasured.
• The worm plate itself hasn't been re-run. Its baseline (38% too short) is milder than these mouse cells' 54%, so expect a smaller absolute gain than the table above.

7. The practical bit

If you have a Smart-seq3 dataset compiled before this, re-compile it. You'll get a new output directory — that's expected, not a bug. The old matrices are built on roughly 40% less data than they could have been.

The 784-worm run was stopped at about two hours rather than spend another twenty producing matrices that threw away 39% of their input. That was the right call.

---

Tables, method, and the STAR flag details: docs/research/smartseq3-tn5-read-through.md. The ownership argument: ADR-0048.