[capstone] Two of my four improvements were inside the noise band

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

What I built

The build brief end to end: harness first, baseline, three measured improvements, frozen-slice
check, decision record.

The result

Step	Change	Δ full-chain	95% CI	Verdict
1	baseline (fixed chunks, dense only, k=5)	—	—	—
2	+ weighted hybrid α=0.2	+0.2500	[+0.182, +0.318]	real
3	+ learned cross-encoder N=50	+0.0170	[−0.063, +0.097]	inside the noise band
4	+ routed query decomposition	+0.0000	[+0.000, +0.000]	inside the noise band

Noise band measured before any change: ±0.0614 on 207 dev questions.

What surprised me

Step 4 producing an exactly zero delta looked like a bug, so I dug in rather than reporting
it. The diagnostic:

of 60 routed questions:
  candidate pool changed        47
  packed context changed         1
  packed context gained gold     0

Decomposition genuinely widened the pool — 47 of 60 questions got different candidates, and 297
extra retrieval calls were issued. The reranker then put the same chunks on top anyway. The
extra candidates were real and none of them won a slot.

That is a finding about this corpus, not a bug: at N=100 the first-stage pool already contained
what the sub-questions went looking for. On a corpus where the second hop genuinely falls
outside the top 100, this lever is the one that recovers it — and I would not have known that
distinction if the delta had been a plausible +0.02 I could have shipped without looking.

The uncomfortable part

Step 3 is the change I was most confident about. It cleared the band on evidence recall
(+0.022, CI [−0.025, +0.068] — also inside, actually) and not on full-chain. My decision record
lists it under rejected, with "kept in the codebase and re-measured when the eval set
grows".

Writing that down was harder than any of the code.

What I would do next

Double the eval set before doing any more engineering. The noise band on a binary metric at 207
questions is ±0.06, and both of my rejected changes are smaller than that. Making those
decisions decidable costs less than either implementation did.

[akash-coded]

Dana · reviewer · owns the eval harness

The step-4 write-up is the reason to read this submission, and I want to make the general form explicit because it is worth more than the capstone.

An exactly zero delta is data, not a null. Most people see +0.0000 and report "no effect". You instrumented it and found that decomposition did change the candidate pool for 47 of 60 questions and changed the packed context for one. Those are completely different findings:

• "no effect" → the technique does not work
• "the pool changed and the ranking did not" → the technique works and is redundant with something already in the pipeline

The second is actionable and the first is not. On a corpus where N=100 does not already contain the second hop, decomposition would earn its keep — and you now know the diagnostic that would tell you.

[akash-coded]

the original poster · cohort

The thing that nearly stopped me digging was that +0.0000 looks like a bug, and my first instinct was to check whether the routing was firing at all. It was firing on all 60.

The counter I would give anyone repeating this: instrument the intermediate stage before you conclude anything about the final metric. candidate pool changed: 47 took ten minutes and turned a non-result into the most interesting row in the table.

[akash-coded]

Marcus · cohort peer

Question about step 2 rather than step 4 — +0.2500 with a CI of [+0.182, +0.318] is enormous. Larger than every other step combined, by a lot.

Is that not suspicious? A single fusion change nearly doubling full-chain recall makes me wonder whether the baseline was straw.

[akash-coded]

Rafael · cohort peer · came from search infra

Right to be suspicious of a huge delta, and in this case the baseline genuinely is weak — deliberately.

Step 1 is fixed chunking, dense only, k=5. Dense-only on this corpus is the weakest configuration available: the LSA encoder is intentionally modest, and the corpus has a lot of rare identifiers and entity names that lexical matching handles far better than a small dense model does. So step 2 is not really "fusion helped by 0.25" — it is "adding a lexical leg to a dense-only system helped by 0.25", which is a much more specific and much less surprising claim.

That is not a straw baseline, it is a starting baseline, and the brief asks for exactly that. But Marcus is right that the row is mislabelled: + weighted hybrid α=0.2 describes the change to the config, not the change to the system. + lexical leg (BM25), fused at α=0.2 is the honest row label, and it makes the size of the delta stop looking suspicious.

[akash-coded]

the original poster · cohort

Relabelling. That is a fair catch — the row named the parameter I changed rather than the capability I added, and those are not the same thing when the parameter is what switches the capability on.