My reranker improved evidence recall but full-chain recall is 'inside the noise band'. Do I ship it?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

paired_bootstrap says my change is "real" on evidence recall and "inside the noise band" on
full-chain recall. Which one do I believe?

What I have already tried

Re-ran both. Same verdicts. Checked I am comparing the same question set.

The numbers

evidence_recall     delta +0.0334  CI [+0.0101, +0.0584]  → real
full_chain_recall   delta +0.0338  CI [-0.0097, +0.0821]  → inside the noise band

The point estimates are nearly identical. Why do the verdicts differ?

Where

Notebook 04 §4.10

[akash-coded]

Both verdicts are correct, and understanding why is worth more than the answer to "should I
ship".

Why the intervals differ despite equal point estimates. evidence_recall is continuous —
a question can score 0, 0.5 or 1.0 depending on how many gold items arrived.
full_chain_recall is binary per question: 1 only if every gold item arrived, 0
otherwise. Binary outcomes have much higher variance per sample, so the same effect size needs
more questions to become distinguishable from zero.

You have not measured two different effects. You have measured one effect with two different
amounts of statistical power.

So do you ship it? On this evidence, yes — with the honest sentence attached:

"Ships on evidence recall, which cleared the band. Full-chain did not clear it on this eval
set, and that is stated rather than rounded away."

That is close to verbatim what the capstone's decision record generates, and it is the sentence
an interview panel is listening for.

Two things to do next, in this order:

1. Check the frozen slice. A gain that appears on dev and vanishes on frozen is
   overfitting, and that matters far more than which metric cleared the band.
2. Note the eval-set size in your record. With 207 questions the noise band on a binary
   metric is around ±0.06. Doubling the set is the cheapest way to make this decision decidable
   and it costs less than any of the engineering — that belongs in "revisit when".

The thing not to do is pick whichever metric cleared and report only that one. Everyone can
tell, and it is the fastest way to lose credibility in a review.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Priya · cohort peer

Sorry, I want to push on this, because I think the answer as written is one step away from something I would get pulled up on in review.

You ran two metrics. One cleared the band, one did not. You are shipping on the one that cleared. From the outside that is indistinguishable from picking the metric that agreed with you.

I do not think that is what happened here — but what stops it from being what happens next time?

[akash-coded]

That is the right challenge and it deserves a real answer rather than "trust me".

What stops it is that the metric was chosen before the number existed. evidence_recall is the primary metric for a retrieval change in this repo; that is written into the PR template and into .github/eval-baseline.json, both of which predate your branch. full_chain_recall is a secondary metric that is reported always, not consulted selectively.

The distinction that matters:

	Pre-registered	Reported
evidence_recall	primary, decision metric	✔
full_chain_recall	secondary, guard-rail	✔
everything else	not a decision metric	✔

If the verdicts had been reversed — full-chain cleared, evidence did not — the change would not ship, and that asymmetry is the entire reason the primary metric is named in advance.

Where your instinct is exactly right: if someone reports only the metric that cleared, that is the failure mode you are describing, and it is invisible in a diff. Which is why the template requires every metric in the baseline file to appear in the PR body, cleared or not.

[akash-coded]

Priya · cohort peer

That satisfies me. The pre-registration is doing the work, not the ordering of the numbers.

Suggestion then: the answer above should say "the primary metric cleared" rather than "evidence recall cleared". The first sentence carries the justification; the second one does not, and I read past it twice.

[akash-coded]

Fair. I have opened #13 against the README for the same reason — the wording implies a choice was made after seeing the numbers, when the choice was made in the template. Wording bug, not a methodology bug, but they look identical from outside.

[akash-coded]

Marcus · cohort peer

On the practical side — can you not just push the bootstrap harder? If the confidence interval on full-chain is [-0.0097, +0.0821], that is only just touching zero. Bump n_boot from its default of 2,000 up to 100,000 and it should tighten up enough to clear.

I have done this before on a different project and the intervals definitely got narrower.

[akash-coded]

Dana · reviewer · owns the eval harness

This is the single most common misunderstanding about the bootstrap, so it is worth being precise rather than just saying no.

There are two different sources of error and resamples only touch one of them.

1. Sampling variability — you measured 207 questions, not the population of all possible questions. This is what the confidence interval is reporting. It is a function of n (your 207) and the variance of the per-question differences. Resamples do not affect it at all.
2. Monte Carlo error — the bootstrap estimates those interval endpoints by resampling, and that estimate is itself noisy. This error shrinks as 1/√B.

Raising n_boot from 2,000 to 100,000 makes the endpoints you print more stable — run it twice and you will get the same numbers to more decimal places. It does not make the interval narrower, because the interval is not describing the bootstrap, it is describing the eval set.

Easy to confirm on your own machine, and worth doing once so it sticks:

from nanorag.metrics import paired_bootstrap

for n_boot in (2_000, 10_000, 100_000):
    r = paired_bootstrap(before, after, key="full_chain_recall", n_boot=n_boot, seed=11)
    lo, hi = r["ci"]
    print(f"n_boot={n_boot:>7,}  [{lo:+.4f}, {hi:+.4f}]  "
          f"width={hi - lo:.4f}  verdict={r['verdict']}")

The width will sit around 0.092 at every setting and verdict will stay inside the noise band. The digits past the third will stop moving. That is the whole effect.

Worth knowing while you are in there: paired_bootstrap uses percentile intervals — it sorts the resampled deltas and takes the 2.5th and 97.5th. Switching to BCa would correct for skew and bias in the bootstrap distribution and would move the endpoints — but not reliably toward narrower, and never enough to rescue an effect this size.

What you probably saw on the other project was either a different n, or a switch of interval method.

[akash-coded]

Marcus · cohort peer

Ran it. Widths 0.0918, 0.0918, 0.0917, verdict unchanged at every setting. You are right and I had this genuinely backwards — I had been treating n_boot as "more data".

For anyone else reading: n_boot is how carefully you measure the uncertainty. n is how much uncertainty there is. Only one of those is under your control after the eval set is fixed.

[akash-coded]

Dana · reviewer · owns the eval harness

Since the question underneath all this is "how many questions would I actually need", here is the arithmetic rather than a shrug. This is the part interviewers push on and most candidates hand-wave.

What you have. 207 questions, observed delta +0.0338, 95% CI half-width ≈0.046.

For the interval to just exclude zero, the half-width has to fall below the point estimate. Half-width scales as 1/√n, so:

n_needed ≈ 207 × (0.046 / 0.0338)²  ≈  207 × 1.85  ≈  383

But 383 is the optimistic answer, and you should not quote it. It assumes the true effect equals the one you measured, and a delta that only just misses significance is more likely than not an overestimate — the winner's curse. Budget roughly double:

≈ 700–800 questions for 80% power at this effect size

The thing worth internalising: on a binary metric like full-chain recall, per-question variance is p(1-p) ≈ 0.25 — about as bad as it gets, and it peaks exactly at the 0.5 accuracy where most interesting systems sit. evidence_recall is continuous over {0, 0.5, 1.0}, so its variance is lower and the same effect becomes visible sooner. You did not measure two effects. You measured one effect at two different levels of statistical power — which is what the answer above says, and the arithmetic is why.

Practical consequence: growing the eval set from 207 to ~800 is a day of corpus.py work. Every engineering option on the table costs more than that and moves the metric less. Grow the eval set first.

[akash-coded]

the original poster · cohort

Following up three weeks later, since this thread taught me more than the change did.

I did what Dana said and grew the frozen slice to 812 questions before touching the reranker again. Re-ran the identical change:

evidence_recall     delta +0.0281  CI [+0.0164, +0.0402]  → real
full_chain_recall   delta +0.0116  CI [-0.0038, +0.0271]  → inside the band

Two things I did not expect.

1. Both point estimates came down. +0.0338 → +0.0116 on full-chain. The winner's curse is not a theoretical caveat, it is most of my original effect.
2. Full-chain still has not cleared — but now I believe the "no" instead of suspecting the eval set. That is a completely different sentence to put in a decision record.

Shipped on evidence recall with the full-chain result stated. The reviewer's question was "how do you know 207 was too few" and I had an actual answer, which I would not have had before this thread.