[office hours · 2026-09-08] Eval sets, noise bands, and why your delta is not real

[akash-coded]

Office hours. One thread per session. Agenda goes up before, notes go up after, and the
thread stays open — so the answers are findable by whoever hits the same thing in three months.

When: Monday 8 September 2026 · async, all week. Post questions in this thread at any time.

Agenda

Three things, in order of how often they come up.

1 · "My change improved the metric — is it real?"
Walking through #29 end to end: why two
metrics can give opposite verdicts on the same change, why n_boot is not n, and the
arithmetic for how many eval questions a 2-point delta actually needs. Bring your own numbers.

2 · "How do I build an eval set when the client has no labels?"
The most common real blocker. Notebook 02 is the method;
EX-04
is the practice. The part people skip is null questions, and skipping them means you cannot
measure abstention at all.

3 · Open floor.

How to get a useful answer

Post the numbers, the config that produced them, and what you already tried. A
question with a measurement in it gets answered in one round trip.

Notes

Added after the session, in a reply to this thread.

[akash-coded]

Priya · cohort peer

Question for slot 1, posting early so it is in the queue.

I have a change that clears the band on evidence_recall and not on full_chain_recall, which
is exactly the #29 case. But my eval set is 96 questions, not 207. Does the ~380 figure from that
thread scale, or does something else break at that size?

[akash-coded]

Dana · reviewer · owns the eval harness

It scales, and something else does break — worth taking as the first question on the day.

The scaling is n × (half-width / point-estimate)², so run it with your own numbers rather than
borrowing 380.

What breaks at 96 is more interesting: the bootstrap itself becomes unreliable, not just
wide. Percentile intervals assume the resampled distribution is a decent stand-in for the
sampling distribution, and at n < ~100 on a binary metric that assumption gets thin — you are
resampling from very few distinct outcomes. The interval is not merely wide, its coverage is
off, so it may not be a true 95% interval at all.

Bring the raw per-question rows and we can look at the resampled distribution directly.