[poll] Your change clears the band on one metric and not the other. Do you ship it?

[akash-coded]

Calibration poll. Vote before reading #29,
then read it. The gap between what people vote and what the measurement says is the teaching
content — that is the whole point of asking first.

The situation

You have changed the reranker. Same eval set, same config, 207 questions:

evidence_recall     delta +0.0334   CI [+0.0101, +0.0584]   -> real
full_chain_recall   delta +0.0338   CI [-0.0097, +0.0821]   -> inside the noise band

The point estimates are nearly identical. The verdicts are not.

What do you do?

• Ship it. The primary metric cleared; report both.
• Do not ship. Two metrics, one failed, that is a no.
• Ship, but only after growing the eval set and re-measuring.
• Cannot decide without knowing which metric was named first.

After you vote

Read the thread. Then ask yourself whether your answer would have changed if the verdicts had
been swapped — full-chain cleared, evidence did not. If it would, you were reasoning about the
numbers. If it would not, you were reasoning about the process, and the process is the answer.

[akash-coded]

Dana · reviewer · owns the eval harness

Adding the thing the poll is really testing, for anyone reading after voting.

Three of the four options are defensible. The one that is not is "do not ship" on the
grounds that one metric failed — because that treats inside the noise band as equivalent to
regressed, and it is not. It means not measurable at this eval-set size, which is a
statement about your instrument, not about your change.

The option most people skip is the fourth. If you do not know which metric was named as primary
before the numbers existed, you cannot tell shipping-on-the-primary-metric from
picking-the-one-that-agreed-with-you — and neither can a reviewer.