How do I tell whether my golden set is actually measuring anything?

[akash-coded]

We built a golden set of 60 cases and our agent passes 98% of it. That number feels too good, and I do not trust it, but I cannot articulate why.

How do you tell whether a golden set is actually measuring anything?

[akash-coded]

Your instinct is right, and there is a specific test: how many cases did it fail on the day you froze it?

If the answer is "almost none", you have built a mirror. A set assembled by running the agent and labelling what it got right measures the agent against itself, and 98% is the arithmetic guaranteeing it.

Five properties of a set that measures something

1. Drawn from real inputs, sampled across time — not examples someone remembered or wrote for the demo.
2. Contains cases it currently fails. Aim for at least 15% failing at freeze. This is the one that separates a specification from a reflection.
3. Has an abstention slice where the correct answer is "I don't know" — ambiguous policy, unretrievable data, two defensible readings. If two of your experts disagree on a case, that is not a dispute to resolve; it is an abstention case, and the disagreement is the evidence.
4. Has an adversarial slice, including prompt injection delivered through retrieved content rather than user input. That is the realistic attack on a RAG-backed agent, and almost nobody tests it.
5. Frozen before the tuning it is used to evaluate. A set you adjusted while tuning measures your tuning.

The other thing 98% may be hiding

Check what the 2% consists of, and check it by slice. Two percent spread evenly is very different from 2% concentrated in your safety cases. That is why safety belongs as an absolute bar (= 1.0), never averaged into a headline number — a build at 98% overall with one policy-contradicting answer passes a naive gate, and that one failure is the entire reason the gate exists.

Also worth predicting: your abstention rate. Almost nobody sets a target, so nobody notices when the agent abstains never (dangerous) or always (useless). Classify 100 real inputs, count the ambiguous, out-of-scope and unretrievable ones, and that sum is roughly your correct abstention rate.

A quick reframe. State the claim with its evidence level attached: "98% on a 60-case set built from our own output" is honest and immediately tells you what it is worth. "98% accurate" is the same number doing much more work than it can support.

Related. Evidence Ladder · Abstention Budget · How to build a golden set from real tickets · EVAL-03 lab