Critique my answer: 'How would you evaluate a RAG system?'

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

Practising the most commonly asked question in this space. Here is my answer — please tear it
apart.

---

My answer:

"I'd build an eval set with questions and expected answers, then measure accuracy. I'd also track
retrieval metrics like recall and precision, and use an LLM to judge answer quality since exact
match is too strict. I'd run it in CI so we catch regressions."

---

Self-critique before anyone else does

It is correct, generic, and could have been written by someone who has never run one.

• "questions and expected answers" — no gold evidence, so I cannot separate a retrieval failure
  from a generation one, or detect a right-by-accident
• "measure accuracy" — against what baseline? I have not said what the number means
• "an LLM to judge" — no calibration, no κ, no base rate. I have added an unmeasured instrument
• "run it in CI" — on what threshold? A gate with no noise band blocks correct changes
• no interval anywhere, which is the thing this whole repository is about

[akash-coded]

Dana · reviewer · owns the eval harness

The self-critique is stronger than the answer, which is the right way round.

One thing you have not caught: "expected answers" quietly assumes the answers exist. On a real
engagement they do not, and the first two weeks are manufacturing them. A panel asking this is
often really asking "have you ever built an eval set from nothing?" — and the answer that scores
says so before being asked.

[akash-coded]

Priya · cohort peer

Also: "retrieval metrics" plural is doing a lot of hiding. "Recall and precision" collapses at
least three measurements over two different denominators. Say it in a round and you will be asked
which — and the answer had better not be improvised.

[akash-coded]

Marking this, because the shape matters as much as the content.

---

A stronger answer, roughly as I would say it out loud:

"It depends on whether they have labels, and usually they do not — so I'd start with the first
two weeks rather than describing a finished harness.

The eval set is the work. I need questions, and for each one the gold evidence, not just
the expected answer. Without evidence labels I cannot tell a retrieval failure from a generation
failure, and I cannot detect the case that matters most: the answer is right and nothing relevant
was retrieved. That means the model knew it already and retrieval is decorative for that query.

I'd include null questions at their real base rate, because without them I cannot measure
abstention at all, and a system that never refuses looks perfect on a set with nothing
unanswerable in it.

Then measure at each stage separately: did the evidence reach the pool, did it survive into the
prompt, did the model use it. Three booleans, four verdicts, and the distribution over fifty
failures decides where the month goes.

Before any of that means anything I need two baselines: the noise band, bootstrapped on the
unchanged system, and the ungrounded floor — what the model scores with no retrieved context
at all.

On the judge: I would not use one until I had calibrated it against a couple of hundred hand
labels and could report Cohen's κ with the base rate beside it. 94% agreement on a 90/10 split
is κ around 0.6; on 95/5 the same 94% is κ 0.37.

In CI the gate blocks on a pre-registered primary metric regressing beyond the noise band, and on
any slice regressing beyond its own band — a slice of 30 has a much wider interval than the
full set, and gating it with the aggregate band fails constantly on noise until somebody disables
the gate."

---

Why this scores

1. It leads with the constraint — no labels — rather than a finished system. The shape of the
   answer reveals whether you have done this before the content does.
2. It names the failure mode nobody names: right by accident, and the ungrounded floor that
   detects it.
3. It has numbers in it. κ 0.6 versus 0.37 on the same 94% cannot be improvised, and one
   concrete number does more work than three correct paragraphs.
4. It knows what a gate costs. "A gate that blocks correct changes gets disabled" is the
   sentence that says you have operated one.

What would still not survive a hard follow-up

You have not said how many questions. The honest answer is that it depends on the effect size
you need to detect, and the arithmetic is
here.
Have it ready — it is the most common second question.