My RAG answer cites a passage that does not support it. What are we measuring wrong?

[akash-coded]

Our RAG agent produces answers with citations, and reviewers were happy — until someone opened one of the cited passages and it did not support the claim at all. The passage was real and retrieved. It just was not the evidence for that sentence.

The retrieval metrics look fine. What are we actually measuring wrong?

[akash-coded]

You have citation theatre, and your metrics cannot see it because they measure a different property than the one you care about.

There are three separate things here, and most systems build one and claim all three:

	Property	The check
①	Retrieved — relevant text reached the context	Log the passages; a human reads them
②	Cited — the answer names a passage	Assert a non-empty citation on factual claims
③	Verified — the cited passage supports the claim	Entailment check, or a human sample

Your retrieval metrics measure ①. Your reviewers checked ②. Nobody checked ③ — and ③ is the only one that is actually grounding.

Why models produce this. Models are good at naming a plausible passage for an answer they generated independently. Nothing in the generation step forces the answer to come from the passage; it only has to look consistent with it. The result passes eyeballing and fails entailment.

The cheapest useful fix: a human spot check. Pick five cited answers per release, open each cited passage, and ask "does this, alone, support the claim?" Thirty minutes. If you score below five out of five, you have citation theatre and no amount of retrieval tuning will address it — the problem is at generation.

The detection heuristic worth automating. Genuine grounding for a specific claim is usually one or two passages. Answers that cite three or more for a single specific claim are disproportionately synthesised — the model produced an answer "informed by" passages none of which actually contain it. Flag high-citation-count answers for review; it is a one-line filter.

The structural fix. A citation has to be resolvable. If your context builder returns a formatted string, the mapping from [1] back to fare-rules#7.3 exists nowhere, and no automated check is possible at any budget. Return the mapping alongside the context:

{"context": "[1] Involuntary cancellation …",
 "citations": {1: "fare-rules#7.3"}}

With that in hand, an entailment check is about fifteen lines.

Related. Grounding Triangle is this answer in full. RET-05 is a graded lab that builds the resolvable version, with a Break phase that runs a verifier over your output.