Critique my answer: 'How would you separate a retrieval failure from a generation failure?'

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

Practising Q1-adjacent from docs/INTERVIEW-PREP.md. Here is my answer — please tear it
apart.

---

My answer:

"I'd look at whether the right documents were retrieved. If they were and the answer is still
wrong, it's a generation problem. If they weren't, it's a retrieval problem. I'd check the
logs to see what was retrieved for the failing queries and go from there."

---

Self-critique before anyone else does

Reading it back, I think it is correct and shallow. It describes the idea but does not
demonstrate that I have ever done it. Specifically:

• "The right documents" — I never say how I know which those are. That silently assumes gold
  evidence exists, which is the whole hard part.
• Two buckets, not four. It collapses "never retrieved" with "retrieved and dropped during
  packing", and those have completely different fixes.
• "Check the logs" assumes a trace exists. In most engagements it does not, and saying so is
  itself a signal.
• No mention of what I would do with the distribution of failures.

What I think a better version does

Names four verdicts rather than two, names the trace fields each needs, and ends with the
distribution being the work plan.

Would appreciate a critique of the rewrite as much as the original.

[akash-coded]

Your self-critique is better than most first answers, and it identifies the right three gaps.
Building on it:

The four verdicts, and the question that separates each pair:

1. Is any gold evidence in the packed context? No → retrieval fault, and do not touch the
   prompt. Saying that clause out loud is worth points, because reaching for the prompt first
   is the most common wrong instinct.
2. Was every gold chunk in the top-N pool? No → first-stage recall. Fix: chunking, encoder,
   fusion weights, ANN parameters, filters.
3. Did the packed context keep it, intact and attributed? No → ranking or packing. Fix:
   reranker, k, dedup, truncation.
4. Is the answer entailed by the packed evidence? No → generation. Fix: grounding
   instruction, abstention, citation contract, model.

If all four pass and the answer is still marked wrong — suspect the label, the question, or
your rubric. Ambiguous gold answers are the most under-reported source of "regressions", and
a candidate who volunteers that has clearly done this on real data.

Two things to add that would move you from good to strong:

First, the honest caveat about gold evidence. In a real engagement nobody hands you labels.
The answer to "how do I know which documents were right?" is that you build that set in week
one — sample 100 real failures, label the evidence by hand. That is notebook 02's whole
lesson and it is the part that makes the rest possible.

Second, end on the distribution rather than the procedure. "70% retrieval misses is a
chunking and hybrid problem; 70% grounding failures is a prompt and abstention problem. Those
are completely different four-week plans." That sentence is what tells a panel you have used
this to decide something rather than to diagnose one query.

One thing to drop: "check the logs". Say trace and name the fields — retrieved ids and
scores per stage, the packed context with provenance, the response, per-stage latency. If the
system does not have that, your first deliverable is the trace, and saying so is a stronger
answer than assuming it exists.

Your rewrite instinct is right. Post it and we will go again.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Marcus · cohort peer

The self-critique is already better than the answer. One thing I would add: I would open with the distribution, not the taxonomy. Panels have heard the taxonomy. Something like "I'd pull fifty failures and bucket them, because the shape of the distribution decides what I work on" gets to the point faster.

[akash-coded]

Priya · cohort peer

I would push back slightly. Leading with "I'd pull fifty failures" without saying what you are bucketing them into sounds like process for its own sake. The taxonomy is what makes the bucketing meaningful — it just needs to be two sentences, not a lecture.

Order it: the four verdicts, then the distribution, then what each bucket implies. That is thirty seconds and it does not sound rehearsed.

[akash-coded]

Rafael · cohort peer · came from search infra

Practical warning from having been on the other side of this panel: do not name four buckets unless you can name the trace field that distinguishes each pair. The follow-up is always "how would you tell those two apart", and a taxonomy you cannot operationalise is worse than two honest buckets.

If you say "retrieved but dropped during packing", be ready for "what in your logs tells you that", instantly, without thinking.

[akash-coded]

the original poster · cohort

That is the question I would have fallen over on. Writing the mapping out so I actually know it:

Verdict	The field that proves it
Retrieval miss	gold ∉ trace.candidates
Packing loss	gold ∈ candidates, gold ∉ packed
Generation failure	gold ∈ packed, answer_correct == 0
Right by accident	gold ∉ packed, answer_correct == 1

The fourth row is the one I would never have listed unprompted, and it is the one that matters most for whether the system works at all.

[akash-coded]

Marking this one as the answer because it is the version I would want to hear, and because the shape matters as much as the content.

---

A stronger answer, roughly as I would say it out loud:

"I'd separate them with the trace rather than by intuition, because the two-bucket version hides the case that actually matters.

For each failure I check three things: was the gold evidence in the candidate pool, did it survive into the packed context, and was the answer right. That gives four verdicts, not two.

Retrieval miss — never in the pool. Chunking, encoder, or the analyzer. Packing loss — in the pool, not in the context. That is k, the reranker, or a packing constraint, and it is the bucket people miss because retrieval metrics look healthy. Generation failure — evidence was right there and the answer is still wrong. Right by accident — the answer is correct and the evidence was never retrieved. That one is the most important, because it means the model is answering from parametric memory and the whole retrieval system is decorative for that query.

Then I'd look at the distribution over fifty or so failures, because that decides where the work goes. If it is mostly packing loss, widening retrieval is wasted effort — I have seen a team spend a month on the encoder for a problem that was k=5.

If there is no trace, that is the first finding. I'd say so on day one and instrument before promising a number, because without it every one of these verdicts is a guess."

---

Why this scores. Panels are not testing whether you know the word "reranker". They are testing three things:

1. Do you reach for evidence or for intuition? "I'd check the trace" versus "I'd look at whether the right documents were retrieved" — the second is a wish, the first is a procedure.
2. Do you know what you cannot see? Naming the missing trace as a finding rather than a blocker is the difference between a consultant and a candidate.
3. Have you actually done this? The "team spent a month on the encoder" line is doing real work. One concrete failure you have lived through beats any amount of taxonomy.

What still would not survive a hard follow-up: you have not said how you know what the gold evidence is for a production query with no labels. The honest answer is that you do not, and the first week is spent manufacturing an eval set — which is notebook 02 and the real work. Have that ready; it is the most common second question.