Why is answer_correct so low on temporal questions when retrieval looks fine?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

Temporal questions show evidence recall 0.807 but answer correctness 0.061. Is retrieval
lying to me?

What I have already tried

Spot-checked five temporal questions — the gold evidence really is in the packed context.

Where

Notebook 02 §2.2

[akash-coded]

Retrieval is not lying. You have found the seam between the retrieval lane and the answer
lane, and it is a good thing you looked.

The offline reader is extractive. It selects and cites supporting sentences from the
packed evidence. It cannot derive anything. A temporal question like "Did growth accelerate
or slow between Q2 and Q3?" needs the reader to compare two numbers and emit a word —
"accelerated" — that appears in neither chunk. The evidence is right there; the reader has no
mechanism to combine it.

This is stated in notebook 00's honesty inventory and it is deliberate: an extractive reader is
faithful by construction, which lets the whole retrieval half of the curriculum be measured
without an LLM adding its own variance to every number.

Why this is a feature for teaching. It is a clean demonstration of the deck's central claim
that an end-to-end score never tells you which stage moved. Run the fault-isolation tree on one
of these questions: Q1 yes, Q2 yes, Q3 yes, Q4 — is the answer entailed by the evidence? — and
you land on generation fault, correctly, with no ambiguity.

What to do about it:

• If you want the answer lane to work, set up Bedrock or the Claude API. BedrockGenerator
  drops into the same interface and the retrieval numbers will not move at all — which is
  itself the demonstration.
• If you are working on retrieval, report the retrieval lane and say why the answer lane is
  low. That is not a workaround; separating the lanes is the point.

The transferable version: when one lane moves and the others do not, you have located your
problem. A team watching only end-to-end quality would see 0.061 here and start tuning the
retriever, which is already doing its job.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Marcus · cohort peer

My first instinct here was that answer_correct is just a bad metric. It is string matching — if the model says "March 2024" and the gold says "2024-03", the token overlap check fails and you score a zero on an answer that is correct.

Is this not simply a scoring artefact rather than a real failure?

[akash-coded]

Dana · reviewer · owns the eval harness

Worth checking rather than assuming, because it is a real failure mode of that metric — just not this one. Here is how to rule it out in about a minute, which is the habit worth taking from this more than the answer.

from nanorag.metrics import answer_correct

for r in rows:
    if r["question_type"] == "temporal" and r["answer_correct"] == 0.0:
        print(f"gold: {r['gold_answer']!r}")
        print(f"got : {r['answer'][:120]!r}")
        print(f"  exact-mode: {answer_correct(r['answer'], r['gold_answer'], mode='exact')}")
        print()

Read twenty of those. If the answers look right and only the formatting differs, you have found a scoring artefact and you should fix the metric.

On this corpus they do not look right. The answers are fluent, on-topic, drawn from the correct documents — and they do not contain the derived fact the question asks for. Scoring them by hand gives roughly the same number the metric does.

For completeness on the metric itself, since your objection is a good one and the answer is specific: answer_correct(mode="contains") normalises both strings, splits gold on non-alphanumerics, keeps tokens longer than two characters, takes the first six, and requires 60% of them to appear. For a gold answer of "2024-03-15" the surviving token is ["2024"] — "03" and "15" are too short. So "March 2024" does score 1.0. The metric is more forgiving than you assumed. That is why it is not the explanation here.

[akash-coded]

Marcus · cohort peer

Checked twenty by hand as suggested. Fourteen are fluent sentences about the right entities that never state a date relation at all. You are right, and my objection was the comfortable answer rather than the true one.

[akash-coded]

Rafael · cohort peer · came from search infra

If retrieval is fine and the metric is fine, what is left is the generator — and I think the answer is that we are asking an extractive reader to do arithmetic.

ExtractiveGenerator is the default. It scores sentences in the packed context and returns the best one verbatim. A temporal question like "which incident was resolved first" has an answer that appears in no single sentence — it has to be derived by comparing two dates in two different chunks.

An extractive reader cannot produce a fact that is not literally present. So retrieval succeeds, packing succeeds, and generation cannot possibly succeed. Is that the whole story?

[akash-coded]

That is the whole story, and you have arrived at the most important distinction in the curriculum, so let me put a frame around it.

The four verdicts. "Retrieval problem or generation problem" is the two-bucket version people give in interviews. The useful version has four, because each has a different owner and a different fix:

Verdict	Evidence in pool?	Survived packing?	Answer correct?	Fix
Retrieval miss	✗	—	✗	chunking, encoder, N, fusion
Packing loss	✓	✗	✗	k, reranker, packing constraints
Generation failure	✓	✓	✗	the generator, or the prompt
Right by accident	✗	✗	✓	the scariest one — model knew it without evidence

Temporal questions on this corpus are squarely in row three. evidence_recall 0.807 and full_chain_recall healthy say rows one and two are fine. answer_correct 0.061 says row three.

Why the repo ships this way rather than fixing it. Swap in AnthropicGenerator or BedrockGenerator and temporal correctness jumps immediately — a real model does the comparison. That is a one-line change and it is documented in notebook 05.

It is not the default because the default has to run with no API key, and a curriculum whose first cell requires a credential is a curriculum most people never run. The cost of that choice is exactly the confusion in this thread: a metric that looks like a retrieval failure and is not.

Which is the lesson, not the bug. Row four is the reason to care. If you only measure answer_correct, an ungrounded model that happens to know the answer scores identically to a working RAG system. UngroundedGenerator exists in generate.py purely so you can measure that gap — run it and you will see a non-zero score with no retrieval at all. That number is your "right by accident" floor, and every answer_correct you report should be read against it.

[akash-coded]

Rafael · cohort peer · came from search infra

The UngroundedGenerator baseline is the part I had not thought to run. Measuring what the model gets right with no context at all, and subtracting, is obviously the correct thing to do and I have never once done it.