Why does Recall@N go up but full-chain recall stay flat?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

I widened n_candidates from 100 to 400 and Recall@N improved, but full_chain_recall barely
moved. Why?

What I have already tried

• Confirmed the extra candidates are real — the pool genuinely contains more gold chunks
• Checked it is not a caching artefact by restarting the kernel
• Sliced by question_type: the flatness is worst on comparison and temporal

The numbers

N=100   Recall@N 0.938   full-chain 0.469
N=400   Recall@N 0.974   full-chain 0.481

Where

Notebook 01 §1.3

[akash-coded]

This is the most useful confusion in the whole curriculum, so it is worth answering at length.

Recall@N and full_chain_recall measure different stages. Recall@N is about the
candidate pool: did stage one find the evidence at all? full_chain_recall is about the
packed context: did every gold item for a question survive into the k chunks the model
actually reads?

Widening N only affects the first. The second is bounded by k.

Concretely: a two-hop comparison question needs both entities' evidence in the packed context.
At k=8, with a global relevance ranking, it is entirely possible for six of your eight slots
to go to one entity. Adding 300 more candidates does not create a ninth slot.

How to confirm it on your own run — this is the diagnostic worth learning:

for r in rows:
    if r["full_chain_recall_at_N"] == 1.0 and r["full_chain_recall"] == 0.0:
        print(r["qid"], r["question_type"])   # found it, then dropped it

If that list is long, your bottleneck is stage three, not stage one. Notebook 01 §1.3 measures
the two lines separately for exactly this reason: the gap between them is precision work,
and no amount of widening N closes it.

What actually moves it:

1. Raise k — the blunt fix; costs tokens and context precision
2. A better reranker — reorder so both hops land in the top 8
3. A packing constraint — notebook 02 §2.2 reserves slots per entity for comparison
   questions and moves full-chain recall +0.065 with the same candidates and the same k

Option 3 is the interesting one. A global relevance ranking cannot express a constraint about
coverage, so the constraint belongs in the packer rather than being wished for in the
reranker.

The general form of this is worth carrying: when a metric at stage n improves and a metric
at stage n+1 does not, the bottleneck moved. That is not a disappointing result — it is the
measurement doing its job.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Marcus · cohort peer

Reading this, the fix seems obvious though? If k=8 is the binding constraint, set k=20. You said yourself that widening N does not create a ninth slot — so create twelve more slots.

What am I missing that makes this not the first thing to try?

[akash-coded]

Dana · reviewer · owns the eval harness

You are not missing anything about the mechanism — raising k genuinely does move full-chain recall. What you are missing is the second column.

Measured on the same 207 questions, structural chunking, same reranker:

 k   full_chain_recall   context_precision   prompt tokens   $/1k queries
 4         0.361              0.71               2,180          $ 6.55
 8         0.469              0.52               4,090          $12.27
16         0.548              0.31               7,910          $23.73
32         0.591              0.17              15,400          $46.20

Full-chain recall is still climbing at k=32. So is the bill, faster. And context_precision — the share of packed chunks that are actually gold — has fallen through the floor: at k=32 roughly five of every six chunks the model reads are irrelevant.

That matters beyond cost, because answer_correct does not track full-chain recall up this curve. It peaks around k=12–16 on this corpus and then declines, which is the "lost in the middle" effect #38 is about. You can retrieve the evidence and still lose the answer by burying it.

So raising k is the correct first experiment and the wrong place to stop. It is the blunt instrument the answer above lists first for exactly that reason.

[akash-coded]

Marcus · cohort peer

That is a much better table than the one in my head. The context_precision column is the one I would never have looked at — I was treating "did the gold arrive" as the only question.

So the packing constraint from notebook 02 buys the same recall at k=8 that raising k buys at ~16, without the precision collapse. That is the actual argument.

[akash-coded]

Rafael · cohort peer · came from search infra

A question about the arithmetic rather than the fix, because I want to check I understand what full-chain recall is.

If per-passage recall is 0.938 at N=100, and a two-hop question needs both passages, should full-chain not be about 0.938² = 0.880? The measured number is 0.469. That is not a small discrepancy, it is off by a factor of nearly two.

Either I have the wrong model of the metric or something else is going on.

[akash-coded]

Your model is right; it is being applied at the wrong stage. This is worth spelling out because it is the single most common thing people get wrong about multi-hop retrieval, and it comes up in interviews constantly.

0.938 is Recall@N — the candidate pool of 100. 0.469 is full-chain recall over the packed context of k=8. Different denominators, different stages. Compare like with like and the picture is different:

full_chain_recall_at_N   (over the N=100 pool)   0.871
full_chain_recall        (over the k=8 packed)   0.469

0.871 against your predicted 0.880 — the conjunction model is essentially right at that stage. The entire gap from 0.871 to 0.469 happens during packing. That is the number that says where the work is.

Now the more interesting half of your question: why is 0.871 slightly above 0.938² = 0.880? It is not — it is slightly below, but only just, and both are much higher than independence would give across the whole set, because independence is the wrong assumption.

Two gold passages for one question are frequently in the same document, or in documents that share entities and therefore co-retrieve. Retrieving one makes retrieving the other more likely, not independent of it. Formally, P(A∧B) = P(A)·P(B|A) and P(B|A) > P(B) here. Positive correlation lifts the conjunction above the independence floor.

So the independence estimate is a lower bound worth computing, because:

• if measured ≈ predicted, your hops are genuinely independent — usually means evidence is scattered and packing has to work hard;
• if measured ≫ predicted, your hops co-occur — often means your chunking has split one coherent passage in two, which is worth knowing;
• if measured ≪ predicted, something is actively suppressing the second hop. Dedup by document is the usual culprit — a diversity constraint that caps chunks per document will systematically drop the second hop of a same-document pair.

That last one has bitten this repo. It is why pack_context takes the constraint as a parameter rather than hard-coding it.

[akash-coded]

Rafael · cohort peer · came from search infra

full_chain_recall_at_N being a separate field is the bit I had not registered — I was comparing the wrong two numbers.

The dedup-by-document failure mode is a good one. On my previous job we had exactly that and diagnosed it for a fortnight as an embedding problem.

[akash-coded]

the original poster · cohort

Closing the loop. Ran the diagnostic from the accepted answer:

found-then-dropped: 84 of 207 questions
  comparison   41
  temporal     22
  inference    17
  null          4

84 questions where the evidence was in the pool and did not survive packing — against 27 where it was never retrieved at all. My bottleneck was three-to-one on the stage I was not working on.

Implemented the per-entity slot reservation from notebook 02 §2.2 for comparison questions only:

full_chain_recall   0.469 → 0.531   CI [+0.021, +0.104]   real
context_precision   0.52  → 0.51    (unchanged)
prompt tokens       4,090 → 4,140   (+1.2%)

Same k, same candidates, +0.062 for 1.2% more tokens. Raising k to 16 bought +0.079 for 93% more tokens and wrecked precision. Not close.