Lost in the Middle (Liu et al., 2023) — is the U-curve still true, and does it matter?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

Discussion thread for the week 5 reading.

The paper: arXiv:2307.03172. Holding the evidence set
constant and moving only the position of the relevant document, accuracy traces a U — strongest
at the start or end of the context, weakest in the middle.

Three questions to argue about

1 · The paper is from 2023 and the models have changed. Does the finding still hold?

Worth being careful here in both directions. "Models got better so this is obsolete" is a
comfortable belief that would let us skip a measurement. But the effect size genuinely does
vary by model and task, and some of the specific results were on models nobody serves now.

The position the curriculum takes: this is a paper to replicate, not to cite. Notebook 05
§5.4 gives you the harness — force the gold chunk into position 1, mid and last with the
evidence set held constant, and report the spread. That is a one-run experiment against a real
model and almost nobody does it.

2 · If the effect is real, which mitigation is worth its cost?

The four the deck names, in cost order: keep k small (free, and saves money); order by score and
interleave edges (free, one line in the packer); restate the question after the evidence (~15
tokens); test it rather than assume it (one experiment).

Notice the first one is free and helps for a second reason — fewer chunks in the middle at
all. Worth asking whether the other three are worth doing if you have done the first properly.

3 · What does this imply about long-context models replacing retrieval?

The sharper version of the question. If accuracy degrades with position in a 4k context, what
happens in a 1M one? The deck's long-context matrix says accuracy "improves then plateaus or
declines past a model-specific length", which is careful phrasing for "nobody has a general
answer".

What I would like people to post

If you have access to a model, run the experiment on your own eval set and post the spread.
Three numbers and one sentence about your setup. That is a more useful contribution to this
thread than any amount of argument about the paper.

[akash-coded]

Priya · cohort peer

Taking question 1 — I ran the replication from §5.4 rather than arguing about it, since that is what the thread asks for.

Forced the gold chunk into position 1, middle, and last with k=8 and the evidence set held constant:

position    answer_correct    n
first             0.612      207
middle            0.541      207
last              0.588      207

Spread of 0.071 between best and worst, and the shape is a U — first > last > middle, which matches the paper's qualitative finding.

Two caveats I would not want anyone to skip. First, this is the extractive generator, so it is measuring the packer and the reader's position sensitivity, not a large model's. Second, k=8 is a short context; the paper's effect grows with context length and 8 chunks is at the tiny end. The honest reading is that the effect is present and small here, and that says very little about a 200k-token context.

[akash-coded]

Dana · reviewer · owns the eval harness

This is the right experiment and the caveats are the right caveats. One addition that changes how to read the number.

Run it with the ungrounded generator as a control. If position sensitivity shows up when the model has no retrieved context at all, then part of what you are measuring is not position — it is the packer changing which chunk gets truncated at the token cap, or the prompt-assembly order interacting with the extraction heuristic.

On a clean run that control should be flat. If it is not, subtract it before quoting 0.071.

[akash-coded]

Marking Priya's replication as the answer, because a measured spread on our own corpus is worth more here than any amount of argument about the paper, and because the caveats attached to it are more useful than the number.

Where I would land on the three questions.

1 · Does it still hold? Directionally yes, magnitude unknown and model-specific. Priya's 0.071 is real on this setup and does not license a claim about GPT-scale models on long contexts. The curriculum's position stands: replicate, do not cite. A paper that is cheap to re-run and often re-run wrongly is exactly the paper to make people re-run.

2 · Which mitigation is worth its cost? Keep k small, and largely stop there. It is free, it saves money, it reduces the middle you are worried about, and — from #28 — it protects context_precision, which tracks answer quality better than full-chain recall does past k≈16. The edge-interleaving in the packer is one line and harmless. Question restatement is where I would stop: ~15 tokens per query is not free at volume and I have not seen it clear a noise band on this corpus.

3 · Long context replacing retrieval? The question is usually posed as capability and it is actually economics. Even where a long-context model can attend across 500k tokens, you are paying to process 500k tokens on every query to answer something a 4k-token context would have answered — and prompt caching changes that arithmetic without eliminating it. Notebook 07 has the model.

The retrieval-shaped argument that survives: retrieval is not only about fitting in the window, it is about access control, freshness, attribution and cost. None of those four are solved by a longer context. A chunk you retrieved can be permission-filtered, cited, invalidated, and paid for once. A document you pasted into a prompt can be none of those things.