[negative result] Contextual chunking cost 2.4× storage and did not clear the band

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

What I tried

Contextual chunking — the Anthropic recipe. Prepend a generated sentence situating each chunk
in its parent document, then embed and BM25-index the augmented chunk. Notebook 01 introduces
it; I wanted to know whether to make it the default.

The result

Strategy      Chunks  Storage×  Index ms  Evidence recall  Full-chain
structural     2,430      1.35       183            0.807       0.686
contextual     2,430      2.35       210            0.767       0.671

Worse on both quality metrics, at 2.4× the storage and a per-reindex compute bill.

Why I think it did not help here

Two reasons, and I think the first is the real one.

1 · Our chunks do not lose their referent. Structural chunking already carries the document
title and the heading path into every chunk, so a chunk reading "Engineering will widen the
interval" already says which product and which incident it belongs to. Contextual chunking is a
fix for chunks that have been orphaned from their context — and ours have not been.

2 · The offline describe() is a template, not a model. It adds source, date and entity
names, most of which are already in the chunk. With a real model generating a genuinely
situating sentence the result would likely differ, and I have not tested that.

What I would tell a client

Contextual chunking is a real technique with a reported 49% reduction in failed retrievals on
Anthropic's corpus. It is a fix for a specific failure — chunks that are unretrievable in
isolation — and you should confirm you have that failure before paying for it.

The diagnostic is cheap: sample 20 chunks, read them cold, and ask whether you could tell what
they are about. If you can, your chunking is already carrying context and this will not help.

What this cost me

About three hours, and it is the submission I am most pleased with. The storage column is what
made the decision, and I would not have looked at it if the recall column had gone the other
way — which is itself worth knowing about myself.

[akash-coded]

Priya · cohort peer

Strong write-up, but I think the headline overstates what was tested. You say contextual chunking "did not clear the band" — what you tested was a template that concatenates metadata you already had in the chunk.

The Anthropic recipe generates a situating sentence with a model, using the full parent document as context. That is a different operation. Your section 2 acknowledges this, and then the title does not.

I do not think the result is wrong. I think it is scoped much more narrowly than the framing suggests.

[akash-coded]

the original poster · cohort

Accepting this. The honest title is "a template-based approximation of contextual chunking did not clear the band on a corpus whose chunks already carry their heading path", which is unwieldy but true.

What I would defend: the diagnostic in the write-up is the transferable part, and it does not depend on which implementation you use. Sample twenty chunks, read them cold, ask whether you can tell what they are about. If yes, you do not have the failure this technique fixes — and no implementation of it will help, model-generated or otherwise.

[akash-coded]

Dana · reviewer · owns the eval harness

Both right, and the resolution is a rule worth adopting for every negative result in this repo:

State the mechanism you tested, not the technique you were inspired by.

"Contextual chunking does not work" is unfalsifiable marketing in reverse. "Prepending already-present metadata to chunks that already carry their heading path does not improve recall on this corpus, at 2.4× storage" is a claim someone can check, disagree with, and reuse.

I have retitled nothing — the write-up should stand as posted with this thread attached, because the correction is the teaching content.

[akash-coded]

Marcus · cohort peer

On the cost side — is 2.4× storage actually a decision-grade number? Storage is the cheapest thing in the stack. On S3 that is fractions of a cent per GB-month. It feels like the write-up is treating a rounding error as the deciding factor.

[akash-coded]

Rafael · cohort peer · came from search infra

Storage is cheap. Storage multipliers are not, and this is the mistake I watched a team make for two quarters.

What 2.4× actually multiplies:

• The embedding backfill. 2.4× the text is 2.4× the tokens through the encoder. On a 40M-chunk corpus that is the difference between a weekend and most of a week on GPU, and you pay it again on every encoder upgrade.
• The index. Vector storage is unchanged (same chunk count, same dimensions), but the BM25 postings list grows with the augmented text, and that memory is on the serving path, not in S3. RAM at cluster prices, not object-store prices.
• The re-embed cycle. This is the one that actually hurts. A parent document changing now invalidates the generated context for every chunk in it, not just the chunk that changed. You have coupled chunks that were previously independent, so your incremental update path partially collapses into a rebuild path.
• Cache locality. 2.4× the bytes per chunk is fewer chunks per cache line and per page.

The storage bill is a rounding error. The dependency structure change it signals is not, and it is invisible in the storage column. That is what makes 2.4× decision-grade — not the dollars, the coupling.

[akash-coded]

Marcus · cohort peer

The re-embed coupling is the argument and I had completely missed it. Changing one paragraph invalidating an entire document's chunks is a genuinely different operational shape.

[akash-coded]

Dana · reviewer · owns the eval harness

Adding the thing the write-up does that I want everyone to copy, since it is easy to miss under the result.

The last section says:

"the storage column is what made the decision, and I would not have looked at it if the recall column had gone the other way — which is itself worth knowing about myself."

That sentence is the most valuable thing on this page. It is a pre-registration failure caught and reported by the person who committed it. The recall column going the other way would have made storage feel like a detail; storage was only decisive because the result was already negative.

The fix is procedural and it is now in the PR template: name the columns that can veto a change before you run it. If storage can veto, it can veto in both directions.