[negative result] Query decomposition changed 47 of 60 candidate pools and moved nothing

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

What I tried

Routed multi-hop questions through a decomposition step — split into sub-questions, retrieve for
each, union the pools — then measured.

The result

                    full_chain_recall   delta      95% CI              verdict
baseline                  0.686           —          —                  —
+ decomposition           0.686        +0.0000   [+0.000, +0.000]   inside the band

An exactly zero delta. Not small. Zero, with a zero-width interval.

Why I did not stop there

+0.0000 looks like a bug, so my first instinct was that the routing was not firing. I
instrumented it instead of assuming:

of 60 routed questions:
  routing fired                     60
  candidate pool changed            47
  extra retrieval calls issued     297
  packed context changed             1
  packed context gained gold         0

Decomposition worked. It changed the pool for 47 of 60 questions and issued 297 extra retrieval
calls. The reranker then put the same chunks on top anyway.

The finding

This is not "decomposition does not work". It is:

On a corpus where N=100 already contains what the sub-questions go looking for, decomposition
is redundant with the first stage. The extra candidates were real, and none of them won a slot.

The diagnostic that made this a finding instead of a null

full_chain_recall_at_N was already 0.871 at baseline. The pool was not the bottleneck —
packing was. Decomposition widens the pool, which is the stage that was already fine.

What I would tell a client

Before paying for decomposition, check full_chain_recall_at_N. If it is already high, the second
hop is in your pool and decomposition has nothing to find. The precondition is a pool that is
missing the second hop, and it is one number away from being checked.

[akash-coded]

Priya · cohort peer

The instrumentation is the part worth copying. +0.0000 is the kind of result people report as "no
effect" and move on from, and "no effect" and "worked and was redundant" are completely different
findings with different next actions.

Ten minutes of counting turned a null into a precondition.

[akash-coded]

Dana · reviewer · owns the eval harness

Endorsing, and naming the general form because it comes up constantly:

An exactly-zero delta with a zero-width interval is not a null result. It is a signal that your
change did not reach the metric. Either it did not fire, or it fired and was absorbed by a later
stage. Both are worth knowing and neither is "no effect".

The instrumentation that distinguishes them is always counting at the intermediate stage, and it is
always cheaper than the debate about whether to keep the feature.

[akash-coded]

Rafael · cohort peer · came from search infra

One caveat for anyone reusing this: the result is conditional on N=100 and on this corpus's
fact-graph construction, where questions and documents share vocabulary by design.

On a corpus with real vocabulary mismatch, sub-questions phrase things differently and would
plausibly reach documents the original query does not. That is the case where decomposition earns
its keep, and this corpus structurally cannot show it.

Worth stating in the write-up, because someone will otherwise cite this as evidence against
decomposition in general.

[akash-coded]

Correct, and added to the finding. It is the same limitation #37 has and
the same one the whole synthetic corpus has — which is precisely what
the extension project exists to test.