[negative result] HyDE cost 2 model calls per query and did not clear the band

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

What I tried

HyDE — generate a hypothetical answer to the query, embed that instead of the query, retrieve
against it. #9 proposes it; I wanted to know
whether to close that issue.

The result

                     evidence_recall   full_chain   latency p95   calls/query
baseline (hybrid)          0.807          0.686         310 ms         0
+ HyDE                     0.821          0.691       1,240 ms         1
delta                     +0.0140       +0.0050
95% CI                [-0.0071, +0.0356]  [-0.031, +0.041]
verdict            inside the band   inside the band

Four times the latency and a model call per query, for a delta whose interval crosses zero.

Why I think it did not help here

1 · The vocabulary gap it fixes is small on this corpus. HyDE helps when the query and the
documents use different language — a short colloquial question against formal prose. Our eval
questions are generated from the same fact graph as the documents, so they already share
vocabulary. The corpus construction removes the problem HyDE solves.

2 · The lexical leg was already covering it. Slicing by class, the only questions where HyDE
moved anything were inference ones, where the +0.031 was real in isolation but is one class out
of four and does not survive the multiple-comparison argument.

What I would tell a client

HyDE is a real technique with a real mechanism, and the mechanism is vocabulary mismatch
between queries and documents. Measure that gap before paying for it: sample 30 queries, look
at whether the terms people use appear in the documents at all. If they do, HyDE has nothing to
close.

What this cost me

Two hours, and I would rather have this result than the version where I ship it because the
point estimate was positive.

[akash-coded]

Marcus · cohort peer

Pushing back on closing #9 off this. Your generator for the hypothetical document is the
extractive one, isn't it? That cannot write a plausible hypothetical answer — it can only copy
sentences that are already in the corpus.

If so, you have not tested HyDE. You have tested "retrieve with a slightly different query made
of corpus text", which is nearly a no-op.

[akash-coded]

Priya · cohort peer

Checking the write-up — it says one model call per query, so it is not the extractive path.
Marcus's objection would be decisive if it were, though, and it is the first thing I would have
asked too.

[akash-coded]

Marcus · cohort peer

Fair, I misread the calls/query column. Withdrawn — though it is worth the write-up saying
which generator explicitly, because that is the first thing anyone will check.

[akash-coded]

Dana · reviewer · owns the eval harness

Endorsing the finding and the framing, with one addition that makes it more reusable.

The transferable form is a precondition, not a verdict. "HyDE did not help" is corpus-specific
and near-useless to the next reader. "HyDE closes a vocabulary gap between queries and
documents; measure the gap before paying for it" transfers to every corpus.

The diagnostic in the last section is the valuable part and should be promoted above the numbers:
sample 30 queries, check whether their terms appear in the documents at all. That is ten minutes
and it tells you whether the technique has anything to do.

Not closing #9. It stays open with this
thread linked and the precondition written into it, because on a corpus with a real vocabulary
gap the answer would very likely be different — and that is what the issue is now for.