Sufficiency check: a cheap model call, or a trained classifier?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The problem

The abstention issue in the tracker establishes that no retrieval-score threshold works. The
deck prescribes a "separate, cheap model call with a strict schema". Before building that, I
want to argue about whether a trained classifier would be better.

Constraints

• Runs on every query in the escalate-by-default design, so it is inside the latency budget
• Must not itself become a component nobody can evaluate
• Has to work offline in this repo, with a documented path to the model version

Option A — a cheap model call

prompt: question + packed evidence → {"sufficient": bool, "missing": [str]}

For: understands entailment, which is the actual question. Handles novel phrasings.
No training data needed.
Against: 200–500 ms and real money on every query. A second model to version, pin and
calibrate. Its failure mode is confident agreement, which is the same failure mode we are
trying to fix.

Option B — a trained classifier over pair features

Features like the reranker's, plus: does the top evidence contain an entity of the type the
question asks for; does any single sentence cover the question's rare terms; margin between
top-1 and top-5 scores.

For: microseconds, free, deterministic. Trainable on the existing null set.
Against: needs labelled data we would have to keep producing. Almost certainly learns
surface features rather than entailment — and the notebook already showed four surface signals
sitting near chance. It would also be trained on our nulls, which are synthetic.

My current view

Option A, but not on every query. Run it only when a cheap gate fires — top score below a
permissive threshold, or the question's rare terms not co-occurring in any single packed chunk.
That keeps the model call off the majority of traffic while catching the cases where it is
needed.

That is the same escalation shape as single-shot-then-loop, which makes me think it is the
right structure rather than a hedge.

What my own design costs

• Two components instead of one, and the cheap gate has its own precision/recall to
  maintain
• The gate's false negatives are invisible — a null question that sails past the gate never
  reaches the model call, and nothing flags it
• Offline, the model call has to be a stand-in, which means the repo can demonstrate the
  architecture but not the quality

What I would measure first

The cheap gate's recall on the null set, alone. If it cannot catch 80% of nulls at a tolerable
over-refusal rate, the escalation structure does not work and it is Option A on every query.

What I want challenged

Is "gate then model" a real design or am I optimising a cost that has not been measured? I have
not priced Option A on every query against the actual traffic shape.

[akash-coded]

Rafael · cohort peer · came from search infra

Option B is dead on arrival for a reason the write-up nearly says and then does not commit to.

You note the classifier "would be trained on our nulls, which are synthetic". That is not a caveat, it is the whole result. Our null questions are generated by removing a fact from the graph — so the surface signature of a null in this corpus is an artefact of the generator, not of insufficiency. A classifier will learn the generator, score beautifully in-distribution, and transfer to nothing.

The notebook already shows four surface signals near chance. That is the experiment, and it came out negative.

[akash-coded]

Priya · cohort peer

Agreed on the conclusion, but I want to separate two claims because I think only one of them holds.

"Trained on synthetic nulls" is a data problem — fixable with real labels, which any real engagement will accumulate anyway.

"Learns surface features rather than entailment" is an architecture problem — not fixable by more data of the same kind, because the features on offer genuinely do not contain the signal.

The first is a reason to defer B. The second is a reason to reject it. Worth being clear which one you are asserting, because they imply different roadmaps.

[akash-coded]

Rafael · cohort peer · came from search infra

Fair distinction, and I am asserting the second. Score margin, entity-type match and rare-term coverage are all retrieval confidence proxies. Sufficiency is a question about whether the evidence entails the answer, and entailment is not a monotone function of retrieval confidence — a perfectly retrieved passage can be adjacent to the question and not answer it. No amount of real labels adds a feature that is not there.

[akash-coded]

Marcus · cohort peer

Is there not a third option nobody has costed? Run the model call, but not on every query.

The write-up says "runs on every query in the escalate-by-default design" and then prices Option A against that. But you could gate it: only invoke the sufficiency check when a cheap signal says the query is near the boundary. High retrieval confidence and high margin, skip it. Everything else, pay the 300 ms.

That does not need the classifier to be correct. It needs it to be correct on the easy cases, which is a much weaker requirement.

[akash-coded]

Dana · reviewer · owns the eval harness

This is the right idea and it has a name — a cascade, and it is how essentially every production system in this shape actually works. Worth being precise about what it costs, because the trap is subtle.

The requirement is not "correct on easy cases", it is high precision on the skip decision. Every query you skip is one where you have decided, without checking, that the evidence is sufficient. A false skip is a confident wrong answer to a handler on a phone call — the exact failure the whole design exists to prevent.

So the cheap gate needs a threshold set for very high precision on the skip class, and you accept low recall: skip only when you are extremely confident, pay for everything else. On our data that would mean skipping perhaps the top 20–30% of queries by margin, not 80%.

That still removes a quarter of the cost for a bounded risk, and it is measurable — the abstention machinery in metrics.abstention_scores is the right harness. But notice what happened: you have reintroduced the classifier from Option B, with a much weaker job. Rafael's objection does not apply, because you are no longer asking it to detect insufficiency, only to detect obvious sufficiency.

[akash-coded]

Marking this as the answer, because the cascade is the design that comes out of the thread and neither original option is it.

Decision: Option A, with Marcus's gate, staged.

Phase 1 — model call on every query. Ship the thing that works. Pay the 300 ms and the money. Do not optimise a component whose value you have not yet measured.

Phase 2 — measure what the check actually catches. Log every invocation with the retrieval margin. After a few thousand queries you have the joint distribution of margin against the verdict, which is the data the gate needs and which nobody has up front.

Phase 3 — add the gate, precision-first. Set the skip threshold for ≥99% precision on the skip class, measured on held-out data. Report the skip rate; expect 20–30%, not 80%. Re-tune on a schedule, because the distribution moves.

What made Option B wrong as originally posed, and it is the transferable lesson: it asked a surface-feature classifier to answer an entailment question. Rafael's argument is architectural and no quantity of labels fixes it. What made the cascade right is that it asks the same classifier a much easier question — not "is this evidence sufficient" but "is this case obviously easy" — and that question is a monotone function of retrieval confidence.

Reframing a hard question into an easy adjacent one is most of applied ML, and it is worth noticing that it happened here.