[round · research-lab shape] Design the experiment that would falsify this paper

[akash-coded]

Simulated interview round. The shape a research-lab or applied-science panel runs, with the
rubric attached. Written for teaching; the failure modes are from this repository.

The prompt, as it would be given

"Here is a paper claiming a 4-point improvement in retrieval quality from a new chunking method.
The authors report it on three benchmarks. Design the experiment that would show they are
wrong."

What this shape is testing

Not whether you can critique a paper. Whether you distinguish a result from a claim, and
whether you know which parts of a result generalise.

1. Do you attack the precondition or the execution? Junior critiques find methodology
   errors. Senior critiques find the corpus property the result depends on.
2. Do you know that absolute numbers and directional findings falsify differently?
3. Can you say what would make you accept it? A critique that cannot be satisfied is not one.

The 45 minutes

Minutes	What happens
0–5	Clarifying. What does "wrong" mean — the number, the direction, or the mechanism?
5–20	Your experiment
20–35	Pressure on your control condition. This is where it is decided
35–42	"What result would make you believe them?"
42–45	Your questions

The trap

Most candidates design a replication — run the method on a new corpus, see whether 4 points
appears. Necessary, and not falsification: a null result is ambiguous between the method does not
work and this corpus does not have the failure the method fixes.

The experiment that falsifies isolates the mechanism. If the claim is that better chunk
boundaries improve retrieval, a corpus whose boundaries are already good must show no
improvement. If it shows 4 points anyway, the stated mechanism is wrong.

[akash-coded]

Marking this as the answer — the model response, then the rubric.

---

A strong answer, roughly as I would give it:

"First — what would 'wrong' mean here? The number, the direction, or the mechanism? Those need
different experiments and only the third is interesting.

Assume the mechanism. Their claim is that better chunk boundaries improve retrieval, which
implies a precondition: their corpus had bad boundaries. So the falsifying experiment is a
corpus where boundaries are already good — structurally chunked documents carrying their heading
path, each chunk readable in isolation.

If their method still shows 4 points there, it is not doing what they say. It may be doing
something valuable, but the stated mechanism is wrong, and every adoption decision made on that
mechanism is unjustified.

My control matters more than my treatment. I have to measure that my corpus has good
boundaries rather than assert it — sample twenty chunks, read them cold, check whether you can
tell what each is about. Ten minutes, and without it my null result is uninterpretable.

I would also run their method on a corpus deliberately given bad boundaries. If it shows a
larger gain there, that is strong positive evidence for their mechanism, and I would rather
find that than not.

What would make me believe them: a gain that scales with a measured badness-of-boundaries
statistic across several corpora. That is a much stronger claim than three benchmarks, and it is
the paper I wish they had written."

---

Why this scores

Signal	Weak	Strong
Framing	Designs a replication	Asks what "wrong" means, attacks the mechanism
Control	Treatment only	Names the control and how to verify it
Symmetry	Only looks for the null	Also designs the arm that would confirm them
Falsifiability	Cannot say what would convince them	Names the result that would
Humility	"The paper is flawed"	"The claim is broader than the evidence"

We ran this on ourselves

#37 measured contextual chunking on this corpus: worse on both quality
metrics at 2.4× storage. The reason is exactly the precondition above — structural chunking
already carried the heading path, so these chunks were never orphaned.

That is not a refutation of contextual chunking. It demonstrates the technique fixes a failure
we did not have, which is more useful and is what
The Precondition Test is built
from.