Does a contradiction detector know when it cannot tell?
A measurement over ManConCorpus, the standard benchmark for contradictory claims in biomedical literature. The result is negative in both directions, which is why it is worth reporting.
Automated contradiction detection is normally posed as a classification: given two findings, do they conflict? But two published findings can differ because the populations differ, the dose differs, the instrument differs, or the definition differs — and none of those is a contradiction. Deciding conflict therefore requires first establishing comparability: that the two findings were ever about the same thing.
Benchmarks in this area supply comparability by construction. ManConCorpus groups claims under an expert-written PICO question, and its annotators read the whole abstracts. What the corpus ships, and what downstream systems consume, is the single claim sentence.
So the question measured here is narrower than "is this a contradiction":
Reading only the claim sentences, is there enough stated to establish that the two findings concern the same conditions at all?
1. For 77.1% of pairs, the text cannot support any verdict.
Each claim sentence was annotated for six comparability axes, with a hard rule that an axis the sentence does not state is recorded as NOT STATED rather than inferred. Across 259 claims:
| axis | stated |
|---|---|
| outcome measure | 98.5% |
| intervention | 76.4% |
| population | 69.9% |
| conditions | 32.0% |
| methodology | 13.5% |
| measurement | 8.1% |
Of the 728 opposed pairs, 561 (77.1%) never state population, intervention or outcome measure on at least one side. Population is the biggest hole, missing in 55.9% of pairs.
2. A deterministic comparability check returns nothing at all.
Requiring the axes to match before permitting a contradiction verdict yields
0 contradictions out of 728 — 720 different_condition, 8
insufficient_information. Recall 0, precision undefined.
That 98.9% is not a discovery about the corpus. It is string equality failing, and the sharpest example is unarguable:
[YS] population='patients with HCM' outcome='adverse outcome'
[NO] population='HCM patients' outcome='adverse prognosis'
→ "the claims differ on population, outcome_measure"
A softer matcher does not rescue it. Median token overlap between the stated axes of an opposed pair is 0.000, and accepting any shared token whatsoever across all three axes recovers 4 pairs out of 728. The vocabularies do not overlap; the missing component is concept mapping, not normalisation.
3. Language models assert contradiction MORE often when the text says less.
Two local models judged all 728 pairs with three allowed answers — CONTRADICTION, NO_CONTRADICTION, NOT_ENOUGH_INFO — given exactly what the deterministic layer was given. 1,456 judgements, zero unparsed.
| qwen3:14b | qwen3:8b | |
|---|---|---|
| CONTRADICTION | 40.1% | 24.2% |
| NO_CONTRADICTION | 31.3% | 56.2% |
| NOT_ENOUGH_INFO | 28.6% | 19.6% |
Four pre-registered prompts were run against both models — the original, one listing the conditions that must be present and requiring refusal otherwise, one forbidding any use of outside knowledge, and one instructing conservative adjudication. Eight runs, 5,824 judgements. Splitting each by whether the sentences state all three axes:
| run | CONTRADICTION when stated | when missing | difference |
|---|---|---|---|
| 14b · original | 40.1% | 39.4% | −0.7 pp |
| 14b · caution | 11.4% | 21.2% | +9.8 pp (p=0.004) |
| 14b · strict evidence | 17.4% | 25.3% | +7.9 pp (p=0.033) |
| 14b · conservative | 24.0% | 30.3% | +6.4 pp (p=0.112) |
| 8b · original | 19.2% | 25.7% | +6.5 pp (p=0.085) |
| 8b · caution | 0.0% | 1.6% | +1.6 pp (p=0.100) |
| 8b · strict evidence | 12.0% | 17.6% | +5.7 pp (p=0.082) |
| 8b · conservative | 9.0% | 16.6% | +7.6 pp (p=0.015) |
Seven of eight runs assert contradiction more often when the conditions are missing than when they are stated (sign test, one-sided p = 0.035). Correct behaviour is the opposite sign.
Instructing the model to be conservative does not fix it. Caution changes the
overall rate enormously — NOT_ENOUGH_INFO rises from 28.4% to 70.3% on the 14B
model and from 19.6% to 86.3% on the 8B. It does not change the direction. The
inversion is largest under the most cautious prompt.
A mechanism that would explain this, untested and offered as a hypothesis only: when conditions are stated, a model can see that they differ and refuses. When they are absent there is nothing visible to differ, so two bare opposing claims read as a clean conflict. Absence of stated conditions is treated as absence of confounds.
4. Prompt choice changes nearly half the verdicts. The original and cautious prompts agree on only 54.8% of pairs on the same model at temperature 0. The two models on the same prompt agree on 61.5%.
Replication is clean: the 8B model's re-run of the original prompt is identical to the first run, and the 14B's is within 0.5 pp.
Contradiction detection over claim sentences is unwarranted in most cases and the failure is not random. For 77.1% of the pairs, the text does not state what a verdict would require. Language models nevertheless answer, and answer more confidently where the text says less — seven of eight prompt/model combinations assert contradiction more often on pairs missing their conditions. Instructing caution raises refusal by forty points without correcting the direction. A deterministic check that requires the conditions returns nothing at all, because no pair supplies them.
What this does not license. It does not say the models are wrong on any particular pair — there is no ground truth for true contradiction here, and a model may be right from memorised knowledge of the underlying literature. It does not say the corpus is mislabelled; its annotators read whole abstracts. The claim is about warrant: a verdict asserted where the supplied text does not state what would be needed to rule out a difference in setup.
python fetch_corpus.py # verifies SHA-256
python run_baseline.py --prompt prompts/p0_original.txt --model qwen3:14b
python audit_stats.py # PASS/FAIL per numberRequires Ollama and ollama pull qwen3:14b. No API key is
read unless you explicitly pass one, and only extract_axes.py --backend gemini
accepts one — no key path is hardcoded anywhere in this repository.
audit_stats.py recomputes every headline number from the raw files and prints
PASS or FAIL against the value written in RESULTS.md. It re-derives the
checkable / not-checkable split independently rather than trusting the flag
stored in the result files.
One honest caveat about reproduction. The published axis annotations in
data/axes.jsonl were produced with gemini-3.5-flash-lite, and that is
recorded as it was actually run. Re-running extract_axes.py --backend ollama
uses a different annotator and will not reproduce those booleans exactly;
agreement between the two backends has not been measured. Everything downstream
of the axes — every baseline judgement, every rate in the tables above — is local
and reproduces exactly.
fetch_corpus.py download + SHA-256 verify ManConCorpus (not redistributed)
extract_axes.py annotate each claim's six comparability axes
run_baseline.py ask a local model, under a given prompt
measure.py fill rates, verdicts, vocabulary distance, examples
audit_stats.py independent recomputation, PASS/FAIL
prompts/ the pre-registered prompt variants
data/ PMIDs, axis booleans, verdicts — no corpus text
RESULTS.md full write-up, failure cases, limitations
LICENSE-DATA what is in data/ and why it is safe to publish
The full list is in RESULTS.md. The ones that matter most:
- There is no ground truth for true contradiction here. Nothing says the models are wrong on any pair. The claim is about warrant — a verdict asserted where the supplied text does not state what would be needed to rule out a difference in setup. A model may be right from memorised knowledge of the underlying literature, and this design cannot tell that apart.
- Axis annotation is by one model, checked against a re-run, not by human annotators. Null-decision stability across an independent re-run of 40 claims was 97.5%.
- Two models, one family, one quantisation. Their 61% agreement is itself evidence a third would land elsewhere.
- The direction result rests on a sign test, not on the individual runs. Only three of eight runs reach p < 0.05 on their own, and with sixteen tests across the matrix some of those would be expected by chance. The claim is carried by seven of eight runs pointing the same way (p = 0.035), which is real but modest evidence. Pooling the runs gives +5.6 pp at z = 4.4, and that number is not quoted as a result: the eight runs share the same 728 pairs and are not independent, so the pooled interval is too narrow.
- The mechanism is untested. That absent conditions read as absent confounds is a hypothesis consistent with the direction, not something measured here. Testing it would need pairs where conditions are stated and identical, and the corpus contains four.
- This is not a criticism of ManConCorpus. Its annotators read whole abstracts and agreed with each other. This measures what survives into the sentence, which is the unit later systems consume.
ManConCorpus is by Abdulaziz Alamri and Mark Stevenson, University of Sheffield, CC BY-NC-SA 2.0 UK, and is fetched from their page rather than redistributed here. See LICENSE-DATA.
A. Alamri and M. Stevenson (2016). A Corpus of Potentially Contradictory Research Claims from Cardiovascular Research Abstracts. Journal of Biomedical Semantics 7:36. doi:10.1186/s13326-016-0083-z
The deterministic comparability layer under test is
slate.claims.