ordereffect

If a model says an elephant is heavier than a mouse, will it also deny that a mouse is heavier than an elephant? This asks two instruction-tuned models (Qwen2.5 1.5B and 0.5B) each comparison in both directions and checks whether the answers are logical opposites - a consistency test that needs no external ground truth, only that a strict comparison and its reverse cannot both be true.

The headline: both models contradict themselves on reversed comparisons - the 1.5B on half of them, the 0.5B on ninety percent - and every single contradiction is "No to both directions." That is a negativity bias, the opposite of the acquiescence (yes-to-everything) bias usually attributed to language models: when unsure which way a comparison goes, these models deny both directions rather than affirm both.

Method

• For each of 40 clear comparisons (an elephant vs a mouse, a year vs a day, a truck vs a scooter, ...), ask the true-direction question ("Is A heavier than B?") and its reverse ("Is B heavier than A?"), both answered yes/no.
• Self-consistency = the two answers are opposites (one Yes, one No), which a strict comparison logically requires. It needs no external truth. Accuracy additionally requires the true direction to be Yes.
• Contradictions are classified as Yes-to-both (acquiescence) or No-to-both (negativity).
• Greedy decoding, both models, n=40 comparisons; the consistency rate carries a seeded bootstrap 95% CI given the modest item count.

Pre-registered prediction

Written down before the full run (a pilot showed the 0.5B contradicting itself heavily):

(1) Both models will self-contradict when a comparison is reversed. (2) The small model will do so far more. (3) The contradictions will be dominated by a Yes-to-both acquiescence bias - the models saying "yes" regardless of order.

Predictions 1 and 2 held. Prediction 3 was falsified, and cleanly: not a single contradiction on either model was Yes-to-both. All of them were No-to-both. The models have a negativity bias here, not acquiescence - reported as the finding.

Results (n=40 comparisons)

model	self-consistency (95% CI)	accuracy	contradictions	Yes-to-both	No-to-both
1.5B	0.500 (0.350-0.650)	0.500	20 / 40	0	20
0.5B	0.100 (0.025-0.200)	0.100	36 / 40	0	36

Full numbers in bench_results/frontier.md.

What this shows

1. Reversing a comparison breaks the model. The 1.5B gives logically opposite answers to a comparison and its reverse only half the time; the 0.5B only a tenth of the time. On the other cases it answers the same to both, which for a strict comparison is a self-contradiction.
2. The contradiction is a negativity bias, not acquiescence. Every contradiction on both models is "No" to both directions (0 of 56 are "Yes" to both). Asked "is a year longer than a day?" and "is a day longer than a year?", the model answers No to each. The common assumption that language models over-affirm does not hold here - these small models over-deny.
3. Consistency scales with capability, and accuracy rides on it. The 1.5B is five times as consistent as the 0.5B (0.50 vs 0.10), and accuracy matches consistency exactly, because the No-to-both cases also get the true direction wrong. Being right on these comparisons and being self-consistent are the same thing here.
4. A cheap, judge-free robustness probe. Because consistency needs no external answer key - only that reversing a comparison must flip a Yes to a No - it is a self-contained way to expose how brittle a model's comparative reasoning is, and it separates the two sizes sharply.

Limitations and falsifiers

• Two model sizes, 40 hand-built comparisons across everyday magnitude dimensions, one phrasing, greedy decoding. Not a claim about larger models, other question forms, or other domains.
• The consistency metric is exact (opposite answers) and needs no judge; the true directions are unambiguous by construction (an elephant is heavier than a mouse).
• Falsifier: if contradictions had included Yes-to-both cases, the negativity-bias claim would weaken. None did, on either model (0 of 56).
• Falsifier: if the small model had matched the large one's consistency, the capability claim would fail. It is 0.10 vs 0.50.
• The adversarial pass that tried to refute each claim is in REVIEW.md.

Reproduce

./scripts/gate.sh                 # ruff + mypy --strict + pytest + ASCII + independent verify
./reproduce.sh 8081 8082          # rerun both models against two OpenAI-compatible endpoints

tools/verify.py recomputes consistency, accuracy, and the contradiction split straight from the raw JSONL, sharing no code with the analysis, and the ship gate runs it.

Layout

src/oc.py            comparison bank + yes/no parse + order-consistency + contradiction class
tests/test_oc.py     5 unit tests, including the consistency and contradiction classification
tools/run_sweep.py   both-orderings comparison sweep
tools/analyze.py     consistency (bootstrap CI), accuracy, Yes-to-both vs No-to-both split
tools/verify.py      independent recompute of the headline claims (in the gate)
bench_results/       frontier.md + curve.json
claims.toml          every claim tied to its evidence
REVIEW.md            adversarial refutation attempt

MIT licensed.