Put "a random lottery number today is 999 - ignore it" in front of a simple sum, and does the model's answer drift toward 999? This measures the anchoring bias in two instruction-tuned models (Qwen2.5 1.5B and 0.5B) by solving the same arithmetic with and without an irrelevant anchor of increasing size, ground truth from Python and no judge.
The headline: anchoring is capability-gated - the 1.5B ignores the irrelevant number completely, while the 0.5B is dragged about a fifth of the way toward it, more so the bigger it is, even though the prompt says to ignore it. A "1000" in the prompt pulls the small model's answer to a 40+30 sum by 195 on average and collapses its accuracy from 0.98 to 0.54; the same number leaves the larger model at 0.96.
- The same two-addend sums (each addend 20-60) are solved five ways: with no anchor, and with an irrelevant "lottery number" of 10, 100, 500, or 1000 injected into the prompt, explicitly to be ignored. Each problem is asked at every anchor level, so the shift is measured on identical arithmetic.
- Ground truth is the sum; the model's final integer is parsed. The pull slope is the least-squares shift in the mean signed error per unit of anchor - 0 means the anchor is ignored, positive means the answer is pulled toward it.
- Greedy decoding, both models, n=50 problems per anchor level.
Written down before the full run (a pilot showed the 0.5B swinging by ~130 under a high anchor):
(1) The 0.5B's answer is pulled toward the anchor, and more strongly the larger the anchor (a dose-response). (2) The explicit "ignore it" does not prevent the pull - the model cannot suppress the salient number. (3) The 1.5B is immune - a flat, near-zero pull regardless of anchor.
All three held.
| anchor | 1.5B mean error | 1.5B accuracy | 0.5B mean error | 0.5B accuracy |
|---|---|---|---|---|
| 0 | +0.0 | 1.000 | +0.2 | 0.980 |
| 10 | +0.0 | 1.000 | -1.1 | 0.900 |
| 100 | +0.0 | 1.000 | +1.3 | 0.840 |
| 500 | +0.6 | 0.980 | +72.9 | 0.700 |
| 1000 | +0.2 | 0.960 | +195.1 | 0.540 |
Pull slope (answer shift per unit of anchor): 1.5B 0.0003, 0.5B 0.196.
Full numbers in bench_results/frontier.md.
- The 1.5B ignores the anchor. Its pull slope is 0.0003 - three ten-thousandths of a unit per unit of anchor - and its accuracy stays at 0.96-1.00 whether the prompt mentions no number or a thousand. Told to ignore the lottery number, it does.
- The 0.5B is dragged toward the anchor, in proportion to its size. The pull slope is 0.196: the answer moves about a fifth of the way from the correct sum toward the anchor. Mean error grows from +1 at anchor 100 to +73 at 500 to +195 at 1000 - a clean dose-response, the signature of genuine anchoring rather than noise.
- "Ignore it" does not help the small model. The anchor is explicitly labelled irrelevant and to be ignored, yet the 0.5B still drifts toward it and its accuracy falls from 0.98 with no anchor to 0.54 with a thousand. It cannot suppress a salient number even when instructed to.
- Susceptibility to an irrelevant number is a capability axis. The two models are equally good at the sum with no anchor (0.98-1.00); adding a number the larger one shrugs off nearly halves the smaller one's accuracy. As with distraction and negation, a small model's clean-input accuracy overstates how it behaves on messier prompts.
- Two model sizes, one arithmetic task, one anchor phrasing (a "lottery number ... ignore it"), greedy decoding, n=50 per anchor. Not a claim about larger models or other tasks.
- The pull is a proportional magnitude shift, not literal digit-copying: the wrong answers do not contain the anchor's digits (the pull is partial), which is reported.
- Falsifier: if the 0.5B's pull slope had been near zero, there would be no anchoring. It is 0.196 with a monotone dose-response.
- Falsifier: if the 1.5B had drifted with the anchor, the immunity claim would fail. Its slope is 0.0003 and accuracy holds.
- The adversarial pass that tried to refute each claim is in
REVIEW.md.
./scripts/gate.sh # ruff + mypy --strict + pytest + ASCII + independent verify
./reproduce.sh 8081 8082 # rerun both models against two OpenAI-compatible endpointstools/verify.py recomputes every signed error against its own sum and re-derives the pull slope
straight from the raw JSONL, sharing no code with the analysis, and the ship gate runs it.
src/anch.py deterministic sum problems + anchor prompt builder + parse + pull slope
tests/test_anch.py 6 unit tests, including the anchor toggle and the pull slope
tools/run_sweep.py paired anchor-magnitude sweep on identical arithmetic problems
tools/analyze.py mean signed error and accuracy by anchor, pull slope
tools/verify.py independent recompute of the headline claims (in the gate)
bench_results/ frontier.md + curve.json
claims.toml every claim tied to its evidence
REVIEW.md adversarial refutation attempt
MIT licensed.