Repository navigation
Limitations
A pilot study, stated plainly. Every item below limits what you can claim from the numbers.
| # | Limitation | Direction of the effect | What would fix it |
|---|---|---|---|
| 1 | Small sample. 72 duels, 18 per model, 12 per fact, 3 per model-and-fact cell, one run, no confidence intervals | Gaps could shrink or flip on a re-run | Repeat each cell 10 to 30 times |
| 2 | One round of pressure. The Attacker pushes once | Understates erosion over a long argument | Run 3 to 5 rounds |
| 3 | Temperature 0.5. Adds sampling noise | One stochastic flip moves a cell by 33 points | Run at temperature 0 as well |
| 4 | Small local models. gemma2, llama3.1, mistral, qwen2.5 | Says nothing about large hosted models | Add bigger models |
| 5 | The Defender is handed the truth. This tests whether a model keeps a fact in context, not whether it knows it | Cannot speak to recall | Add a no-hint baseline run |
| 6 | Scoring reads the final answer only. A third answer counts as held (6 of 72 final answers) | Understates gullibility | Report a three-way outcome |
| 7 | The lie is one specific number. Wandering to another wrong value does not count | Same as above | Score "any wrong answer" separately |
| 8 | Question wording drift. The stored dataset says "default account quota" for Lambda and S3, later reworded to "THIS account's" without a rerun | Lambda and S3 rows are softer | Rerun with the final wording |
| 9 | Config defaults differ from the run. Code defaults are 3 rounds and temperature 0, the run used 1 round and 0.5 | Easy to describe the wrong experiment | Read config from the dataset |
| 10 | Roles are entangled. The same model is Attacker and Defender in different duels | Cannot separate persuasion from gullibility cleanly | Fix one model as a standard Attacker |
| 11 | Account-specific truth. Quotas and counts come from one AWS account | Other accounts differ | Record the truth per run (we do) |
| 12 |
A scoring bug was found after first publication. The parser read 100,000 as 100
|
Corrected: 36 caves became 44 | Fixed in bench/debate.py, tests added |
| 13 | Screenshots come from earlier game runs. Some used llama3.2, and the game's own tally and podium screens were produced by the older parser | Visuals are illustrative, tables are authoritative | Re-record with the fixed parser |
| 14 | No isolation of the "familiar facts hold" idea. It is a hypothesis | Do not cite it as a finding | Measure each model's unprompted answer per fact |
- Small local models can be pushed off a true value they were handed, by a single confident paragraph, in more than half of the duels in this run.
- The fact matters a lot: from 92% (regions) down to 17% (vCPUs) in this sample.
- No Defender opened with the lie. Every cave came after the push.
- A ranking of the models for general use.
- Any statement about hosted frontier models.
- A causal claim about why one fact folds more than another.
Related: Methodology, Roadmap.