Does self-consistency (sample many chain-of-thought paths, take the majority answer) beat greedy decoding on GSM8K, and does the benefit hold as the model gets smaller? This measures the full accuracy-vs-vote-count curve for two instruction-tuned models (Qwen2.5 1.5B and 0.5B), with an exact-match oracle and no judge, and pins down why voting helps when it helps.
The headline: the payoff from self-consistency scales with model capability. On the 1.5B majority voting adds a large, significant +13 points (0.700 -> 0.830, McNemar p=0.002). On the 0.5B the same procedure adds only +7 points and does not reach significance (p=0.19). The reason is measurable: the weak model's sampled answers are more than twice as scattered (2.73 vs 1.18 bits of answer entropy), so there is no confident majority to concentrate on.
- For each question we record one greedy answer (temperature 0) and 16 sampled answers (temperature 0.7, distinct seeds), on the same 100-question GSM8K slice, same prompt.
- Self-consistency accuracy at k is derived offline: majority-vote the first k of the 16 samples (ties break toward the earliest-appearing answer). One sweep fixes the whole accuracy-vs-k curve, so the curve is internally consistent.
- Scoring is exact last-integer match against the gold
#### Nfield. No judge. - Answer entropy = Shannon entropy (bits) of a question's 16 sampled answers; it measures how scattered the model's reasoning outcomes are.
- Two sizes (1.5B and 0.5B) so the scaling of the benefit is validated, not assumed.
Written down before the full run (a small pilot probe hinted at a size split):
Self-consistency accuracy will rise with k and saturate, helping the 1.5B. On the 0.5B it will be flat-to-negative, because its sampled paths are more scattered so majority voting concentrates on a confident-but-wrong mode. The benefit will track single-path signal quality / answer entropy, and entropy will be higher on the 0.5B and on the questions the model gets wrong.
What held and what did not (reported honestly):
- Held: the 1.5B gains significantly and monotonically, saturating by k~8. The entropy mechanism held strongly - the 0.5B is 2.3x more scattered, and in both models the greedy-wrong questions carry ~2x the entropy of the greedy-right ones.
- Too pessimistic: the 0.5B was not flat-to-negative; it gained a small +7 points. But that gain is not statistically significant at n=100 (p=0.19), so the honest summary is "underpowered positive," not "voting hurts." The pilot's apparent harm was small-sample noise, and this is exactly why the study was run at n=100 with a hypothesis test.
| model | greedy | self-consistency (k=16) | gain | McNemar p | mean entropy (bits) | entropy right / wrong |
|---|---|---|---|---|---|---|
| 1.5B | 0.700 | 0.830 | +0.130 | 0.0023 | 1.179 | 0.749 / 2.182 |
| 0.5B | 0.250 | 0.320 | +0.070 | 0.1892 | 2.731 | 2.211 / 2.904 |
Accuracy vs vote count k:
k 1.5B 0.5B
1 0.700 0.220
2 0.700 0.220
4 0.790 0.250
8 0.820 0.320
16 0.830 0.320
Full numbers in bench_results/frontier.md.
- Self-consistency significantly helps the stronger model (+13 points, p=0.002), monotone in k and saturating by roughly 8 votes - past that, more samples buy little.
- The benefit scales with capability. The 0.5B gets a smaller, non-significant bump. More votes do not manufacture a signal that single paths do not carry.
- Answer entropy is the mechanism, and it is diagnostic. The weak model's paths are 2.3x more scattered. Within each model, the questions it gets wrong are the high-entropy ones (1.5B 2.18 vs 0.75 bits; 0.5B 2.90 vs 2.21). When the sampled answers disagree, majority voting has no confident mode to lock onto - so entropy predicts where voting fails.
The practical read: self-consistency is worth its 16x token cost on a capable model and much weaker on a small one; the answer-entropy of a handful of samples tells you, per question, whether voting will help before you pay for all of them.
- Two model sizes, one benchmark, one sampling temperature (0.7). Not a claim about larger models, other tasks, or other temperatures.
- The 0.5B result is a non-significant positive, i.e. underpowered at n=100; it is reported as such, not as "voting hurts."
- Falsifier: if the 1.5B's McNemar p had been >=0.05, the "significant help" claim dies. It is 0.002.
- Falsifier: if the 0.5B's answers were not more scattered than the 1.5B's, the entropy mechanism dies. They are (2.73 vs 1.18 bits).
- The adversarial pass that tried to refute each claim is in
REVIEW.md.
./scripts/gate.sh # ruff + mypy --strict + pytest + ASCII + independent verify
./reproduce.sh 8081 8082 # rerun both models against two OpenAI-compatible endpointstools/verify.py recomputes greedy accuracy, self-consistency accuracy, the McNemar test, and
answer entropy straight from the raw JSONL, sharing no code with the analysis, and the ship
gate runs it.
src/selfcon.py exact scoring + majority vote + accuracy-at-k + answer entropy
tests/test_selfcon.py 8 unit tests, including the vote tie-break and the entropy measure
tools/run_sweep.py greedy + 16 sampled paths per question (concurrent draws)
tools/analyze.py accuracy-vs-k curve, entropy split, McNemar self-consistency vs greedy
tools/verify.py independent recompute of the headline claims (in the gate)
bench_results/ frontier.md + curve.json
claims.toml every claim tied to its evidence
REVIEW.md adversarial refutation attempt
MIT licensed.