sortfrontier

How long a list can a small language model sort, and - more interestingly - how does it fail? This sorts random integer lists of growing length with two instruction-tuned models (Qwen2.5 1.5B and 0.5B), direct answer versus step-by-step, and classifies every failure, with ground truth from Python's sorted() and no judge.

The headline is the failure mode, not the frontier: sorting breaks by a shift from comparison errors to tracking errors. On short lists the models return the right numbers in the wrong order (a comparison mistake); on long lists they drop or invent numbers entirely (a tracking mistake), and by 16 elements ~97% of all errors are of this tracking kind on both models. And, against prediction, chain-of-thought does not help sorting - it significantly hurts the small model - the opposite of what it does for arithmetic.

Method

• Lists of length L in {3, 5, 8, 12, 16}, integers 1-99 with duplicates allowed, generated deterministically per length so every model and both conditions see the identical inputs and CoT-vs-direct is paired.
• Two conditions (direct: "reply with only the sorted numbers"; CoT: "think step by step, final sorted list on the last line"), both models, n=40 per cell.
• Ground truth is sorted(input); the model's returned numbers are parsed and scored three ways: correct (exactly sorted), order error (same multiset, wrong order - a comparison failure), or set error (multiset changed, numbers dropped or altered - a tracking failure).
• The frontier is the longest list sorted at >=0.5 accuracy, contiguous from the shortest up.

Pre-registered prediction

Written down before the full run (a pilot showed clean degradation):

(1) Sort accuracy falls with list length. (2) The smaller model's frontier is shorter. (3) CoT extends the frontier, as it does for arithmetic. (4) Failures shift from order errors at short lists to set errors at long lists, more so on the small model.

Predictions 1, 2, and 4 held. Prediction 3 was falsified: CoT does not extend the sort frontier on either model (a wash on the 1.5B, McNemar p=0.12) and significantly hurts the 0.5B (frontier 5 to 3, McNemar p=0.002). That reversal, reported here rather than buried, is what makes the result interesting - see "A theory of when CoT helps" below.

Results (n=40 per cell)

Exact-sort accuracy by list length, and the frontier:

1.5B      L=3   L=5   L=8   L=12  L=16   frontier
direct    0.80  0.68  0.60  0.40  0.12      8
cot       0.97  0.82  0.65  0.35  0.10      8

0.5B      L=3   L=5   L=8   L=12  L=16   frontier
direct    0.78  0.70  0.30  0.00  0.03      5
cot       0.53  0.47  0.28  0.05  0.00      3

Fraction of direct-condition errors that are set errors (dropped or altered numbers):

          L=3   L=5   L=8   L=12  L=16
1.5B      0.00  0.08  0.25  0.67  0.97
0.5B      0.00  0.00  0.43  0.70  0.97

Full numbers in bench_results/frontier.md.

What this shows

1. Sorting degrades with length and the small model's frontier is shorter (5 versus 8). No surprise, and the setup for the interesting part.
2. The failure mode shifts from comparison to tracking. At length 3, essentially every error is an order error - the model has the right numbers but orders them wrong. By length 16, ~97% of errors are set errors - the model has dropped, duplicated, or invented numbers. The wall is not sorting logic; it is holding all the numbers at once.
3. CoT does not help sorting, and hurts the small model. On the 1.5B it is a wash; on the 0.5B it lowers the frontier from 5 to 3 (McNemar p=0.002), because reasoning out loud through a long, evolving list is itself where numbers get dropped - CoT adds set errors rather than preventing them.

A theory of when CoT helps

Across these studies, chain-of-thought helps exactly when the intermediate state it has to carry is small:

• Multi-digit addition and long chains of additions (companion studies): the running state is a single number. CoT helps a lot.
• Retrieval multiple choice (companion study): no useful intermediate state. CoT hurts.
• Sorting (here): the intermediate state is the whole list - as large as the answer. CoT does not help and hurts the weaker model, because maintaining that large state in text is as error-prone as the task itself.

So "let the model think step by step" is not free advice: it helps when the steps compress the problem to a small running state and backfires when they do not.

Limitations and falsifiers

• Two model sizes, one task, one prompt pair per condition, greedy decoding, n=40 per cell. Not a claim about larger models or other algorithms.
• The order/set error split is exact (multiset comparison), so the mode-shift finding does not depend on a threshold.
• Falsifier: if set errors had not come to dominate at long lists, the tracking-failure claim would fail. They reach 0.97 on both models.
• Falsifier: if CoT had extended the frontier, prediction 3 would have held; it did not, and it hurt the small model significantly.
• The adversarial pass that tried to refute each claim is in REVIEW.md.

Reproduce

./scripts/gate.sh                 # ruff + mypy --strict + pytest + ASCII + independent verify
./reproduce.sh 8081 8082          # rerun both models against two OpenAI-compatible endpoints

tools/verify.py recomputes the frontiers, the paired CoT-vs-direct McNemar, and the set-error fractions straight from the raw JSONL, sharing no code with the analysis, and the ship gate runs it.

Layout

src/srt.py           deterministic lists + robust parse + exact/multiset scoring + error class
tests/test_srt.py    7 unit tests, including the error classification and the frontier
tools/run_sweep.py   list-length x condition sweep, identical lists per cell
tools/analyze.py     accuracy by length, frontiers, CoT extension, error composition
tools/verify.py      independent recompute of the headline claims (in the gate)
bench_results/       frontier.md + curve.json
claims.toml          every claim tied to its evidence
REVIEW.md            adversarial refutation attempt

MIT licensed.