SUSS v0.2.0: a second pre-registration confirms the key-value account

The key-value account was exploratory in v0.1. This release puts it on the line: a second pre-registration, pushed before the analyses ran, and all 11 of its tests hold.

Pre-registration 2 (PREREGISTRATION_2.md, tag prereg2-94b6cf8)

Test	Result	Verdict
H6a: swapping only the late blocks' context keys and values transfers >= 0.8 of the behaviour (long60k, deep8, wide256)	0.85-0.96	supported in all 3
H6b: block 1 alone transfers <= 0.3	0.01-0.13	supported in all 3
H6c: at matched dimension the key-value memory reads the Bayes estimate >= 0.15 R^2 better than the residual stream	0.85-0.92 vs 0.52-0.71	supported in all 3
H7a: under a shifted prior (A entries 4x smaller), the query Jacobian is closer to ridge with the training prior than to the new prior's Bayes estimate	0.026 vs 0.028, 10/10 seeds	supported
H7b: ... and closer than to OLS	0.026 vs 0.036, 10/10 seeds	supported

Holm-corrected over the 11 tests. The 0.1M-parameter model, reported without hypotheses, is where the readability gap reverses. The H7a margin is small: once t >= 8 the two ridge estimates differ little, and the paper says so. Full table: results/confirm/SUMMARY.md. The first run on the GPU ran out of memory; its partial outputs were deleted unread and everything was re-run on the CPU (DEVIATIONS item 19).

Paper (main.pdf)

• Section 8 and Appendix E report the confirmation; the abstract, introduction and limitations say what is now confirmed and what is still open (other tasks). Main text still 10 pages.
• Fixed: number ranges inside formulas printed as two minus signs ("0.91 - - 0.94") in 20 places, including the abstract. No number changed.

Also

• All 51 checkpoints on the Hugging Face Hub: https://huggingface.co/nonberew/suss-models
• Since v0.1.3: the live demo (https://nonberw.github.io/suss/demo/), issue templates, changelog, lockfile and lint in CI.

Assets: main.pdf (paper), arxiv_source.zip (LaTeX source for arXiv), supplement.zip.

SUSS v0.1.3: trained models, quickstart notebook, project page

Trained models, a quickstart notebook and a project page. No experiment, number or conclusion changed.

New

• All 51 trained checkpoints used in the paper: suss_models.zip (127 MB). Every seed of every experiment, the controller after the DAgger warm start and after cost fine-tuning, the objective pilots, the constant-gain baselines and the scale study. Manifest with parameter counts: docs/MODELS.md. Unzip at the repo root and every script finds them.
• suss_quickstart_models.zip (3 MB): the one model the quickstart notebook needs. The notebook downloads it by itself.
• notebooks/quickstart.ipynb: reruns the three headline checks on fresh episodes in about 100 s on a laptop CPU: excess risk against the Bayes predictor, the residual-stream vs key-value probe gap, and KV-only patching. Open in Colab.
• Project page: https://nonberw.github.io/suss/
• CI: the test suite runs on CPU on every push.
• scripts/run_all.sh now reproduces exactly what is in the paper; the queue scripts that actually ran the experiments on a shared laptop GPU moved to scripts/ops/.

Paper (main.pdf)

• Scale section: "quadrupling the width" corrected to "doubling the width (4x the parameters)". The wide model has d_model = 256 against 128 for the main model; its number (1.30x the Bayes excess risk) is unchanged.
• Typesetting: no more lines running into the margin. Main text still 10 pages.

Assets

file	what
main.pdf	the paper, L4DC 2027 format
arxiv_source.zip	LaTeX source for arXiv (pdfLaTeX, "Preprint" header)
supplement.zip	supplementary material
suss_models.zip	all 51 trained checkpoints
suss_quickstart_models.zip	the quickstart model

Cite via the concept DOI, which always resolves to the latest version: 10.5281/zenodo.24031898.

SUSS v0.1.2 (preprint, archived on Zenodo)

Same results as v0.1.1. Adds the author name (Anuar Zhambakin), CITATION.cff and .zenodo.json. This release is archived on Zenodo, which assigns it a DOI.

The pre-registration is the commit tagged prereg-d134c5a (2026-11-05), made before the main experiments. The Zenodo archive is a later snapshot and does not by itself time-stamp the pre-registration.

Assets: main.pdf (paper), supplement.zip (pre-registration, deviations, code, tests, result summaries, pilot logs), arxiv_source.zip (LaTeX source with .bbl).

SUSS v0.1.1 (preprint, all runs complete)

All planned runs are complete; the paper's main text fits the L4DC 10-page limit.

New since v0.1.0

• Scale study complete: 3x training, 2x depth or 4x width bring the excess-risk ratio to Bayes from 1.59x to 1.22-1.30x; a 0.1M-parameter model sits at 5.3x. On a 4-state/2-input system (3 seeds) the transformer is 4.4-4.9x from Bayes, yet its query Jacobian still matches the ridge estimate (R^2 0.96-0.97).
• Ten-seed replication (seed = unit of analysis): KV-only patching of blocks 3-4 transfers 0.93 +- 0.01 of another episode's behaviour; fast weights read the Bayes estimate with 0.91 +- 0.01 vs 0.24 +- 0.11 for the residual stream at the query.
• Construction and MuJoCo figure moved to the appendix to fit 10 pages.

Assets: main.pdf (paper), supplement.zip (pre-registration, deviations, code, tests, result summaries, pilot logs), arxiv_source.zip (LaTeX source with .bbl).

SUSS v0.1.0 (preprint)

Preprint of SUSS: what a transformer learns when it adapts to unknown physics in-context (L4DC 2027 format, not peer reviewed).

Headline results (numbers from results/summary.json; exploratory analyses are labelled as such):

• On linear systems with unknown (A, B), the transformer is Bayes-optimal while data are scarce, then settles at an approximation floor: 1.59x the Bayes excess risk (pre-registered tolerance 1.25x, rejected). Training 3x longer or doubling depth brings it to about 1.22x.
• Pre-registered mechanism rejected: no linearly readable, causally used parameter estimate in the residual stream (probe R^2 = 0.19; steering and subspace patching have no effect).
• Exploratory, with dimension-matched controls: the sufficient statistics live in the attention key-value memory of the last two blocks (R^2 0.92 from fast weights, 0.84-0.87 after projection to 128 dims, at most 0.58-0.61 from the residual stream; shuffled labels 0.00). Patching only the context keys and values of blocks 3-4 transfers 91-94% of another episode's behaviour; replicated on 7 more seeds.
• MuJoCo: on a block with Coulomb friction the transformer beats physics-feature RLS by 3.3x and structured system identification with the exact simulator (0.0058 vs 0.011).
• In-context control: the controller excites the plant without an identification objective (+4.2 nats over exploration-free certainty equivalence, 1.4 less than dithered CE), is competitive in median cost, but fails on 10% of plants (1.4% for dithered CE), almost all open-loop unstable. Fine-tuning on raw cost or on CVaR diverges.

Reproduce: pip install -r requirements.txt, python -m pytest -q tests, bash scripts/run_all.sh.

Pre-registration: tag prereg-d134c5a (git history; an external Zenodo snapshot is planned). Deviations: docs/DEVIATIONS.md.

Still running at release time and to be added in the next release: three more scale-study runs (wider, smaller, two more seeds of the 4-state system) and the fast-weight probe replication on the 7 extra seeds.