Cross-Scale Downstream Pareto — the next king-selection rule

[bitzic]

TL;DR

Our most recent post published the verdict that the 20-step proxy was not tracking longer-run truth (ρ=0.203, 2 promotion errors of 12; pre-registered before any data landed). The initial plan was to replace it with a multi-scale Pareto ladder at small parameter counts, scoring val_bpb against sealed token streams. On further review of where the 2026 open-source pretraining community has converged, the ladder is moving up and the metric is moving off val_bpb. The new rule is called Cross-Scale Downstream Pareto. Every accepted king's recipe will also be trained at a celebrated scale and released as a public model. This post lays out the new rule, the evidence behind it, and a transfer-credibility test that has to land before the rule freezes.

Why the initial small-scale Pareto ladder wasn't enough

Two pieces of external evidence and one of our own kept pointing in the same direction once we read them carefully.

The first is community coordination. Keller Jordan's modded-nanogpt is a public speedrun: 124M-parameter GPT-2-class architecture trained on 8×H100 against the FineWeb validation set, with a target of val ≤ 3.28. As of this writing the leaderboard has 83 records. This is the rung where independent researchers actually compete on recipe improvements in the open. A subnet that selects recipes at d=384 (~14M nonembed params) sits an order of magnitude below where the field's open coordination already lives.

The second is transfer credibility. DCLM (Li et al. 2024, arXiv:2406.11794, §3.2 Figure 3) reports that data-curation rankings across 10 methods at the 400M-parameter scale correlate with the same rankings at 7B at Pearson r = 0.838, rising to r = 0.956 at 1B and r = 0.982 at 3B. Karpathy's nanochat #420 ladder runs d=10–20 with CORE-22. Below 400M, recipe rankings drift from frontier-scale truth. A selection rule that crowns recipes at 14M params is selecting for what works at 14M params, not for what compounds at scale.

The third is our own internal admission. Our king-criterion-review recommendation states verbatim: "the current gate cannot distinguish 'lowers bpb because it makes a better LLM' from 'lowers bpb because it overfits the eval distribution in a way fine-tuning would erase'" — a self-indictment that should have been the headline, not a §8 footnote.

The metric question is sharper than the scale question. Karpathy named val_bpb on a private corpus as the "evaluation crisis" axis in March 2025. The 2026 follow-up Forecasting Downstream Performance of LLMs With Proxy Metrics (arXiv:2605.18607) reports mean Spearman ρ = 0.81 for downstream-proxy ranking vs ρ = 0.36 for cross-entropy — more than 2× the signal at the metric layer alone, independent of scale.

The new rule — Cross-Scale Downstream Pareto

Each submission is evaluated at three scales, then judged on a downstream-task ensemble at the top rung.

S₁ : d=256, L=4   — ~30 seconds on a single H100. val_bpb on a sealed stream.
                    Cheap pre-screen. Reject if val_bpb regresses by ≥3σ at this rung.

S₂ : d=512, L=12  — ~5–10 minutes on a single H100. val_bpb on a sealed stream.
                    Intermediate check; sits in the DCLM-validated transfer band.

S₃ : d=768, L=12  — ~70 minutes on a single H100 in bf16 (~124M nonembed params).
                    Trained on FineWeb-Edu sealed shards.
                    Evaluated on CORE-22 (Karpathy's nanochat eval, 22 datasets)
                    plus a private hardness-graded subset:
                    HellaSwag-hard / ARC-easy / OpenBookQA / TinyMMLU.

The decision predicate. A challenger crowns iff:

1. No regression at any (rung, eval) cell beyond a per-cell calibrated threshold (the pre-screen, the val_bpb-on-S₂ check, AND every downstream metric at S₃);
2. At least one statistically significant win on the downstream ensemble at S₃, or on val_bpb at S₂.

There is no single-axis dominance backdoor. There is no benchmark-blind clause. The validator chooses the seeds from on-chain randomness, the eval streams come from a sealed pool with per-epoch rotation, and the per-cell noise floors are calibrated against the baseline recipe before any submission is scored.

Cost: each submission costs roughly 70–90 minutes of one H100 plus the cheap pre-rungs. That is materially more than the small-scale ladder we initially planned (about 50× more per submission), but the rung is the same one the rest of the field competes on, and the downstream ensemble is the metric the field actually cites. We think that is the right tradeoff for a "compounding artifact" mission.

The artifact is a model, not just a recipe

Going forward, every recipe that earns the crown is trained at 254M parameters / 1B FineWeb-Edu tokens and released as a public model on Hugging Face. The signed on-chain receipt pairs a RecipeReleased event with a DemonstrationModelReleased event, so the recipe and the model are bound. The whitepaper promised a compounding artifact; the artifact has to be downloadable, not just diff-able.

Transfer-credibility test

Before the new rule freezes, we will run a Karpa-internal version of the DCLM-style cross-scale rank-correlation test: 12 candidate recipes, scored under the Cross-Scale Downstream Pareto rule, then independently trained at 1B parameters / 30B tokens (using OLMo-2-1B as the published reference recipe baseline) and ranked. The pre-registered pass threshold is ρ ≥ 0.6 between the new rule's S₃ rankings and the 1B-scale ground-truth rankings.

If the test fails — if the new rule does not rank-correlate with 1B-scale truth at ρ ≥ 0.6 — we don't adopt it. We escalate the ceiling further, or revisit the metric ensemble, until the rule passes its own pre-registered transfer-credibility test. That is the discipline we owe the corpus.

What's open for community input

Three things would benefit from the community's eyes before the new rule freezes.

1. The downstream ensemble. CORE-22 plus a private HellaSwag-hard / ARC-easy / OpenBookQA / TinyMMLU subset is one defensible cut, but it is not the only one. Anyone who has run rank-correlation studies of evals against frontier-scale outcomes — favorable axes (MMLU-Pro / MATH-Lvl5-mini / GPQA-easy / IFEval-lite) or unfavorable axes (the LLM-as-judge tradition, the open-ended eval tradition) — please weigh in. We want the ensemble that maximizes rank-correlation per dollar of validator compute.

2. The S₃ rung specifically. d=768 / L=12 / 124M-class on FineWeb-Edu matches the modded-nanogpt coordination point. An argument we want to hear: should we move S₃ to d=1024 / L=18 / ~250M to sit fully inside DCLM's transfer-credible band, at the cost of ~3× more H100 time per submission?

3. Replication. The 12-recipe proxy-validation experiment is fully published. We would value a third-party replication with a different recipe set — especially recipes from axes we did not cover (optimizer family swaps beyond AdamW, attention variants beyond QK-Norm, longer-context schedules, FP8 / NVFP4 numerics). If anyone wants to run a community replication funded from the protocol's runner-up pool, open an Issue or reply here.

Receipts

Proxy-validation experiment, code + data + pre-registration: karpa/experiments/2026-06-proxy-validation/
modded-nanogpt: https://github.com/kellerjordan/modded-nanogpt
DCLM (Li et al. 2024) §3.2 Fig 3 transfer-correlation: https://arxiv.org/abs/2406.11794
Forecasting Downstream Performance of LLMs With Proxy Metrics (2026): https://arxiv.org/abs/2605.18607
Karpathy "evaluation crisis" (March 2025): https://x.com/karpathy/status/1896266683301659068
Karpathy nanochat ladder discussion: karpathy/nanochat#420
DataDecide (small experiments → 1B prediction): https://arxiv.org/abs/2504.11393
"Can Small Training Runs Reliably Guide Data Curation?": https://arxiv.org/abs/2512.24503