Cross LLM Comparison
How much does the LLM generation matter for autonomous hyperparameter optimization? This page tracks cross-generation comparison experiments: the same dataset, same hardware, same codebase, same training budget — but different Claude model versions driving the optimization.
| Metric | ClimbMix | FineWeb-Edu | Cosmopedia-v2 | SlimPajama |
|---|---|---|---|---|
| S4.0 baseline | 1.2969 (pre-opt) | 1.4088 (defaults) | 0.9640 (pre-opt) | 1.5410 (pre-opt) |
| S4.6 baseline | 1.3213 (defaults) | 1.3710 (defaults) | 0.9806 (defaults) | 1.5312 (defaults) |
| S4.0 best | 1.2959 | 1.3424 | 0.9606 | 1.5259 |
| S4.6 best | 1.2997 | 1.3416 | 0.9549 | 1.5267 |
| S4.0 improvement | −0.08% | −4.71% | −0.35% | −1.0% |
| S4.6 improvement | −1.63% | −2.14% | −2.63% | −0.29% |
| S4.0 keeps | 1 (1.0%) | 17 (19.3%) | 4 (3.9%) | 3 (3.0%) |
| S4.6 keeps | 8 (6.8%) | 18 (18.0%) | 16 (16.0%) | 2 (2.0%) |
| S4.0 crashes | 11 (11.0%) | 2 (2.3%) | 2 (1.9%) | 0 (0.0%) |
| S4.6 crashes | 5 (4.2%) | 1 (1.0%) | 0 (0.0%) | 2 (2.0%) |
The scorecard: Sonnet 4.6 leads 2-1-1. Sonnet 4.6 wins on FineWeb-Edu (barely) and Cosmopedia-v2 (decisively). Sonnet 4.0 wins on ClimbMix (absolute, with baseline advantage) and SlimPajama (clearly — better absolute result and more keeps). The models tie on crash rates overall. SlimPajama is notable as the hardest dataset where both models barely improve from baseline — the optimization landscape is extremely flat.
| Metric | Sonnet 4.0 (Mar 19) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.2969 (pre-optimized) | 1.3213 (defaults) | Different starting points |
| Best val_bpb | 1.2959 | 1.2997 | Sonnet 4.0 (absolute) |
| Improvement | −0.08% | −1.63% | Sonnet 4.6 (20× more) |
| Keeps | 1 / 100 (1.0%) | 8 / 118 (6.8%) | Sonnet 4.6 (7× more) |
| Crashes | 11 / 100 (11.0%) | 5 / 118 (4.2%) | Sonnet 4.6 (2.6× fewer) |
| Exps to first keep | 84 | 25 | Sonnet 4.6 (3.4× faster) |
| Parameters optimized | 1 | 3 (synergistic) | Sonnet 4.6 |
What Each Model Found (click to expand)
Sonnet 4.0's single keep (exp84):
UNEMBEDDING_LR = 0.0041 # was 0.004 — a 2.5% tweak
# Result: 1.2959 (−0.0010 from 1.2969 baseline)Sonnet 4.6's optimal configuration (exp112):
WARMDOWN_RATIO = 0.63 # was 0.5 — 26% longer annealing
SCALAR_LR = 0.3 # was 0.51 — 41% lower
ADAM_BETAS = (0.71, 0.95) # was (0.8, 0.95) — 11% lower momentum
# Result: 1.2997 (−0.0216 from 1.3213 baseline)| Metric | Sonnet 4.0 (Mar 17) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.4088 (defaults) | 1.3710 (defaults) | S4.6 starts better |
| Best val_bpb | 1.3424 | 1.3416 | Sonnet 4.6 (by 0.0008) |
| Improvement | −4.71% | −2.14% | Sonnet 4.0 (more room) |
| Keeps | 17 / 87 (19.3%) | 18 / 99 (18.0%) | Comparable |
| Crashes | 2 / 87 (2.3%) | 1 / 99 (1.0%) | Sonnet 4.6 |
| Exps to first keep | 3 | 3 | Tied |
| Parameters optimized | 6 | 9 | Sonnet 4.6 |
Configuration Comparison (click to expand)
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 32 | ✅ |
| MATRIX_LR | 0.053 | 0.055 | ≈ |
| EMBEDDING_LR | 0.59 | 0.45 | ❌ |
| SCALAR_LR | 0.3 | 0.23 | ❌ |
| UNEMBEDDING_LR | 0.003 | 0.006 | ❌ (2× different) |
| WEIGHT_DECAY | 0.12 | 0.2 | ❌ |
| WARMDOWN_RATIO | 0.47 | 0.77 | ❌ (64% different) |
| FINAL_LR_FRAC | 0.0 | 0.1 | ❌ |
| ADAM β1 | 0.8 | 0.67 | ❌ |
| ADAM β2 | 0.95 | 0.975 | ❌ |
Both models converge to essentially the same val_bpb (within 0.0008) via completely different configurations. Only architecture (AR=32) and matrix LR agree — proving that FineWeb-Edu's optimization landscape has multiple near-equivalent local optima.
| Metric | Sonnet 4.0 (Mar 20) | Sonnet 4.6 (Mar 24) | Winner |
|---|---|---|---|
| Baseline val_bpb | 0.9640 (pre-optimized) | 0.9806 (defaults) | S4.0 starts better |
| Best val_bpb | 0.9606 | 0.9549 | Sonnet 4.6 (by 0.006) |
| Improvement | −0.35% | −2.63% | Sonnet 4.6 (7.5× more) |
| Keeps | 4 / 102 (3.9%) | 16 / 100 (16.0%) | Sonnet 4.6 (4× more) |
| Crashes | 2 / 102 (1.9%) | 0 / 100 (0.0%) | Sonnet 4.6 |
| Exps to first keep | 98 | 5 | Sonnet 4.6 (20× faster) |
| Parameters optimized | 4 | 10 (all different) | Sonnet 4.6 |
This is Sonnet 4.6's strongest showing. It starts from a worse baseline and still ends with a significantly better result. Zero crashes. First keep at exp5 (vs exp98 for Sonnet 4.0). And the architecture divergence is striking — Sonnet 4.6 found ASPECT_RATIO=21 while every Sonnet 4.0 dataset converged to AR=32.
Sonnet 4.6 systematically walked ASPECT_RATIO down from the default:
| ASPECT_RATIO | val_bpb | Steps | tok/sec | Outcome |
|---|---|---|---|---|
| 32 (default) | 0.9806 | 437 | 47.6K | Baseline |
| 24 (exp28) | 0.9662 | 480 | 52.2K | Keep |
| 22 (exp59) | 0.9600 | 510 | 55.6K | Keep |
| 21 (exp61) | 0.9595 | 511 | 55.6K | Keep |
| 20 (exp58) | 0.9686 | 480 | 52.3K | Discard |
AR=20 was too narrow (tried before 22 and 21). The sweet spot is AR=21–22, where the model gets 17% more gradient steps than AR=32 while staying wide enough to maintain representation capacity. Sonnet 4.0 never explored below AR=32 on any of five datasets.
This means the "AR=32 is hardware-optimal" conclusion from the Sonnet 4.0 cross-dataset comparison was actually a local optimum — there's a better one at AR=21, at least for Cosmopedia-v2's synthetic textbook data.
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 21 | ❌ Major divergence |
| MATRIX_LR | 0.0435 | 0.068 | ❌ (56% higher) |
| EMBEDDING_LR | 0.4 | 0.50 | ❌ |
| SCALAR_LR | 0.4 | 0.13 | ❌ (68% lower) |
| UNEMBEDDING_LR | 0.0033 | 0.006 | ❌ (82% higher) |
| WEIGHT_DECAY | 0.15 | 0.1 | ❌ |
| WARMDOWN_RATIO | 0.5 | 0.66 | ❌ |
| FINAL_LR_FRAC | 0.0 | 0.02 | ❌ |
| ADAM β1 | 0.8 | 0.65 | ❌ |
| ADAM β2 | 0.95 | 0.955 | ≈ |
Every single parameter differs. Unlike FineWeb-Edu where both models found equivalent val_bpb via different paths, here Sonnet 4.6's path leads to a definitively better result. The AR=21 architecture is genuinely superior for Cosmopedia-v2 — more gradient steps in the time budget with sufficient model width for synthetic textbook data.
| Dimension | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Architecture | Never explored below AR=32 | Systematic walk: 32→24→22→21 |
| First keep | exp98 (near end of run) | exp5 (immediate) |
| LR re-tuning | Static LRs throughout | Re-tuned after each AR change |
| SCALAR_LR | Kept at 0.4 (default) | Walked down: 0.23→0.18→0.13 |
| β2 exploration | Never explored | Fine-tuned 0.95→0.955 |
| Crash rate | 1.9% | 0.0% |
Sonnet 4.6's key advantage was architectural exploration. While Sonnet 4.0 accepted AR=32 and focused on fine-tuning within that architecture, Sonnet 4.6 systematically tested narrower models and re-tuned all learning rates for each new architecture. This is a fundamentally more thorough optimization strategy.
| Metric | Sonnet 4.0 (Mar 20) | Sonnet 4.6 (Mar 24) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.5410 (pre-optimized) | 1.5312 (defaults) | S4.6 starts better |
| Best val_bpb | 1.5259 | 1.5267 | Sonnet 4.0 (by 0.0008) |
| Improvement | −1.0% | −0.29% | Sonnet 4.0 (3.4× more) |
| Keeps | 3 / 100 (3.0%) | 2 / 100 (2.0%) | Sonnet 4.0 |
| Crashes | 0 / 100 (0.0%) | 2 / 100 (2.0%) | Sonnet 4.0 |
| Exps to first keep | Not available | 63 | — |
| Parameters optimized | 3 | 3 | Tied |
Sonnet 4.0's first clear win. SlimPajama is the hardest dataset for both models — the optimization landscape is extremely flat, with both models achieving minimal improvement from baseline. Sonnet 4.0 achieves a slightly better absolute result (1.5259 vs 1.5267) with more keeps and zero crashes.
What Each Model Found:
Sonnet 4.0 best:
ASPECT_RATIO = 32, MATRIX_LR = 0.0435, EMBEDDING_LR = 0.4
SCALAR_LR = 0.4, WEIGHT_DECAY = 0.15, WARMDOWN_RATIO = 0.5
# Result: 1.5259 | mem: 13.7 GB | MFU: 16.9%Sonnet 4.6 best (exp91):
ASPECT_RATIO = 21, MATRIX_LR = 0.068, EMBEDDING_LR = 0.50
SCALAR_LR = 0.13, WEIGHT_DECAY = 0.1, WARMDOWN_RATIO = 0.70
FINAL_LR_FRAC = 0.01, ADAM_BETAS = (0.55, 0.945)
# Result: 1.5267 | mem: 14.5 GB | MFU: 15.6%SlimPajama contains data from 7 diverse sources (CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, StackExchange). This creates an extremely broad distribution that's hard to optimize for — any hyperparameter change that helps one source type may hurt another. The result is a flat optimization landscape where the baseline is already near-optimal for this balanced mixture.
Key observations:
- Both models found only 2–3 keeps — the lowest for any dataset
- Sonnet 4.6 struggled with reproducibility — repeated attempts to replicate the baseline result failed (exp36–exp49)
- Training variance dominates — the 0.0008 difference between the two best results is within noise
| Dataset | Sonnet 4.0 | Sonnet 4.6 | Winner |
|---|---|---|---|
| ClimbMix | 11.0% | 4.2% | S4.6 (2.6× fewer) |
| FineWeb-Edu | 2.3% | 1.0% | S4.6 (2.3× fewer) |
| Cosmopedia-v2 | 1.9% | 0.0% | S4.6 (zero crashes) |
| SlimPajama | 0.0% | 2.0% | S4.0 (zero crashes) |
Sonnet 4.6 crashes less on 3 of 4 datasets, but SlimPajama breaks the pattern — Sonnet 4.0 achieved a perfect zero-crash run while Sonnet 4.6 had 2 crashes (both from aggressive batch size changes).
| Dataset | Sonnet 4.0 params changed | Sonnet 4.6 params changed |
|---|---|---|
| ClimbMix | 1 (UNEMBEDDING_LR) | 3 (WARMDOWN, SCALAR_LR, β1) |
| FineWeb-Edu | 6 | 9 |
| Cosmopedia-v2 | 4 | 10 (all parameters) |
| SlimPajama | 3 | 3 |
Sonnet 4.6 generally explores more parameter dimensions, but on SlimPajama both models changed the same number of parameters — the flat landscape limited both equally.
| Dataset | Sonnet 4.0 AR | Sonnet 4.6 AR | Agreement? |
|---|---|---|---|
| ClimbMix | 32 | 32 (via defaults) | ✅ |
| FineWeb-Edu | 32 | 32 | ✅ |
| Cosmopedia-v2 | 32 | 21 | ❌ |
| SlimPajama | 32 | 21 (inherited default) | ❌ (but S4.6 didn't choose it) |
Note: Sonnet 4.6's SlimPajama run inherited AR=21 from its Cosmopedia-v2 defaults. The agent tested AR=14, 16, 24, and 28 but all were worse — confirming AR=21 is at least locally optimal for SlimPajama too, though it didn't deliver better results than Sonnet 4.0's AR=32.
For absolute performance: it depends on the dataset. Sonnet 4.6 wins decisively on Cosmopedia-v2 via architecture exploration, ties on FineWeb-Edu and ClimbMix, but loses on SlimPajama. The score is 2-1-1 — neither model dominates.
For optimization efficiency: mixed. Sonnet 4.6 finds improvements faster on datasets where improvement exists (first keep at exp5 on Cosmopedia-v2 vs exp98 for S4.0). But on flat landscapes like SlimPajama, the extra exploration doesn't translate to better results.
For scientific insight: absolutely. Sonnet 4.6's AR=21 discovery on Cosmopedia-v2 overturns the "AR=32 is hardware-optimal" conclusion. And SlimPajama's near-identical results from two models with completely different configs confirms that the optimization landscape for diverse multi-source data is genuinely flat — not just hard for one particular LLM.
For robustness: Sonnet 4.0 wins on SlimPajama. Sonnet 4.6 spent 14 experiments (exp36–49) unable to reproduce its own baseline, suggesting sensitivity to training variance. Sonnet 4.0 never exhibited this issue.
| Dataset | Sonnet 4.0 Status | Sonnet 4.6 Status |
|---|---|---|
| climbmix | ✅ 101 experiments, 1 keep | ✅ 119 experiments, 8 keeps |
| fineweb-edu | ✅ 88 experiments, 17 keeps | ✅ 100 experiments, 18 keeps |
| cosmopedia-v2 | ✅ 103 experiments, 4 keeps | ✅ 101 experiments, 16 keeps |
| slimpajama | ✅ 101 experiments, 3 keeps | ✅ 101 experiments, 2 keeps |
| fineweb-edu-high | ✅ 101 experiments, 20 keeps | 🔲 Planned |
-
Does Sonnet 4.6's advantage hold for educational text?Answered: Yes, marginally. Sonnet 4.6 achieves slightly better absolute val_bpb (1.3416 vs 1.3424) with comparable keep rate. -
Will Sonnet 4.6 discover the same two-cluster split?Partially answered. Sonnet 4.6 found very different configs for FineWeb-Edu vs ClimbMix, confirming the split. But each config differs from Sonnet 4.0's — so the "cluster" is a basin rather than a point. -
Does the multi-optima phenomenon extend to general text?Answered: Yes, dramatically. Cosmopedia-v2 has multiple optima at different depths (0.9549 vs 0.9606). SlimPajama also has multiple optima but at the same depth (1.5259 vs 1.5267). -
Can we combine insights from both models? Sonnet 4.0 found that low weight decay helps FineWeb-Edu; Sonnet 4.6 found that low momentum and high β2 help. Could a combined configuration beat both?
-
Is AR=21 dataset-specific or general?Partially answered. AR=21 works well for Cosmopedia-v2 (synthetic textbooks) and is at least locally optimal for SlimPajama (inherited, tested, not beaten). But AR=32 won on ClimbMix and FineWeb-Edu. The optimal AR may depend on data homogeneity — Cosmopedia-v2 (single-source synthetic) benefits from depth, while diverse/educational text benefits from width. -
Will FineWeb-Edu-High reveal the tiebreaker? The current score is 2-1-1. FineWeb-Edu-High had Sonnet 4.0's highest keep rate (20%) — can Sonnet 4.6 match it? This is the last dataset and will determine the overall series outcome.
See individual run pages: Sonnet 4.6 SlimPajama (Mar 24) | Sonnet 4.6 Cosmopedia-v2 (Mar 24) | Sonnet 4.6 FineWeb-Edu (Mar 22) | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 SlimPajama (Mar 20) | Sonnet 4.0 Cosmopedia-v2 (Mar 20) | Sonnet 4.0 FineWeb-Edu (Mar 17) | Sonnet 4.0 ClimbMix (Mar 19) | Cross-Dataset Comparison