Cross LLM Comparison
How much does the LLM generation matter for autonomous hyperparameter optimization? This page tracks cross-generation comparison experiments: the same dataset, same hardware, same codebase, same training budget — but different Claude model versions driving the optimization.
| Metric | ClimbMix | FineWeb-Edu | Cosmopedia-v2 |
|---|---|---|---|
| S4.0 baseline | 1.2969 (pre-optimized) | 1.4088 (defaults) | 0.9640 (pre-optimized) |
| S4.6 baseline | 1.3213 (defaults) | 1.3710 (defaults) | 0.9806 (defaults) |
| S4.0 best | 1.2959 | 1.3424 | 0.9606 |
| S4.6 best | 1.2997 | 1.3416 | 0.9549 |
| S4.0 improvement | −0.08% | −4.71% | −0.35% |
| S4.6 improvement | −1.63% | −2.14% | −2.63% |
| S4.0 keeps | 1 (1.0%) | 17 (19.3%) | 4 (3.9%) |
| S4.6 keeps | 8 (6.8%) | 18 (18.0%) | 16 (16.0%) |
| S4.0 crashes | 11 (11.0%) | 2 (2.3%) | 2 (1.9%) |
| S4.6 crashes | 5 (4.2%) | 1 (1.0%) | 0 (0.0%) |
The headline: Cosmopedia-v2 is the first dataset where Sonnet 4.6 unambiguously wins — both in absolute val_bpb (0.9549 vs 0.9606) and in optimization depth (16 keeps vs 4). More importantly, Sonnet 4.6 discovered ASPECT_RATIO=21, breaking the universal AR=32 consensus from all five Sonnet 4.0 datasets. The "AR=32 is hardware-optimal" conclusion was an artifact of the LLM's exploration strategy, not a true hardware constraint.
| Metric | Sonnet 4.0 (Mar 19) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.2969 (pre-optimized) | 1.3213 (defaults) | Different starting points |
| Best val_bpb | 1.2959 | 1.2997 | Sonnet 4.0 (absolute) |
| Improvement | −0.08% | −1.63% | Sonnet 4.6 (20× more) |
| Keeps | 1 / 100 (1.0%) | 8 / 118 (6.8%) | Sonnet 4.6 (7× more) |
| Crashes | 11 / 100 (11.0%) | 5 / 118 (4.2%) | Sonnet 4.6 (2.6× fewer) |
| Exps to first keep | 84 | 25 | Sonnet 4.6 (3.4× faster) |
| Parameters optimized | 1 | 3 (synergistic) | Sonnet 4.6 |
What Each Model Found (click to expand)
Sonnet 4.0's single keep (exp84):
UNEMBEDDING_LR = 0.0041 # was 0.004 — a 2.5% tweak
# Result: 1.2959 (−0.0010 from 1.2969 baseline)Sonnet 4.6's optimal configuration (exp112):
WARMDOWN_RATIO = 0.63 # was 0.5 — 26% longer annealing
SCALAR_LR = 0.3 # was 0.51 — 41% lower
ADAM_BETAS = (0.71, 0.95) # was (0.8, 0.95) — 11% lower momentum
# Result: 1.2997 (−0.0216 from 1.3213 baseline)| Metric | Sonnet 4.0 (Mar 17) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.4088 (defaults) | 1.3710 (defaults) | S4.6 starts better |
| Best val_bpb | 1.3424 | 1.3416 | Sonnet 4.6 (by 0.0008) |
| Improvement | −4.71% | −2.14% | Sonnet 4.0 (more room) |
| Keeps | 17 / 87 (19.3%) | 18 / 99 (18.0%) | Comparable |
| Crashes | 2 / 87 (2.3%) | 1 / 99 (1.0%) | Sonnet 4.6 |
| Exps to first keep | 3 | 3 | Tied |
| Parameters optimized | 6 | 9 | Sonnet 4.6 |
Configuration Comparison (click to expand)
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 32 | ✅ |
| MATRIX_LR | 0.053 | 0.055 | ≈ |
| EMBEDDING_LR | 0.59 | 0.45 | ❌ |
| SCALAR_LR | 0.3 | 0.23 | ❌ |
| UNEMBEDDING_LR | 0.003 | 0.006 | ❌ (2× different) |
| WEIGHT_DECAY | 0.12 | 0.2 | ❌ |
| WARMDOWN_RATIO | 0.47 | 0.77 | ❌ (64% different) |
| FINAL_LR_FRAC | 0.0 | 0.1 | ❌ |
| ADAM β1 | 0.8 | 0.67 | ❌ |
| ADAM β2 | 0.95 | 0.975 | ❌ |
Both models converge to essentially the same val_bpb (within 0.0008) via completely different configurations. Only architecture (AR=32) and matrix LR agree — proving that FineWeb-Edu's optimization landscape has multiple near-equivalent local optima.
| Metric | Sonnet 4.0 (Mar 20) | Sonnet 4.6 (Mar 24) | Winner |
|---|---|---|---|
| Baseline val_bpb | 0.9640 (pre-optimized) | 0.9806 (defaults) | S4.0 starts better |
| Best val_bpb | 0.9606 | 0.9549 | Sonnet 4.6 (by 0.006) |
| Improvement | −0.35% | −2.63% | Sonnet 4.6 (7.5× more) |
| Keeps | 4 / 102 (3.9%) | 16 / 100 (16.0%) | Sonnet 4.6 (4× more) |
| Crashes | 2 / 102 (1.9%) | 0 / 100 (0.0%) | Sonnet 4.6 |
| Exps to first keep | 98 | 5 | Sonnet 4.6 (20× faster) |
| Parameters optimized | 4 | 10 (all different) | Sonnet 4.6 |
This is Sonnet 4.6's strongest showing. It starts from a worse baseline and still ends with a significantly better result. Zero crashes. First keep at exp5 (vs exp98 for Sonnet 4.0). And the architecture divergence is striking — Sonnet 4.6 found ASPECT_RATIO=21 while every Sonnet 4.0 dataset converged to AR=32.
Sonnet 4.6 systematically walked ASPECT_RATIO down from the default:
| ASPECT_RATIO | val_bpb | Steps | tok/sec | Outcome |
|---|---|---|---|---|
| 32 (default) | 0.9806 | 437 | 47.6K | Baseline |
| 24 (exp28) | 0.9662 | 480 | 52.2K | Keep |
| 22 (exp59) | 0.9600 | 510 | 55.6K | Keep |
| 21 (exp61) | 0.9595 | 511 | 55.6K | Keep |
| 20 (exp58) | 0.9686 | 480 | 52.3K | Discard |
AR=20 was too narrow (tried before 22 and 21). The sweet spot is AR=21–22, where the model gets 17% more gradient steps than AR=32 while staying wide enough to maintain representation capacity. Sonnet 4.0 never explored below AR=32 on any of five datasets.
This means the "AR=32 is hardware-optimal" conclusion from the Sonnet 4.0 cross-dataset comparison was actually a local optimum — there's a better one at AR=21, at least for Cosmopedia-v2's synthetic textbook data.
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 21 | ❌ Major divergence |
| MATRIX_LR | 0.0435 | 0.068 | ❌ (56% higher) |
| EMBEDDING_LR | 0.4 | 0.50 | ❌ |
| SCALAR_LR | 0.4 | 0.13 | ❌ (68% lower) |
| UNEMBEDDING_LR | 0.0033 | 0.006 | ❌ (82% higher) |
| WEIGHT_DECAY | 0.15 | 0.1 | ❌ |
| WARMDOWN_RATIO | 0.5 | 0.66 | ❌ |
| FINAL_LR_FRAC | 0.0 | 0.02 | ❌ |
| ADAM β1 | 0.8 | 0.65 | ❌ |
| ADAM β2 | 0.95 | 0.955 | ≈ |
Every single parameter differs. Unlike FineWeb-Edu where both models found equivalent val_bpb via different paths, here Sonnet 4.6's path leads to a definitively better result. The AR=21 architecture is genuinely superior for Cosmopedia-v2 — more gradient steps in the time budget with sufficient model width for synthetic textbook data.
| Dimension | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Architecture | Never explored below AR=32 | Systematic walk: 32→24→22→21 |
| First keep | exp98 (near end of run) | exp5 (immediate) |
| LR re-tuning | Static LRs throughout | Re-tuned after each AR change |
| SCALAR_LR | Kept at 0.4 (default) | Walked down: 0.23→0.18→0.13 |
| β2 exploration | Never explored | Fine-tuned 0.95→0.955 |
| Crash rate | 1.9% | 0.0% |
Sonnet 4.6's key advantage was architectural exploration. While Sonnet 4.0 accepted AR=32 and focused on fine-tuning within that architecture, Sonnet 4.6 systematically tested narrower models and re-tuned all learning rates for each new architecture. This is a fundamentally more thorough optimization strategy.
| Dataset | Sonnet 4.0 | Sonnet 4.6 | Reduction |
|---|---|---|---|
| ClimbMix | 11.0% | 4.2% | 2.6× fewer |
| FineWeb-Edu | 2.3% | 1.0% | 2.3× fewer |
| Cosmopedia-v2 | 1.9% | 0.0% | ∞ (zero crashes) |
Sonnet 4.6 crashes less across all datasets, and achieved a perfect zero-crash run on Cosmopedia-v2.
| Dataset | Sonnet 4.0 params changed | Sonnet 4.6 params changed |
|---|---|---|
| ClimbMix | 1 (UNEMBEDDING_LR) | 3 (WARMDOWN, SCALAR_LR, β1) |
| FineWeb-Edu | 6 | 9 |
| Cosmopedia-v2 | 4 | 10 (all parameters) |
Sonnet 4.6 consistently explores more parameter dimensions and builds compositional improvements. On Cosmopedia-v2, it changed every single parameter from default — including architecture — while Sonnet 4.0 changed only 4.
| Dataset | Sonnet 4.0 AR | Sonnet 4.6 AR | Agreement? |
|---|---|---|---|
| ClimbMix | 32 | 32 (via defaults) | ✅ |
| FineWeb-Edu | 32 | 32 | ✅ |
| Cosmopedia-v2 | 32 | 21 | ❌ |
The Cosmopedia-v2 AR divergence is the first time the two models disagree on architecture. This challenges the universal AR=32 finding from Sonnet 4.0 and suggests that AR=32 may be a local optimum — good enough that Sonnet 4.0 never explored past it, but not globally optimal.
For absolute performance: increasingly yes. On ClimbMix and FineWeb-Edu, both models reached similar val_bpb. But on Cosmopedia-v2, Sonnet 4.6 found a definitively better result (0.9549 vs 0.9606) by discovering a superior architecture that Sonnet 4.0 never explored.
For optimization efficiency: significantly. Sonnet 4.6 finds improvements faster (first keep at exp5 vs exp98 on Cosmopedia-v2), crashes less (zero crashes vs 1.9%), and explores more of the parameter space (10 parameters changed vs 4).
For scientific insight: absolutely. Sonnet 4.6's AR=21 discovery on Cosmopedia-v2 overturns the "AR=32 is hardware-optimal" conclusion from all five Sonnet 4.0 datasets. A single LLM's exploration strategy can create false consensus — cross-LLM comparison reveals what a single model misses.
| Dataset | Sonnet 4.0 Status | Sonnet 4.6 Status |
|---|---|---|
| climbmix | ✅ 101 experiments, 1 keep | ✅ 119 experiments, 8 keeps |
| fineweb-edu | ✅ 88 experiments, 17 keeps | ✅ 100 experiments, 18 keeps |
| cosmopedia-v2 | ✅ 103 experiments, 4 keeps | ✅ 101 experiments, 16 keeps |
| slimpajama | ✅ 101 experiments, 3 keeps | 🔲 Planned |
| fineweb-edu-high | ✅ 101 experiments, 20 keeps | 🔲 Planned |
-
Does Sonnet 4.6's advantage hold for educational text?Answered: Yes, marginally. Sonnet 4.6 achieves slightly better absolute val_bpb (1.3416 vs 1.3424) with comparable keep rate (18% vs 19.3%), fewer crashes, and deeper parameter exploration. -
Will Sonnet 4.6 discover the same two-cluster split?Partially answered. Sonnet 4.6 found very different configs for FineWeb-Edu vs ClimbMix, confirming the split. But each config differs from Sonnet 4.0's — so the "cluster" is a basin rather than a point. -
Does the multi-optima phenomenon extend to general text?Answered: Yes, dramatically. Cosmopedia-v2 has multiple optima — and unlike FineWeb-Edu where both models reached the same depth, here the optima are at different depths (0.9549 vs 0.9606). Sonnet 4.6 found a genuinely better one via architecture exploration. -
Can we combine insights from both models? Sonnet 4.0 found that low weight decay helps FineWeb-Edu; Sonnet 4.6 found that low momentum and high β2 help. Could a configuration combining both (WD=0.12, β1=0.67, β2=0.975) beat both models' results?
-
Is Sonnet 4.6's β2 discovery transferable? The systematic β2 walk was never attempted by Sonnet 4.0 on any dataset. Testing β2 > 0.95 across all datasets could unlock additional improvements.
-
Is AR=21 dataset-specific or general? Sonnet 4.6 found AR=21 optimal for Cosmopedia-v2 but kept AR=32 for ClimbMix and FineWeb-Edu. Will SlimPajama and FineWeb-Edu-High stay at AR=32 or discover narrower architectures? This will determine whether the AR optimum is data-dependent or whether Sonnet 4.0 simply failed to explore it.
See individual run pages: Sonnet 4.6 Cosmopedia-v2 (Mar 24) | Sonnet 4.6 FineWeb-Edu (Mar 22) | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 Cosmopedia-v2 (Mar 20) | Sonnet 4.0 FineWeb-Edu (Mar 17) | Sonnet 4.0 ClimbMix (Mar 19) | Cross-Dataset Comparison