Cross LLM Comparison
How much does the LLM generation matter for autonomous hyperparameter optimization? This page tracks cross-generation comparison experiments: the same dataset, same hardware, same codebase, same training budget — but different Claude model versions driving the optimization.
| Metric | ClimbMix | FineWeb-Edu |
|---|---|---|
| S4.0 baseline | 1.2969 (pre-optimized) | 1.4088 (defaults) |
| S4.6 baseline | 1.3213 (defaults) | 1.3710 (defaults) |
| S4.0 best | 1.2959 | 1.3424 |
| S4.6 best | 1.2997 | 1.3416 |
| S4.0 improvement | −0.08% | −4.71% |
| S4.6 improvement | −1.63% | −2.14% |
| S4.0 keeps | 1 (1.0%) | 17 (19.3%) |
| S4.6 keeps | 8 (6.8%) | 18 (18.0%) |
| S4.0 crashes | 11 (11.0%) | 2 (2.3%) |
| S4.6 crashes | 5 (4.2%) | 1 (1.0%) |
The headline: On both datasets, Sonnet 4.6 achieves comparable or better absolute val_bpb, with fewer crashes and compositional optimization strategies that Sonnet 4.0 lacks. The models converge to different configurations that produce nearly identical results — especially on FineWeb-Edu where the final bpb differs by only 0.0008.
| Metric | Sonnet 4.0 (Mar 19) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.2969 (pre-optimized) | 1.3213 (defaults) | Different starting points |
| Best val_bpb | 1.2959 | 1.2997 | Sonnet 4.0 (absolute) |
| Improvement | −0.08% | −1.63% | Sonnet 4.6 (20× more) |
| Keeps | 1 / 100 (1.0%) | 8 / 118 (6.8%) | Sonnet 4.6 (7× more) |
| Crashes | 11 / 100 (11.0%) | 5 / 118 (4.2%) | Sonnet 4.6 (2.6× fewer) |
| Exps to first keep | 84 | 25 | Sonnet 4.6 (3.4× faster) |
| Parameters optimized | 1 | 3 (synergistic) | Sonnet 4.6 |
What Each Model Found (click to expand)
Sonnet 4.0's single keep (exp84):
UNEMBEDDING_LR = 0.0041 # was 0.004 — a 2.5% tweak
# Result: 1.2959 (−0.0010 from 1.2969 baseline)Sonnet 4.6's optimal configuration (exp112):
WARMDOWN_RATIO = 0.63 # was 0.5 — 26% longer annealing
SCALAR_LR = 0.3 # was 0.51 — 41% lower
ADAM_BETAS = (0.71, 0.95) # was (0.8, 0.95) — 11% lower momentum
# Result: 1.2997 (−0.0216 from 1.3213 baseline)| Metric | Sonnet 4.0 (Mar 17) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.4088 (defaults) | 1.3710 (defaults) | S4.6 starts better |
| Best val_bpb | 1.3424 | 1.3416 | Sonnet 4.6 (by 0.0008) |
| Improvement | −4.71% | −2.14% | Sonnet 4.0 (more room) |
| Keeps | 17 / 87 (19.3%) | 18 / 99 (18.0%) | Comparable |
| Crashes | 2 / 87 (2.3%) | 1 / 99 (1.0%) | Sonnet 4.6 |
| Exps to first keep | 3 | 3 | Tied |
| Parameters optimized | 6 | 9 | Sonnet 4.6 |
What Each Model Found:
Sonnet 4.0 best (exp75):
ASPECT_RATIO = 32, MATRIX_LR = 0.053, EMBEDDING_LR = 0.59
UNEMBEDDING_LR = 0.003, SCALAR_LR = 0.3, WEIGHT_DECAY = 0.12
WARMDOWN_RATIO = 0.47
# Result: 1.3424 | mem: 15.5 GB | MFU: 19.3%Sonnet 4.6 best (exp98):
ASPECT_RATIO = 32, MATRIX_LR = 0.055, EMBEDDING_LR = 0.45
UNEMBEDDING_LR = 0.006, SCALAR_LR = 0.23, WEIGHT_DECAY = 0.2
WARMDOWN_RATIO = 0.77, FINAL_LR_FRAC = 0.1, ADAM_BETAS = (0.67, 0.975)
# Result: 1.3416 | mem: 14.5 GB | MFU: 17.9%This is the most important result of the cross-LLM comparison so far. Both models converge to essentially the same val_bpb (within 0.0008) but via completely different configurations:
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 32 | ✅ |
| MATRIX_LR | 0.053 | 0.055 | ≈ |
| EMBEDDING_LR | 0.59 | 0.45 | ❌ |
| SCALAR_LR | 0.3 | 0.23 | ❌ |
| UNEMBEDDING_LR | 0.003 | 0.006 | ❌ (2× different) |
| WEIGHT_DECAY | 0.12 | 0.2 | ❌ |
| WARMDOWN_RATIO | 0.47 | 0.77 | ❌ (64% different) |
| FINAL_LR_FRAC | 0.0 | 0.1 | ❌ |
| ADAM β1 | 0.8 | 0.67 | ❌ |
| ADAM β2 | 0.95 | 0.975 | ❌ |
Only architecture (AR=32) and matrix LR agree. The warmdown ratios differ by 64%. The weight decay values differ by 67%. The Adam betas diverge in opposite directions. Yet both produce 1.342x val_bpb.
This proves that FineWeb-Edu's optimization landscape contains multiple near-equivalent local optima — a flat basin with multiple valleys at the same depth. The LLM's optimization strategy determines which optimum it finds, but not the quality of the result.
| Dimension | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Architecture discovery | Found AR=32 at exp34 (late breakthrough) | Found AR=32 at exp32 (systematic search) |
| LR schedule | Short warmdown (0.47), no final LR | Long warmdown (0.77) + residual final LR (0.1) |
| Regularization | Lower weight decay (0.12) | Default weight decay (0.2) |
| Momentum | Default momentum (β1=0.8) | Low momentum (β1=0.67) + high β2 (0.975) |
| Compensation strategy | Low WD + default momentum | Default WD + low momentum |
| β2 exploration | Never explored | Systematic walk: 0.95→0.96→0.97→0.98→0.975 (4 consecutive keeps) |
| Crash rate | 2.3% | 1.0% |
The two models found complementary compensation strategies: Sonnet 4.0 compensates for educational text's diversity with lower regularization, while Sonnet 4.6 compensates with lower momentum. Both achieve the same effective behavior — the model can fit diverse patterns — but through different mechanisms.
| Dataset | Sonnet 4.0 | Sonnet 4.6 | Reduction |
|---|---|---|---|
| ClimbMix | 11.0% | 4.2% | 2.6× fewer |
| FineWeb-Edu | 2.3% | 1.0% | 2.3× fewer |
Sonnet 4.6 consistently produces fewer crashes. On ClimbMix it learned from batch size failures quickly; on FineWeb-Edu it only crashed once (a DEVICE_BATCH_SIZE / 3 integer division issue).
| Dataset | Sonnet 4.0 params changed | Sonnet 4.6 params changed |
|---|---|---|
| ClimbMix | 1 (UNEMBEDDING_LR) | 3 (WARMDOWN, SCALAR_LR, β1) |
| FineWeb-Edu | 6 | 9 |
Sonnet 4.6 consistently explores more parameter dimensions and builds compositional improvements. On FineWeb-Edu, it changed 9 parameters (including FINAL_LR_FRAC and both Adam betas), while Sonnet 4.0 changed 6 (never touching betas or final LR).
For absolute performance: barely. Both models converge to similar val_bpb on both datasets. The optimization landscape seems to have a natural floor that both models can reach.
For optimization efficiency: significantly. Sonnet 4.6 finds improvements faster (first keep at exp25 vs exp84 on ClimbMix), crashes less (2.6× fewer on ClimbMix, 2.3× fewer on FineWeb-Edu), and explores more of the parameter space.
For scientific insight: yes. Sonnet 4.6's divergent FineWeb-Edu configuration reveals that the educational text optimization landscape is fundamentally different from ClimbMix — it has multiple near-equivalent optima, while general text datasets converge to a single optimum. Sonnet 4.0's single path couldn't reveal this.
| Dataset | Sonnet 4.0 Status | Sonnet 4.6 Status |
|---|---|---|
| climbmix | ✅ 101 experiments, 1 keep | ✅ 119 experiments, 8 keeps |
| fineweb-edu | ✅ 88 experiments, 17 keeps | ✅ 100 experiments, 18 keeps |
| cosmopedia-v2 | ✅ 103 experiments, 4 keeps | 🔲 Planned |
| slimpajama | ✅ 101 experiments, 3 keeps | 🔲 Planned |
| fineweb-edu-high | ✅ 101 experiments, 20 keeps | 🔲 Planned |
-
Does Sonnet 4.6's advantage hold for educational text?Answered: Yes, marginally. Sonnet 4.6 achieves slightly better absolute val_bpb (1.3416 vs 1.3424) with comparable keep rate (18% vs 19.3%), fewer crashes, and deeper parameter exploration. -
Will Sonnet 4.6 discover the same two-cluster split?Partially answered. Sonnet 4.6 found a very different FineWeb-Edu config than its own ClimbMix config, confirming educational text needs different parameters. But the specific config differs from Sonnet 4.0's FineWeb-Edu config — so the "cluster" may be a basin rather than a point. -
Does the multi-optima phenomenon extend to general text? FineWeb-Edu clearly has multiple equivalent optima. Do ClimbMix, Cosmopedia-v2, and SlimPajama also? The Cosmopedia-v2 comparison will test this.
-
Can we combine insights from both models? Sonnet 4.0 found that low weight decay helps FineWeb-Edu; Sonnet 4.6 found that low momentum and high β2 help. Could a configuration combining both (WD=0.12, β1=0.67, β2=0.975) beat both models' results?
-
Is Sonnet 4.6's β2 discovery transferable? The systematic β2 walk (0.95→0.975, 4 consecutive keeps) was never attempted by Sonnet 4.0 on any dataset. Testing β2 > 0.95 across all datasets could unlock additional improvements.
See individual run pages: Sonnet 4.6 FineWeb-Edu (Mar 22) | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 FineWeb-Edu (Mar 17) | Sonnet 4.0 ClimbMix (Mar 19) | Cross-Dataset Comparison