Cross LLM Comparison
How much does the LLM generation matter for autonomous hyperparameter optimization? This page tracks cross-generation comparison experiments: the same dataset, same hardware, same codebase, same training budget — but different Claude model versions driving the optimization.
Stock Baseline Comparison — ClimbMix Complete (3 models). All models start from identical stock upstream defaults (AR=64). Haiku 4.5 wins ClimbMix (1.2953) via architecture discovery (AR=32, depth reduction, LLLL). Sonnet 4.6 places 2nd (1.3093) via systematic AR walk to 19 and SLLL window pattern. Sonnet 4.0 finishes 3rd (1.3588) — architecturally blind, never changed AR from 64.
Key insight: Architecture > Optimizer Tuning. Models that discovered architecture changes (smaller AR, reduced depth) gained 2–3× more gradient steps and dramatically better results. S4.0's perfect optimizer tuning on 251 steps couldn't compete with Haiku's 697 steps.
Legacy: Sonnet 4.0 vs 4.6 (pre-optimized): Complete (5/5 datasets). Sonnet 4.6 wins 3–2 on absolute val_bpb from pre-optimized baselines.
| Metric | ClimbMix | FineWeb-Edu | Cosmopedia-v2 | SlimPajama | FineWeb-Edu-High |
|---|---|---|---|---|---|
| S4.0 baseline | 1.2969 | 1.4088 | 0.9640 | 1.5410 | 1.3730 |
| S4.6 baseline | 1.3213 | 1.3710 | 0.9806 | 1.5312 | 1.3478 |
| S4.0 best | 1.2959 | 1.3424 | 0.9606 | 1.5259 | 1.3463 |
| S4.6 best | 1.2997 | 1.3416 | 0.9549 | 1.5267 | 1.3345 |
| S4.0 improvement | −0.08% | −4.71% | −0.35% | −1.0% | −1.97% |
| S4.6 improvement | −1.63% | −2.14% | −2.63% | −0.30% | −0.99% |
| S4.0 keeps | 1 | 17 | 4 | 3 | 20 |
| S4.6 keeps | 8 | 18 | 16 | 2 | 13 |
| S4.0 crashes | 11 | 2 | 2 | 0 | 5 |
| S4.6 crashes | 5 | 1 | 0 | 2 | 2 |
| Winner (absolute) | S4.0 | S4.6 | S4.6 | S4.0 | S4.6 |
Final scorecard: Sonnet 4.6 wins 3–2 on absolute val_bpb (FineWeb-Edu, Cosmopedia-v2, FineWeb-Edu-High). Sonnet 4.0 wins on ClimbMix and SlimPajama — both general text datasets where it started from pre-optimized baselines.
Note: The Sonnet runs used inconsistent baselines (S4.0 started from pre-optimized configs, S4.6 from partially-reset defaults). Future model comparisons (Haiku, Opus) will use stock upstream defaults for fair comparison — see methodology.
| Metric | Sonnet 4.0 (Mar 19) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Best val_bpb | 1.2959 | 1.2997 | Sonnet 4.0 |
| Improvement | −0.08% | −1.63% | Sonnet 4.6 (20×) |
| Keeps | 1 (1.0%) | 8 (6.8%) | Sonnet 4.6 |
| Crashes | 11 (11.0%) | 5 (4.2%) | Sonnet 4.6 |
Sonnet 4.0 wins on absolute val_bpb (pre-optimized baseline advantage), but Sonnet 4.6 found 20× more improvement and 7× more keeps.
| Metric | Sonnet 4.0 (Mar 17) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Best val_bpb | 1.3424 | 1.3416 | Sonnet 4.6 (by 0.0008) |
| Improvement | −4.71% | −2.14% | Sonnet 4.0 (more room) |
| Keeps | 17 (19.3%) | 18 (18.0%) | Comparable |
| Crashes | 2 (2.3%) | 1 (1.0%) | Sonnet 4.6 |
Near-identical results via completely different configs — 8 of 10 parameters differ. Proves FineWeb-Edu has multiple near-equivalent optima.
| Metric | Sonnet 4.0 (Mar 20) | Sonnet 4.6 (Mar 24) | Winner |
|---|---|---|---|
| Best val_bpb | 0.9606 | 0.9549 | Sonnet 4.6 (by 0.006) |
| Improvement | −0.35% | −2.63% | Sonnet 4.6 (7.5×) |
| Keeps | 4 (3.9%) | 16 (16.0%) | Sonnet 4.6 (4×) |
| Crashes | 2 (1.9%) | 0 (0.0%) | Sonnet 4.6 |
Sonnet 4.6's strongest showing. Discovered ASPECT_RATIO=21 (vs AR=32), breaking the AR=32 consensus from all Sonnet 4.0 runs. Zero crashes.
| Metric | Sonnet 4.0 (Mar 20) | Sonnet 4.6 (Mar 25) | Winner |
|---|---|---|---|
| Best val_bpb | 1.5259 | 1.5267 | Sonnet 4.0 (by 0.0008) |
| Improvement | −1.0% | −0.30% | Sonnet 4.0 |
| Keeps | 3 (3.0%) | 2 (2.0%) | Comparable |
| Crashes | 0 (0.0%) | 2 (2.0%) | Sonnet 4.0 |
The hardest dataset to optimize. Both models find SlimPajama nearly impervious — total improvements of just 1.0% and 0.30%. Sonnet 4.6 went 62 experiments without a single keep before finding a triple-synergy combination (β1, FINAL_LR_FRAC, WARMDOWN_RATIO together). Sonnet 4.0 edges out on absolute val_bpb.
| Metric | Sonnet 4.0 (Mar 21) | Sonnet 4.6 (Mar 25) | Winner |
|---|---|---|---|
| Best val_bpb | 1.3463 | 1.3345 | Sonnet 4.6 (by 0.012) |
| Improvement | −1.97% | −0.99% | Sonnet 4.0 (more room) |
| Keeps | 20 (20.0%) | 13 (12.9%) | Sonnet 4.0 |
| Crashes | 5 (5.0%) | 2 (2.0%) | Sonnet 4.6 |
Sonnet 4.6 wins decisively on absolute val_bpb (1.3345 vs 1.3463). The signature finding: a spectacular β2 walk — 5 consecutive keeps as β2 decreased from 0.98→0.975→0.970→0.968→0.966→0.964. Sonnet 4.0 had more keeps (20 vs 13) but started from a worse baseline with more room to improve.
FineWeb-Edu-High Configuration Comparison (click to expand)
| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 21 | ❌ |
| MATRIX_LR | 0.047 | 0.068 | ❌ |
| EMBEDDING_LR | 0.375 | 0.60 | ❌ |
| SCALAR_LR | 0.39 | 0.18 | ❌ |
| WEIGHT_DECAY | 0.10 | 0.08 | ≈ |
| WARMDOWN_RATIO | 0.48 | 0.75 | ❌ |
| FINAL_LR_FRAC | 0.085 | 0.05 | ❌ |
| ADAM β1 | 0.66 | 0.45 | ❌ |
| ADAM β2 | 0.95 | 0.964 | ❌ |
| WINDOW_PATTERN | SSSL | SSSS | ❌ |
| MLP_RATIO | 4.25 | 4.0 | ❌ |
Every single parameter differs. Yet Sonnet 4.6 achieves 0.9% better val_bpb — the largest absolute gap in any comparison.
Starting with Haiku 4.5, all model comparison runs use stock upstream defaults as their baseline. This ensures every model starts from an identical, unoptimized configuration:
# Stock upstream defaults (used for all new model comparisons)
ASPECT_RATIO = 64 # Original upstream default
EMBEDDING_LR = 0.6
MATRIX_LR = 0.04
SCALAR_LR = 0.5
WEIGHT_DECAY = 0.2
ADAM_BETAS = (0.8, 0.95)
WARMDOWN_RATIO = 0.5
FINAL_LR_FRAC = 0.0
WINDOW_PATTERN = "SSSL"The --reset-defaults flag in run_suite.py resets train_mlx.py to these values before each dataset, preventing cross-model and cross-dataset contamination.
Why this matters: The initial Haiku run (aborted) inherited Sonnet 4.6's accumulated config (AR=21, β2=0.964), giving it a baseline (1.2966) that already beat Sonnet 4.0's best (1.2959). That invalidated the comparison. Stock defaults ensure each model is measured by its own optimization ability.
| Metric | Sonnet 4.0 | Sonnet 4.6 | Haiku 4.5 | Winner |
|---|---|---|---|---|
| Stock baseline | 1.4159 | 1.3859 | 1.4085 | — |
| Best val_bpb | 1.3588 | 1.3093 | 1.2953 | Haiku |
| Δ from baseline | −4.03% | −5.53% | −8.04% | Haiku (2×) |
| Keeps | 15 (15.0%) | 15 (15.0%) | 10 (10.0%) | S4.0/S4.6 tied |
| Crashes | 0 (0.0%) | 4 (4.0%) | 3 (3.0%) | S4.0 |
| First keep | exp2 | exp2 | exp6 | S4.0/S4.6 tied |
| AR discovered | 64 (never changed!) | 19 (6 keeps) | 32 (2 keeps) | S4.6 |
| Depth explored | Once (catastrophic) | Twice (0 keeps) | 3 keeps | Haiku |
| Window pattern | SSSL (default) | SLLL (novel) | LLLL (novel) | S4.6/Haiku |
| LR strategy | Higher (MATRIX=0.067) | Lower (MATRIX=0.035) | Mixed (MATRIX=0.047) | S4.6 adaptive |
| β exploration | β1 walk (0.8→0.635) | Tried, no keeps | β1 walk (0.8→0.77) | S4.0 |
| Multi-param synergy | No | MATRIX_LR+WINDOW | β1+MATRIX_LR (late) | S4.6/Haiku |
| Final throughput | 27.3K tok/sec (251 steps) | 51.6K tok/sec (474 steps) | 76.0K tok/sec (697 steps) | Haiku (2.8×) |
| Final memory | 26.1 GB | 15.5 GB | 11.6 GB | Haiku (56% less) |
Full Sonnet 4.0 analysis → | Full Sonnet 4.6 analysis → | Full Haiku 4.5 analysis →
Sonnet 4.0 — The Conservative Optimizer. Never changed architecture (AR=64 for all 100 experiments). Highest keep rate, zero crashes, precise β1 walk (0.8→0.635). But constrained to 251 gradient steps — perfect optimizer tuning on an oversized model.
Sonnet 4.6 — The Systematic Explorer. Most thorough AR walk (64→48→36→28→20→18→19), unique self-correcting behavior (MATRIX_LR: 0.04→0.06→0.04→0.035). Found novel SLLL window pattern. But missed depth reduction — left performance on the table.
Haiku 4.5 — The Architecture Innovator. Discovered AR=32, depth reduction, and LLLL attention — tripling throughput to 697 steps. Binary-search weakness (19 HEAD_DIM experiments, 0 keeps). Late-game synergy at exp90 combined β1 with MATRIX_LR for the winning result.
| Model | Final Steps | Final val_bpb | Architecture Changes |
|---|---|---|---|
| S4.0 | 251 | 1.3588 | None |
| S4.6 | 474 | 1.3093 | AR=19 |
| Haiku | 697 | 1.2953 | AR=32, depth−3, LLLL |
More gradient steps = better results, even at lower MFU. The throughput hierarchy (Haiku > S4.6 > S4.0) exactly matches the val_bpb hierarchy. Architecture discovery is the single biggest differentiator in 5-minute training.
ClimbMix complete for all 3 models. FineWeb-Edu, Cosmopedia-v2, SlimPajama, FineWeb-Edu-High will follow with the same stock-baseline methodology.
| Metric | S4.0 | S4.6 | Haiku | Opus (70 exp) |
|---|---|---|---|---|
| Stock baseline | 1.4159 | 1.3859 | 1.4085 | CRASHED (exp0) |
| Best val_bpb | 1.3588 | 1.3093 | 1.2953 | 1.3569* |
| Δ from baseline | −4.03% | −5.53% | −8.04% | −3.75%* |
| Keeps | 15 (15%) | 15 (15%) | 10 (10%) | 12 (17.1%)* |
| Crashes | 0 | 4 | 3 | 9* |
| Crash rate | 0% | 4% | 3% | 12.9%* |
| AR explored | Never | AR=19 | AR=32 | AR=80 (wrong way!) |
| Architecture changes | None | Yes | Yes | None |
| Final throughput | 27.3K (251 steps) | 51.6K (474 steps) | 76.0K (697 steps) | 28.3K (260 steps) |
* In progress — 70/100 experiments, paused due to API 529. Full analysis →
Key observations (70 experiments):
- Highest keep rate of any model (17.1%) — Opus is a steady, methodical optimizer once past the crash phase
- Architecturally blind like S4.0 — never reduced AR, depth, or changed window pattern. AR=80 and depth=10 attempts were catastrophic
- Best val_bpb (1.3569) already within striking distance of S4.0's final (1.3588) — will likely surpass S4.0 but remain well behind architecture-exploring models
- Takes opposite LR strategy to S4.0: reduced EMBEDDING_LR (0.6→0.4) and SCALAR_LR (0.5→0.45) where S4.0 increased them
- Self-correcting: initially tried increasing EMBEDDING_LR (failed), later discovered reducing it was the key improvement (exp63)
Opus ClimbMix will resume when API load subsides. 4 more datasets pending for all models.
| Dataset | S4.0 Best | S4.6 Best | Gap | Winner |
|---|---|---|---|---|
| ClimbMix | 1.2959 | 1.2997 | +0.0038 | S4.0 |
| FineWeb-Edu | 1.3424 | 1.3416 | −0.0008 | S4.6 |
| Cosmopedia-v2 | 0.9606 | 0.9549 | −0.0057 | S4.6 |
| SlimPajama | 1.5259 | 1.5267 | +0.0008 | S4.0 |
| FineWeb-Edu-High | 1.3463 | 1.3345 | −0.0118 | S4.6 |
Sonnet 4.0's wins are narrow (0.0038 and 0.0008) and benefited from pre-optimized baselines. Sonnet 4.6's wins are larger (up to 0.0118 on FineWeb-Edu-High) and came despite starting from worse baselines.
| Dataset | Sonnet 4.0 | Sonnet 4.6 | Reduction |
|---|---|---|---|
| ClimbMix | 11.0% | 4.2% | 2.6× fewer |
| FineWeb-Edu | 2.3% | 1.0% | 2.3× fewer |
| Cosmopedia-v2 | 1.9% | 0.0% | ∞ (zero crashes) |
| SlimPajama | 0.0% | 2.0% | S4.0 wins |
| FineWeb-Edu-High | 5.0% | 2.0% | 2.5× fewer |
| Total | 20 / 494 (4.0%) | 10 / 521 (1.9%) | 2.1× fewer |
Sonnet 4.6 crashes half as often overall. The sole exception is SlimPajama, where Sonnet 4.0 had zero crashes.
| Dataset | S4.0 params changed | S4.6 params changed |
|---|---|---|
| ClimbMix | 1 | 3 |
| FineWeb-Edu | 6 | 9 |
| Cosmopedia-v2 | 4 | 10 |
| SlimPajama | 3 | 4 |
| FineWeb-Edu-High | 8 | 10 |
| Average | 4.4 | 7.2 |
Sonnet 4.6 consistently explores 60% more parameter dimensions and builds compositional improvements across all datasets.
| Dataset | S4.0 AR | S4.6 AR | Agreement? |
|---|---|---|---|
| ClimbMix | 32 | 32 | ✅ |
| FineWeb-Edu | 32 | 32 | ✅ |
| Cosmopedia-v2 | 32 | 21 | ❌ |
| SlimPajama | 32 | 21 (inherited) | ❌ (different defaults) |
| FineWeb-Edu-High | 32 | 21 (inherited) | ❌ (different defaults) |
Sonnet 4.6's Cosmopedia-v2 AR=21 discovery propagated to subsequent runs via defaults. Neither SlimPajama nor FineWeb-Edu-High tried to change it — both explored narrower (AR=14, 16) and wider (AR=24, 26, 28) alternatives and found AR=21 near-optimal.
Sonnet 4.6 wins 3–2 on absolute val_bpb, but the margins are small on 4 of 5 datasets. The optimization landscape has natural performance floors that both models can approach. The exception is FineWeb-Edu-High, where Sonnet 4.6's β2 walk found a clearly better optimum.
By every meta-metric, Sonnet 4.6 outperforms:
- Crashes: 1.9% vs 4.0% (2.1× fewer)
- Parameter dimensions explored: 7.2 vs 4.4 per run (60% more)
- Compositional optimization: Consistently builds synergistic multi-parameter improvements
- β2 discovery: Systematic β2 walks (never attempted by Sonnet 4.0) produced keeps on 3 of 5 datasets
Sonnet 4.0 found AR=32 optimal on all 5 datasets. Sonnet 4.6 found AR=21 optimal on Cosmopedia-v2 and the result propagated — neither SlimPajama nor FineWeb-Edu-High reverted it. The "hardware-optimal architecture" depends on the LLM's exploration strategy, not just the hardware.
On FineWeb-Edu, both models find the same val_bpb via completely different configs (8/10 parameters differ). On FineWeb-Edu-High, every single parameter differs yet Sonnet 4.6 still beats Sonnet 4.0. The optimization landscape has multiple basins at similar or different depths.
Both models find SlimPajama nearly impervious to optimization (0.30–1.0% improvement, 2–3 keeps). Its 7-source diversity creates a flat optimization landscape where no parameter adjustment can meaningfully improve the model's ability to compress such varied data in 5 minutes.
-
Would AR=21 improve Sonnet 4.0's results? Sonnet 4.0 never tested AR<32 on any dataset. Running Sonnet 4.0 with AR=21 on Cosmopedia-v2 would reveal whether the AR=21 advantage is model-specific or universal.
-
Can we combine the best insights from both models? The models found complementary strategies (e.g., low WD vs low momentum on FineWeb-Edu). A hybrid config might beat both.
-
Is Sonnet 4.6's β2 walk transferable to Sonnet 4.0 datasets? The β2 walk was Sonnet 4.6's signature discovery. Testing β2<0.96 on Sonnet 4.0's best configs could unlock further improvements.
-
What would a different model tier find? ➜ Answered for ClimbMix. Three models, three distinct strategies, three different results. Haiku wins via architecture (AR=32, depth reduction, LLLL). S4.6 places 2nd via systematic AR exploration (AR=19, SLLL). S4.0 finishes 3rd — architecturally blind, pure optimizer tuning. Bigger LLM ≠ better optimizer. The smallest model (Haiku) found the deepest optimum. Full 3-model comparison →
See individual run pages: S4.0 Stock ClimbMix (Mar 26) | S4.6 Stock ClimbMix (Mar 26) | H4.5 ClimbMix (Mar 25) | S4.6 FineWeb-Edu-High (Mar 25) | S4.6 SlimPajama (Mar 25) | S4.6 Cosmopedia-v2 (Mar 24) | S4.6 FineWeb-Edu (Mar 22) | S4.6 ClimbMix (Mar 22) | S4.0 FineWeb-Edu-High (Mar 21) | S4.0 SlimPajama (Mar 20) | S4.0 Cosmopedia-v2 (Mar 20) | S4.0 ClimbMix (Mar 19) | S4.0 FineWeb-Edu (Mar 17) | Cross-Dataset Comparison