-
Notifications
You must be signed in to change notification settings - Fork 0
Cross LLM Comparison
How much does the LLM generation matter for autonomous hyperparameter optimization? This page tracks cross-generation comparison experiments: the same dataset, same hardware, same codebase, same training budget — but different Claude model versions driving the optimization.
| Metric | Sonnet 4.0 (Mar 19) | Sonnet 4.6 (Mar 22) | Winner |
|---|---|---|---|
| Baseline val_bpb | 1.2969 (pre-optimized) | 1.3213 (defaults) | Different starting points |
| Best val_bpb | 1.2959 | 1.2997 | Sonnet 4.0 (absolute) |
| Improvement | −0.08% | −1.63% | Sonnet 4.6 (20× more) |
| Keeps | 1 / 100 (1.0%) | 8 / 118 (6.8%) | Sonnet 4.6 (7× more) |
| Crashes | 11 / 100 (11.0%) | 5 / 118 (4.2%) | Sonnet 4.6 (2.6× fewer) |
| Exps to first keep | 84 | 25 | Sonnet 4.6 (3.4× faster) |
| Parameters optimized | 1 | 3 (synergistic) | Sonnet 4.6 |
| Total experiments | 100 | 118 | Comparable |
Sonnet 4.0's single keep (exp84):
# Only change from baseline:
UNEMBEDDING_LR = 0.0041 # was 0.004 — a 2.5% tweak
# Result: 1.2959 (−0.0010 from 1.2969 baseline)Sonnet 4.6's optimal configuration (exp112):
# Three synergistic changes from baseline:
WARMDOWN_RATIO = 0.63 # was 0.5 — 26% longer annealing
SCALAR_LR = 0.3 # was 0.51 — 41% lower
ADAM_BETAS = (0.71, 0.95) # was (0.8, 0.95) — 11% lower momentum
# Result: 1.2997 (−0.0216 from 1.3213 baseline)| Dimension | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Discovery speed | Slow — 83 failed experiments before first keep | Fast — first keep at exp25, compound discovery at exp28 |
| Composition | Never composed improvements — single-parameter tweaks only | Built each keep on prior keeps (warmdown → scalar LR → momentum) |
| Crash learning | Repeated batch size crashes (9 DEVICE_BATCH_SIZE attempts, all crash) | Stopped after 2 DEVICE_BATCH_SIZE crashes |
| Exploration breadth | Broad but unfocused — tried everything once | Phase 1 broad exploration, then focused on promising directions |
| Late-game strategy | Exhaustive fine-tuning around single param | Progressive refinement of a 3-parameter interaction |
| Wasted experiments | ~22 batch size experiments (11 crashes + 11 discards) | ~7 batch size experiments (5 crashes + 2 discards) |
The most striking difference is compositional optimization. Sonnet 4.6 recognized that:
- Longer warmdown (0.63) benefits from lower momentum (β1=0.71) — the optimizer can adapt quickly during the extended decay phase
- Lower scalar LR (0.3) prevents overshooting during the longer high-LR phase
- These three changes interact: each is more effective in the presence of the others
Sonnet 4.0 never attempted to combine improvements — its approach was strictly one-parameter-at-a-time with no stacking.
Sonnet 4.0 crashed 11 times on batch size changes, including 9 DEVICE_BATCH_SIZE attempts at values 3, 5, 6, 7, 9, 10, 11, 12, and 15. Despite every single one crashing, it kept trying.
Sonnet 4.6 crashed 5 times on batch size changes and then stopped. It recognized the pattern and redirected effort to productive hyperparameters.
Both models experienced long unproductive stretches:
- Sonnet 4.0: 83 experiments before any keep (exp1–exp83)
- Sonnet 4.6: 61 experiments between exp28 and exp90
The difference: Sonnet 4.6 had already banked 2 keeps before its plateau, while Sonnet 4.0 had nothing. And Sonnet 4.6 emerged from its plateau with a productive burst (4 keeps in 22 experiments), while Sonnet 4.0 found only 1 keep in its remaining 16 experiments.
Sonnet 4.0's absolute best (1.2959) is still lower than Sonnet 4.6's (1.2997). But this comparison is unfair — Sonnet 4.0 inherited a pre-optimized baseline. If we normalize:
| Starting Point → Best | Improvement |
|---|---|
| Sonnet 4.0: 1.2969 → 1.2959 | −0.0010 (0.08%) |
| Sonnet 4.6: 1.3213 → 1.2997 | −0.0216 (1.63%) |
Sonnet 4.6 covered 20× more ground. Given that Sonnet 4.6 started from harder defaults and still nearly caught up to Sonnet 4.0's absolute result (within 0.004 bpb), there's strong evidence it would surpass Sonnet 4.0's result if given the same starting point.
The following datasets will be tested with both Sonnet 4.0 (existing results) and Sonnet 4.6 (upcoming):
| Dataset | Sonnet 4.0 Status | Sonnet 4.6 Status |
|---|---|---|
| climbmix | ✅ 101 experiments, 1 keep | ✅ 119 experiments, 8 keeps |
| fineweb-edu | ✅ 88 experiments, 17 keeps | 🔲 Planned |
| cosmopedia-v2 | ✅ 103 experiments, 4 keeps | 🔲 Planned |
| slimpajama | ✅ 101 experiments, 3 keeps | 🔲 Planned |
| fineweb-edu-high | ✅ 101 experiments, 20 keeps | 🔲 Planned |
-
Does Sonnet 4.6's advantage hold for educational text? FineWeb-Edu had the highest keep rate (19.3%) with Sonnet 4.0. Will Sonnet 4.6 find even more improvements, or is the optimization landscape already well-explored?
-
Will Sonnet 4.6 discover the same two-cluster split? The Sonnet 4.0 runs revealed that educational text needs different hyperparameters than general text. Will Sonnet 4.6 independently discover the same pattern?
-
Same starting point comparison? The climbmix comparison was complicated by different baselines. Future runs should consider standardizing starting points for cleaner comparison.
-
Can Sonnet 4.6's climbmix config improve Sonnet 4.0's result? Sonnet 4.6 found WARMDOWN_RATIO=0.63, SCALAR_LR=0.3, ADAM β1=0.71 from defaults. Applying these to the pre-optimized baseline (which Sonnet 4.0 started from) might push below 1.2959.
See individual run pages: Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 ClimbMix (Mar 19) | Cross-Dataset Comparison