-
Notifications
You must be signed in to change notification settings - Fork 0
Haiku 4.5 ClimbMix Agent Run Mar 25 2026
First Haiku 4.5 comparison run: Claude Haiku 4.5 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). This is the first run using the new fair-baseline methodology — every model starts from identical, unoptimized hyperparameters.
Key finding: Haiku wins ClimbMix. Starting from the worst baseline of any model (1.4085 vs Sonnet's pre-optimized ~1.30), Haiku achieved the best absolute val_bpb of 1.2953 — beating both Sonnet 4.0 (1.2959) and Sonnet 4.6 (1.2997). It discovered novel improvements both Sonnets missed (depth reduction, all-local attention, UNEMBEDDING_LR) and, in a dramatic late comeback, finally achieved multi-parameter synergy at exp90 — combining β1 tuning with MATRIX_LR adjustment.
Status: Complete (100/100 experiments)
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 100 (exp0–exp99) |
| Kept | 10 (10.0%) |
| Discarded | 87 (87.0%) |
| Crashed | 3 (3.0%) |
| Best val_bpb | 1.2953 (exp91) |
| Baseline val_bpb | 1.4085 (stock defaults, AR=64) |
| Total Improvement | −0.1132 (8.04%) |
| LLM Backend | Claude Haiku 4.5 |
| Results | results/haiku-4-5-20251001/climbmix/ |
This run uses the fair-baseline methodology. All models start from identical stock upstream defaults:
| Parameter | Stock Default |
|---|---|
| ASPECT_RATIO | 64 |
| EMBEDDING_LR | 0.6 |
| MATRIX_LR | 0.04 |
| SCALAR_LR | 0.5 |
| WEIGHT_DECAY | 0.2 |
| ADAM_BETAS | (0.8, 0.95) |
| WARMDOWN_RATIO | 0.5 |
| FINAL_LR_FRAC | 0.0 |
| WINDOW_PATTERN | SSSL |
The stock AR=64 baseline (1.4085) is significantly worse than the Sonnet baselines (S4.0: 1.2969, S4.6: 1.3213), which started from partially-optimized configs. Despite this handicap, Haiku achieved the best absolute result of any model on ClimbMix.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.4085 | — | Baseline (stock defaults) | Starting point |
| exp6 | 1.4081 | −0.03% | WARMDOWN_RATIO 0.5 → 0.2 | Less aggressive LR decay |
| exp19 | 1.3729 | −2.53% | ASPECT_RATIO 64 → 48 | Architecture walk begins |
| exp20 | 1.3473 | −4.35% | ASPECT_RATIO 48 → 32 | Rediscovers AR=32 |
| exp30 | 1.3260 | −5.86% | DEPTH reduced (−2 layers) | Novel — neither Sonnet tried |
| exp32 | 1.3166 | −6.52% | WINDOW_PATTERN SSSL → LLLL | Novel — all-local attention |
| exp44 | 1.3152 | −6.62% | DEPTH reduced further (−3 layers) | More gradient steps |
| exp45 | 1.3047 | −7.37% | MATRIX_LR 0.04 → 0.045 | LR tuning on optimized arch |
| exp55 | 1.3018 | −7.58% | UNEMBEDDING_LR 0.004 → 0.0035 | Novel — neither Sonnet touched |
| exp90 | 1.2989 | −7.78% | β1=0.77 + MATRIX_LR=0.046 | First multi-param synergy |
| exp91 | 1.2953 | −8.04% | MATRIX_LR 0.046 → 0.047 | Best result — wins ClimbMix |
Phase 1 — Early Exploration (exp1–18, 18 experiments, 1 keep): Broad sweeps of batch sizes (3 crashes), LR values, weight decay, warmup ratio. Only one keep (exp6, WARMDOWN_RATIO). Haiku learned quickly from batch size crashes — zero crashes after exp5.
Phase 2 — Architecture Discovery (exp19–20, 2 experiments, 2 keeps): The breakthrough phase. Haiku walked ASPECT_RATIO from 64→48→32 in two consecutive keeps, independently rediscovering Sonnet 4.0's AR=32 conclusion. This produced the largest single improvement (−4.35%).
Phase 3 — HEAD_DIM Obsession (exp21–29, 31, 33–39, 19 experiments, 0 keeps): Haiku's biggest weakness. Systematic binary search through HEAD_DIM values: 64, 96, 80, 76, 120, 116, 124, 72, 88, 84, 122, 126, 112, 100, 104. Every single experiment was a discard. HEAD_DIM=128 remained optimal throughout. This consumed 19% of total experiments on a dead parameter.
Phase 4 — Depth + Window Breakthrough (exp30, 32, 40–44, 8 experiments, 3 keeps): Discovered two novel improvements neither Sonnet found: depth reduction (exp30) and all-local attention WINDOW_PATTERN=LLLL (exp32). Further depth reduction kept at exp44. Throughput jumped from 404 to 593 gradient steps.
Phase 5 — LR Micro-Tuning (exp45–55, 11 experiments, 2 keeps): Found MATRIX_LR=0.045 (exp45) and UNEMBEDDING_LR=0.0035 (exp55). But also wasted 6 experiments on MATRIX_LR micro-variations (0.044–0.0465) — all discards.
Phase 6 — The Plateau (exp56–70, 15 experiments, 0 keeps): Exhaustive search of SCALAR_LR, EMBEDDING_LR, WARMDOWN_RATIO, WEIGHT_DECAY — nothing improved. Haiku appeared stuck at 1.3018.
Phase 7 — β1 Discovery & Late Comeback (exp72–91, 20 experiments, 2 keeps): Haiku finally explored Adam betas. After 18 β1 experiments mapping the landscape (0.765–0.80), it found β1=0.77 yielded 1.3028 (exp77) — tantalizingly close but not a keep. Then at exp90, Haiku combined β1=0.77 with MATRIX_LR=0.046 for its first-ever multi-parameter synergy: 1.2989 (keep). Pushing MATRIX_LR to 0.047 at exp91 produced the winning result: 1.2953.
Phase 8 — Final Diminishing Returns (exp92–99, 8 experiments, 0 keeps): Attempted to further refine β1 (0.765–0.769) and MATRIX_LR (0.0475, 0.048) — all discards. The optimum at β1=0.77, MATRIX_LR=0.047 appears stable.
Haiku found four improvements that neither Sonnet 4.0 nor Sonnet 4.6 discovered:
| Discovery | Haiku | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|---|
| Depth reduction | 2 keeps (exp30, 44) | Never tried | Never tried |
| WINDOW_PATTERN=LLLL | Keep (exp32) | Kept SSSL | Found SSSS |
| UNEMBEDDING_LR tuning | Keep (exp55) | Never tried | Never tried |
| β1 + LR synergy | Keep (exp90) — late discovery | Never tried | β2 focus instead |
Haiku's dominant strategy is binary-search: find a promising parameter, then exhaustively bisect the range. This produced the HEAD_DIM obsession (19 values, 0 keeps) and β1 micro-tuning (18 values, 0 direct keeps). But crucially, the β1 exploration wasn't wasted — it mapped the landscape well enough for Haiku to combine β1=0.77 with MATRIX_LR adjustment at exp90, producing its first multi-parameter synergy and the winning result.
This late emergence of synergy (exp90 of 100) is the most distinctive behavioral pattern in any model run. It suggests that even models with "shallow" optimization strategies can discover compositional improvements given sufficient budget.
| Behavior | Sonnet 4.0 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| Best val_bpb | 1.2959 | 1.2997 | 1.2953 |
| Baseline | 1.2969 (pre-opt) | 1.3213 (partial) | 1.4085 (stock) |
| Δ from baseline | −0.08% | −1.63% | −8.04% |
| Strategy archetype | Exhaustive sweeps | Compositional synergy | Binary-search → late synergy |
| AR discovery | AR=32 (pre-set) | AR=21 (novel) | AR=32 (rediscovered) |
| Window pattern | SSSL (default) | SSSS (all-sliding) | LLLL (all-local) |
| Depth exploration | Never tried | Never tried | Yes — 3 keeps |
| HEAD_DIM | Minimal | None | 19 experiments, 0 keeps |
| Multi-param synergy | No | Yes (throughout) | Yes (exp90 — late) |
| β exploration | None | β2 walks (signature) | β1 walks (18 exp) |
| First keep | exp84 | exp25 | exp6 |
| Total keeps | 1 (1.0%) | 8 (6.8%) | 10 (10.0%) |
| Crash rate | 11.0% | 4.2% | 3.0% |
- Best absolute result (1.2953) despite worst starting baseline — Haiku wins ClimbMix
- Highest keep rate (10.0%) and fastest first keep (exp6) of any model
- Novel discoveries: Depth reduction, all-local attention, UNEMBEDDING_LR, β1+LR synergy
- Quick crash learning: Zero crashes after exp5 (lowest crash rate overall at 3.0%)
- Late-game adaptation: Overcame the "shallow optimizer" pattern to discover synergy at exp90
- Binary-search trap: 19% of budget exhaustively searching HEAD_DIM, a dead parameter
- Late synergy: Multi-param discovery at exp90/100 — would miss it with a shorter budget
- β1 micro-tuning addiction: 18 experiments exploring β1 values (0.765–0.80) with only ±0.001 resolution
- Misses β2 entirely: Never attempted β2 changes (S4.6's signature discovery)
- No AR innovation: Converged on AR=32 (same as S4.0), never tested AR<32 (S4.6 found AR=21)
3 crashes (3.0%) — all in the first 5 experiments:
| Exp | Parameter | Issue |
|---|---|---|
| exp1 | TOTAL_BATCH_SIZE → 512 | Numerical instability |
| exp2 | TOTAL_BATCH_SIZE → 1024 | Numerical instability |
| exp5 | DEVICE_BATCH_SIZE reduced | OOM or instability |
Haiku learned from batch crashes quickly and never attempted risky batch changes again. Lowest crash rate of any model.
| Exp | MFU | tok/sec | Peak Mem (GB) | Steps |
|---|---|---|---|---|
| exp0 (baseline) | 20.3% | 22,977 | 26.1 | 212 |
| exp20 (AR=32) | 13.5% | 43,985 | 15.5 | 404 |
| exp32 (LLLL) | 18.3% | 61,666 | 13.3 | 564 |
| exp45 (MATRIX_LR) | 18.6% | 72,594 | 11.6 | 664 |
| exp55 (UNEMBED_LR) | 19.1% | 74,666 | 11.6 | 684 |
| exp91 (final best) | 19.5% | 75,974 | 11.6 | 697 |
Throughput tripled (22K→76K tok/sec), memory dropped 56% (26.1→11.6 GB), and steps increased 3.3× (212→697) through architecture optimization.
# Haiku 4.5 ClimbMix Best (exp91) — WINS ClimbMix
ASPECT_RATIO = 32 # was 64 (stock)
DEPTH = max(2, _hp_defaults['depth'] - 3) # reduced from default
HEAD_DIM = 128 # unchanged
WINDOW_PATTERN = "LLLL" # was "SSSL" — all local attention
MATRIX_LR = 0.047 # was 0.04
EMBEDDING_LR = 0.6 # unchanged
UNEMBEDDING_LR = 0.0035 # was 0.004
SCALAR_LR = 0.5 # unchanged
WEIGHT_DECAY = 0.2 # unchanged
ADAM_BETAS = (0.77, 0.95) # β1 was 0.8
WARMDOWN_RATIO = 0.2 # was 0.5
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.2953 | mem: 11.6 GB | tok/sec: 76.0K | MFU: 19.5% | 697 stepsHaiku's run tells a compelling story about optimization strategy evolution:
- Exp0–55 (The Grid Optimizer): Systematic but shallow. Every keep changed exactly one parameter. HEAD_DIM obsession wasted 19 experiments. Appeared stuck at 1.3018.
- Exp56–70 (The Plateau): 15 straight discards. Every single-parameter change failed. Haiku seemed trapped.
- Exp72–89 (The β1 Mapping): 18 experiments exploring β1 values. Every one was a discard. But Haiku was mapping the landscape — β1=0.77 consistently produced 1.3028, tantalizingly close to the 1.3018 best but not enough alone.
- Exp90 (The Breakthrough): Haiku combined β1=0.77 with MATRIX_LR=0.046 — its first-ever multi-parameter synergy. Result: 1.2989, a new best.
- Exp91 (The Win): Pushed MATRIX_LR to 0.047. Result: 1.2953 — beating both Sonnets.
The lesson: a "shallow" optimizer can evolve into a compositional one given sufficient budget and cumulative learning. Haiku's 18-experiment β1 exploration looked wasteful in isolation, but it provided the landscape knowledge needed for the exp90 breakthrough.
Run complete (100/100 experiments). Haiku 4.5 wins ClimbMix with val_bpb = 1.2953.
See also: Cross-LLM Comparison | Sonnet 4.6 ClimbMix (Mar 22) | Sonnet 4.0 ClimbMix (Mar 19)