-
Notifications
You must be signed in to change notification settings - Fork 0
Sonnet 4.6 ClimbMix Agent Run Mar 22 2026
First cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing the climbmix-400b dataset on the same hardware and codebase previously used by Claude Sonnet 4.0. This run tests whether a newer LLM generation produces better optimization strategies.
Key finding: Sonnet 4.6 is a dramatically better optimizer. It found 8 keeps (vs 1 for Sonnet 4.0), achieved its first improvement 3× faster (exp25 vs exp84), and demonstrated compositional multi-parameter optimization that Sonnet 4.0 never achieved. The 1.63% improvement from baseline dwarfs Sonnet 4.0's 0.08%.
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 119 (exp0–exp118) |
| Kept | 8 (6.8%) |
| Discarded | 106 (89.1%) |
| Crashed | 5 (4.2%) |
| Best val_bpb | 1.2997 (exp112) |
| Baseline val_bpb | 1.3213 (defaults) |
| Total Improvement | −0.0216 (1.63%) |
| LLM Backend | Claude Sonnet 4.6 |
| Results | results/sonnet-4-6/climbmix/ |
This run started from reset defaults (MATRIX_LR=0.05, SCALAR_LR=0.51), not the pre-optimized baseline used by the Sonnet 4.0 climbmix run (MATRIX_LR=0.0435, SCALAR_LR=0.4, EMBEDDING_LR=0.4). This means:
- Sonnet 4.6 baseline: 1.3213 (from defaults)
- Sonnet 4.0 baseline: 1.2969 (pre-optimized)
- Sonnet 4.0's absolute best (1.2959) is still lower than Sonnet 4.6's (1.2997)
- But Sonnet 4.6 achieved 20× more improvement from its starting point (1.63% vs 0.08%)
Fair comparison requires looking at improvement magnitude and optimization behavior, not just absolute val_bpb.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.3213 | — | Baseline (defaults) | Starting point |
| exp25 | 1.3144 | −0.52% | WARMDOWN_RATIO 0.5 → 0.6 | Longer LR annealing |
| exp28 | 1.3091 | −0.92% | + SCALAR_LR 0.51 → 0.4 | Lower scalar LR |
| exp90 | 1.3070 | −1.08% | + SCALAR_LR 0.4 → 0.3 | Even lower scalar LR |
| exp102 | 1.3037 | −1.33% | + ADAM β1 0.8 → 0.77 | Lower momentum |
| exp104 | 1.3014 | −1.50% | + WARMDOWN_RATIO 0.6 → 0.63 | Fine-tune warmdown |
| exp111 | 1.3003 | −1.58% | + ADAM β1 0.77 → 0.74 | Further lower momentum |
| exp112 | 1.2997 | −1.63% | + ADAM β1 0.74 → 0.71 | Optimal momentum |
Phase 1 — Exploration (exp1–24, 24 experiments): Broad sweeps of depth, batch size, LR values, weight decay, window patterns. All discards. Sonnet 4.6 systematically tested and eliminated most hyperparameter dimensions.
Phase 2 — Breakthrough (exp25–28, 4 experiments): Discovered WARMDOWN_RATIO=0.6 (exp25, first keep), then immediately combined with SCALAR_LR=0.4 (exp28, second keep). Two keeps in four experiments — the model recognized synergy between these parameters.
Phase 3 — Plateau (exp29–89, 61 experiments): Long exploration drought. Exhaustive sweeps of weight decay (0.0–0.25), embedding LR, matrix LR, Adam betas, window patterns, architecture changes. Nothing worked. This is the longest plateau in any run.
Phase 4 — Second Wave (exp90–104, 15 experiments): Three keeps in quick succession. SCALAR_LR refined to 0.3 (exp90), Adam β1 tuned to 0.77 (exp102), warmdown fine-tuned to 0.63 (exp104). The model combined these into a coherent configuration.
Phase 5 — Final Push (exp105–112, 8 experiments): Adam β1 driven progressively lower: 0.77 → 0.74 → 0.71, yielding two more keeps. Each step built on the previous.
Phase 6 — Diminishing Returns (exp113–118, 6 experiments): β1=0.68 overshoots. Remaining tweaks (β2, scalar LR, warmdown) all fail. Run ended at natural convergence.
Every keep also improved hardware efficiency:
| Exp | MFU | tok/sec | Peak Mem (GB) | Steps |
|---|---|---|---|---|
| exp0 (baseline) | 14.8% | 48,236 | 15.5 | 442 |
| exp25 | 15.5% | 50,283 | 15.5 | 461 |
| exp28 | 16.1% | 52,315 | 15.5 | 480 |
| exp90 | 16.3% | 52,911 | 15.5 | 486 |
| exp102 | 16.7% | 54,242 | 15.5 | 496 |
| exp104 | 17.0% | 54,890 | 15.5 | 505 |
| exp111 | 17.1% | 55,259 | 15.5 | 508 |
| exp112 | 17.4% | 56,666 | 15.5 | 518 |
MFU rose 17.6%, throughput rose 17.5%, steps rose 17.2%. Memory stayed constant at 15.5 GB. The optimizer changes enabled more efficient training within the same time budget.
5 crashes (4.2%) — all batch size related:
| Exp | Parameter | Issue |
|---|---|---|
| exp19 | TOTAL_BATCH_SIZE halved | Crash |
| exp49 | TOTAL_BATCH_SIZE decreased | Crash (retry of exp19 concept) |
| exp54 | TOTAL_BATCH_SIZE increased | Crash |
| exp84 | DEVICE_BATCH_SIZE = 6 | Crash |
| exp87 | DEVICE_BATCH_SIZE = 7 | Crash |
Sonnet 4.6 learned from batch size crashes quickly — only 5 total, compared to Sonnet 4.0's 11 crashes on the same failure mode (9 DEVICE_BATCH_SIZE attempts).
# Architecture (unchanged from defaults)
ASPECT_RATIO = 32
DEPTH = 8
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
MLP_RATIO = 4.0
# Optimization (3 parameters changed)
WARMDOWN_RATIO = 0.63 # Default: 0.5 → longer LR annealing
SCALAR_LR = 0.3 # Default: 0.51 → lower scalar learning rate
ADAM_BETAS = (0.71, 0.95) # Default: (0.8, 0.95) → less momentum
# Unchanged
MATRIX_LR = 0.05
EMBEDDING_LR = 0.4
UNEMBEDDING_LR = 0.004
WEIGHT_DECAY = 0.2
FINAL_LR_FRAC = 0.0
# Performance
# val_bpb: 1.2997 | mem: 15.5 GB | steps: 518 | MFU: 17.4%Sonnet 4.6's optimization strategy differed fundamentally from Sonnet 4.0's:
| Capability | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Time to first keep | exp84 (84th attempt) | exp25 (25th attempt) |
| Total keeps | 1 | 8 |
| Parameters changed | 1 (UNEMBED_LR ↑2.5%) | 3 (synergistic) |
| Composed improvements | No | Yes — each keep built on prior |
| Crash learning | Slow (11 crashes, repeated) | Fast (5 crashes, stopped) |
| Strategy type | Exhaustive single-param sweeps | Multi-param composition |
Sonnet 4.6 discovered that warmdown ratio, scalar LR, and Adam momentum interact synergistically — a longer warmdown benefits from lower momentum that can adapt quickly during the decay phase, and lower scalar LR prevents overshooting during the extended high-LR phase. This is a genuine optimization insight that Sonnet 4.0 never found.
All 119 experiments (click to expand)
| Exp | Description | val_bpb | Steps | MFU | Status |
|---|---|---|---|---|---|
| exp0 | baseline (no modifications) | 1.3213 | 442 | 14.8 | baseline |
| exp1 | Increase DEPTH | 1.4297 | 213 | 15.8 | discard |
| exp2 | Reduce DEVICE_BATCH_SIZE | 1.3266 | 432 | 14.5 | discard |
| exp3 | Increase WARMDOWN_RATIO 0.5→0.7 | 1.3333 | 416 | 13.9 | discard |
| exp4 | Increase MATRIX_LR 0.05→0.08 | 1.3453 | 381 | 12.7 | discard |
| exp5 | Increase SCALAR_LR 0.51→0.65 | 1.3472 | 391 | 13.1 | discard |
| exp6 | Decrease MATRIX_LR 0.05→0.03 | 1.3859 | 353 | 11.8 | discard |
| exp7 | Decrease WARMDOWN_RATIO 0.5→0.3 | 1.3480 | 372 | 12.4 | discard |
| exp8 | Increase ADAM β1 0.8→0.9 | 1.3386 | 403 | 13.5 | discard |
| exp9 | Decrease TOTAL_BATCH_SIZE | 1.3318 | 411 | 13.8 | discard |
| exp10 | Decrease WEIGHT_DECAY 0.2→0.05 | 1.3388 | 393 | 13.2 | discard |
| exp11 | Decrease EMBEDDING_LR 0.4→0.1 | 1.3436 | 395 | 13.2 | discard |
| exp12 | Decrease ADAM β1 0.8→0.7 | 1.3385 | 387 | 12.9 | discard |
| exp13 | Increase EMBEDDING_LR 0.4→1.0 | 1.3611 | 360 | 12.0 | discard |
| exp14 | Change WINDOW_PATTERN | 1.3483 | 361 | 11.6 | discard |
| exp15 | Decrease ASPECT_RATIO 32→16 | 1.3439 | 737 | 9.6 | discard |
| exp16 | WARMDOWN_RATIO→0.2 + FINAL_LR_FRAC→0.05 | 1.3406 | 403 | 13.5 | discard |
| exp17 | Increase ASPECT_RATIO 32→64 | 1.4068 | 217 | 20.7 | discard |
| exp18 | ADAM β2 0.95→0.99 | 1.3223 | 447 | 15.0 | discard |
| exp19 | Decrease TOTAL_BATCH_SIZE (halved) | — | — | — | crash |
| exp20 | UNEMBEDDING_LR 0.004→0.04 | 1.3272 | 478 | 16.1 | discard |
| exp21 | UNEMBEDDING_LR→0.04 + EMBEDDING_LR↑ | 1.3425 | 388 | 13.0 | discard |
| exp22 | Decrease DEVICE_BATCH_SIZE to 4 | 1.3374 | 400 | 13.4 | discard |
| exp23 | UNEMBEDDING_LR→0.04 (isolated test) | 1.3479 | 416 | 13.9 | discard |
| exp24 | WARMDOWN_RATIO→0.4 + FINAL_LR_FRAC→0.01 | 1.3293 | 419 | 14.0 | discard |
| exp25 | WARMDOWN_RATIO 0.5→0.6 | 1.3144 | 461 | 15.5 | keep |
| exp26 | WARMDOWN_RATIO 0.6→0.75 | 1.3290 | 429 | 14.4 | discard |
| exp27 | MATRIX_LR 0.05→0.04 | 1.3200 | 466 | 15.6 | discard |
| exp28 | SCALAR_LR 0.51→0.4 | 1.3091 | 480 | 16.1 | keep |
| exp29 | EMBEDDING_LR 0.4→0.2 | 1.3127 | 469 | 15.7 | discard |
| exp30 | WEIGHT_DECAY 0.2→0.05 | 1.3238 | 436 | 14.6 | discard |
| exp31 | ADAM β2 0.95→0.98 | 1.3296 | 419 | 14.0 | discard |
| exp32 | MATRIX_LR 0.05→0.035 | 1.3253 | 429 | 14.4 | discard |
| exp33 | ADAM β1 0.8→0.85 | 1.3270 | 426 | 14.3 | discard |
| exp34 | MATRIX_LR 0.05→0.06 | 1.3150 | 462 | 15.5 | discard |
| exp35 | WEIGHT_DECAY 0.2→0.1 | 1.3185 | 452 | 15.2 | discard |
| exp36 | ADAM β1 0.8→0.75 | 1.3171 | 459 | 15.4 | discard |
| exp37 | WARMDOWN_RATIO 0.6→0.65 | 1.3196 | 441 | 14.8 | discard |
| exp38 | ADAM β2 0.95→0.90 | 1.3277 | 424 | 14.2 | discard |
| exp39 | Increase DEPTH | 1.3194 | 449 | 15.1 | discard |
| exp40 | EMBEDDING_LR 0.4→0.3 | 1.3298 | 413 | 13.8 | discard |
| exp41 | WARMUP_RATIO 0.0→0.05 | 1.3309 | 425 | 14.2 | discard |
| exp42 | UNEMBEDDING_LR 0.004→0.02 | 1.3340 | 411 | 13.8 | discard |
| exp43 | EMBEDDING_LR 0.4→0.6 | 1.3253 | 432 | 14.5 | discard |
| exp44 | WEIGHT_DECAY 0.2→0.0 | 1.3450 | 400 | 13.4 | discard |
| exp45 | UNEMBEDDING_LR 0.004→0.0004 | 1.5865 | 400 | 13.4 | discard |
| exp46 | FINAL_LR_FRAC 0.0→0.01 | 1.3390 | 391 | 13.1 | discard |
| exp47 | Decrease DEVICE_BATCH_SIZE | 1.3353 | 401 | 13.4 | discard |
| exp48 | WARMDOWN_RATIO 0.6→0.55 | 1.3473 | 370 | 12.3 | discard |
| exp49 | Decrease TOTAL_BATCH_SIZE | — | — | — | crash |
| exp50 | Change WINDOW_PATTERN (more local) | 1.3167 | 461 | 14.9 | discard |
| exp51 | Change WINDOW_PATTERN | 1.3242 | 431 | 13.9 | discard |
| exp52 | HEAD_DIM 128→64 | 1.4110 | 312 | 10.4 | discard |
| exp53 | MATRIX_LR 0.05→0.055 | 1.3448 | 384 | 12.8 | discard |
| exp54 | Increase TOTAL_BATCH_SIZE | — | — | — | crash |
| exp55 | ADAM β2 0.95→0.93 | 1.3344 | 395 | 13.2 | discard |
| exp56 | WEIGHT_DECAY 0.2→0.15 | 1.3361 | 396 | 13.2 | discard |
| exp57 | ADAM β1 0.8→0.7 | 1.3319 | 409 | 13.7 | discard |
| exp58 | WEIGHT_DECAY 0.2→0.08 | 1.3238 | 436 | 14.6 | discard |
| exp59 | MATRIX_LR 0.05→0.045 | 1.3110 | 461 | 15.5 | discard |
| exp60 | MATRIX_LR 0.05→0.042 | 1.3226 | 434 | 14.6 | discard |
| exp61 | SCALAR_LR 0.4→0.35 | 1.3253 | 430 | 14.4 | discard |
| exp62 | SCALAR_LR 0.4→0.45 | 1.3300 | 425 | 14.2 | discard |
| exp63 | WARMDOWN_RATIO 0.6→0.62 | 1.3544 | 353 | 11.8 | discard |
| exp64 | EMBEDDING_LR 0.4→0.8 | 1.3700 | 327 | 10.9 | discard |
| exp65 | WARMDOWN_RATIO 0.6→0.68 | 1.3703 | 330 | 11.0 | discard |
| exp66 | UNEMBEDDING_LR 0.004→0.04 | 1.4046 | 348 | 11.6 | discard |
| exp67 | WARMDOWN_RATIO 0.6→0.58 | 1.3550 | 357 | 11.9 | discard |
| exp68 | Batch size adjustment | 1.3524 | 363 | 12.1 | discard |
| exp69 | WARMDOWN_RATIO 0.6→0.55 | 1.3520 | 360 | 12.0 | discard |
| exp70 | WEIGHT_DECAY 0.2→0.25 | 1.3095 | 484 | 16.2 | discard |
| exp71 | WEIGHT_DECAY 0.2→0.3 | 1.3249 | 437 | 14.6 | discard |
| exp72 | FINAL_LR_FRAC 0.0→0.005 | 1.3228 | 431 | 14.4 | discard |
| exp73 | WEIGHT_DECAY 0.2→0.22 | 1.3287 | 419 | 14.0 | discard |
| exp74 | WARMUP_RATIO 0.0→0.02 | 1.3377 | 410 | 13.7 | discard |
| exp75 | WEIGHT_DECAY 0.2→0.23 | 1.3185 | 458 | 15.4 | discard |
| exp76 | ADAM β2 0.95→0.97 | 1.3285 | 420 | 14.1 | discard |
| exp77 | ADAM β2 0.95→0.96 | 1.3320 | 405 | 13.6 | discard |
| exp78 | MATRIX_LR 0.05→0.048 | 1.3268 | 418 | 14.0 | discard |
| exp79 | ADAM β1 0.8→0.82 | 1.3212 | 434 | 14.6 | discard |
| exp80 | WEIGHT_DECAY 0.2→0.24 | 1.3299 | 421 | 14.1 | discard |
| exp81 | ADAM β1 0.8→0.78 | 1.3239 | 439 | 14.7 | discard |
| exp82 | DEVICE_BATCH_SIZE halved | 1.3108 | 481 | 16.1 | discard |
| exp83 | WINDOW_PATTERN "SSSL"→"SSLL" | 1.3246 | 425 | 15.3 | discard |
| exp84 | DEVICE_BATCH_SIZE = 6 | — | — | — | crash |
| exp85 | WINDOW_PATTERN "SSSL"→"LSSL" | 1.3239 | 431 | 15.6 | discard |
| exp86 | WEIGHT_DECAY 0.2→0.21 | 1.3209 | 440 | 14.8 | discard |
| exp87 | DEVICE_BATCH_SIZE = 7 | — | — | — | crash |
| exp88 | Increase DEPTH (+2) | 1.3780 | 255 | 19.2 | discard |
| exp89 | EMBEDDING_LR 0.4→0.5 | 1.3162 | 459 | 15.4 | discard |
| exp90 | SCALAR_LR 0.4→0.3 | 1.3070 | 486 | 16.3 | keep |
| exp91 | SCALAR_LR 0.3→0.2 | 1.3132 | 481 | 16.2 | discard |
| exp92 | EMBEDDING_LR 0.4→0.3 | 1.3253 | 421 | 14.1 | discard |
| exp93 | SCALAR_LR 0.3→0.25 | 1.3133 | 463 | 15.6 | discard |
| exp94 | MATRIX_LR 0.05→0.052 | 1.3142 | 460 | 15.4 | discard |
| exp95 | WEIGHT_DECAY 0.2→0.25 | 1.3239 | 431 | 14.4 | discard |
| exp96 | WARMDOWN_RATIO 0.6→0.65 | 1.3244 | 427 | 14.3 | discard |
| exp97 | EMBEDDING_LR 0.4→0.5 | 1.3204 | 438 | 14.7 | discard |
| exp98 | ADAM β1 0.8→0.75 | 1.3084 | 471 | 15.8 | discard |
| exp99 | ADAM β2 0.95→0.97 + β1→0.75 | 1.3348 | 388 | 13.0 | discard |
| exp100 | ADAM β2 0.95→0.93 | 1.3182 | 444 | 14.9 | discard |
| exp101 | SCALAR_LR 0.3→0.35 | 1.3075 | 491 | 16.5 | discard |
| exp102 | ADAM β1 0.8→0.77 | 1.3037 | 496 | 16.7 | keep |
| exp103 | WARMDOWN_RATIO 0.6→0.55 | 1.3038 | 498 | 16.7 | discard |
| exp104 | WARMDOWN_RATIO 0.6→0.63 | 1.3014 | 505 | 17.0 | keep |
| exp105 | WEIGHT_DECAY 0.2→0.25 | 1.3101 | 475 | 16.0 | discard |
| exp106 | WEIGHT_DECAY 0.2→0.15 | 1.3099 | 472 | 15.9 | discard |
| exp107 | MATRIX_LR 0.05→0.04 | 1.3068 | 485 | 16.3 | discard |
| exp108 | MATRIX_LR 0.05→0.055 | 1.3056 | 503 | 16.9 | discard |
| exp109 | EMBEDDING_LR 0.4→0.5 | 1.3037 | 506 | 17.0 | discard |
| exp110 | ADAM β2 0.95→0.93 | 1.3023 | 506 | 17.0 | discard |
| exp111 | ADAM β1 0.77→0.74 | 1.3003 | 508 | 17.1 | keep |
| exp112 | ADAM β1 0.74→0.71 | 1.2997 | 518 | 17.4 | keep |
| exp113 | ADAM β1 0.71→0.68 | 1.3020 | 504 | 17.0 | discard |
| exp114 | ADAM β2 0.95→0.93 | 1.3020 | 507 | 17.1 | discard |
| exp115 | SCALAR_LR 0.3→0.28 | 1.3035 | 493 | 16.6 | discard |
| exp116 | EMBEDDING_LR 0.4→0.45 | 1.3093 | 479 | 16.1 | discard |
| exp117 | WARMDOWN_RATIO 0.63→0.65 | 1.3098 | 478 | 16.1 | discard |
| exp118 | SCALAR_LR 0.3→0.32 | 1.3045 | 506 | 17.0 | discard |
See also: Cross-LLM Comparison | Climbmix Sonnet 4.0 (Mar 19) | Cross-Dataset Comparison