-
Notifications
You must be signed in to change notification settings - Fork 0
Sonnet 4.6 Stock ClimbMix Agent Run Mar 26 2026
Sonnet 4.6 stock-baseline run: Claude Sonnet 4.6 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). Part of the fair-baseline methodology — every model starts from identical, unoptimized hyperparameters.
Key finding: Sonnet 4.6 is the most systematic architecture explorer. It walked ASPECT_RATIO from 64 down to 19 in six consecutive keeps, discovered a unique WINDOW_PATTERN=SLLL, and reversed its own MATRIX_LR decision (0.04→0.06→0.04→0.035) as the architecture evolved. But it missed depth reduction entirely, leaving the final win to Haiku.
Status: Complete (100/100 experiments)
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 100 (exp0–exp99) |
| Kept | 15 (15.0%) |
| Discarded | 81 (81.0%) |
| Crashed | 4 (4.0%) |
| Best val_bpb | 1.3093 (exp91) |
| Baseline val_bpb | 1.3859 (stock defaults, AR=64) |
| Total Improvement | −0.0766 (5.53%) |
| LLM Backend | Claude Sonnet 4.6 |
| Results | results/sonnet-4-6/climbmix/ |
| Parameter | Stock Default |
|---|---|
| ASPECT_RATIO | 64 |
| HEAD_DIM | 128 |
| WINDOW_PATTERN | SSSL |
| MATRIX_LR | 0.04 |
| EMBEDDING_LR | 0.6 |
| UNEMBEDDING_LR | 0.004 |
| SCALAR_LR | 0.5 |
| WEIGHT_DECAY | 0.2 |
| ADAM_BETAS | (0.8, 0.95) |
| WARMDOWN_RATIO | 0.5 |
| FINAL_LR_FRAC | 0.0 |
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 1.3859 | — | Baseline (stock defaults) | Starting point |
| exp2 | 1.3842 | −0.12% | MATRIX_LR 0.04 → 0.06 | Higher LR (later reversed) |
| exp27 | 1.3829 | −0.22% | ASPECT_RATIO 64 → 48 | Architecture walk begins |
| exp28 | 1.3811 | −0.35% | ASPECT_RATIO 48 → 36 | Narrowing continues |
| exp29 | 1.3345 | −3.71% | ASPECT_RATIO 36 → 28 | Major throughput jump |
| exp30 | 1.3267 | −4.27% | ASPECT_RATIO 28 → 20 | Deeper architecture |
| exp33 | 1.3251 | −4.39% | ASPECT_RATIO 20 → 18 | Near-optimal AR |
| exp44 | 1.3248 | −4.41% | ASPECT_RATIO 18 → 19 | Fine-tuned AR upward |
| exp46 | 1.3245 | −4.43% | MATRIX_LR 0.06 → 0.05 | LR walk-back begins |
| exp47 | 1.3212 | −4.67% | MATRIX_LR 0.05 → 0.04 | Reverted to stock! |
| exp57 | 1.3208 | −4.70% | SCALAR_LR 0.5 → 0.4 | Lower scalar LR |
| exp58 | 1.3178 | −4.91% | SCALAR_LR 0.4 → 0.3 | Further reduction |
| exp60 | 1.3173 | −4.95% | WARMDOWN_RATIO 0.5 → 0.55 | Slightly longer warmdown |
| exp69 | 1.3151 | −5.11% | UNEMBEDDING_LR 0.004 → 0.006 | Output layer tuning |
| exp90 | 1.3106 | −5.43% | WINDOW_PATTERN SSSL → SLLL | Novel pattern discovery |
| exp91 | 1.3093 | −5.53% | MATRIX_LR 0.04 → 0.035 | Combined with SLLL |
Phase 1 — Initial LR Exploration (exp1–26, 26 experiments, 1 keep): Broad sweeps of all learning rates, betas, weight decay, warmup, warmdown. Only one keep: MATRIX_LR→0.06 (exp2). Three crashes (depth+1, AR=80, batch size). This phase established that stock LRs were already near-optimal — the real gains would come from architecture.
Phase 2 — Architecture Walk (exp27–34, 8 experiments, 6 keeps): The breakthrough phase. S4.6 systematically walked ASPECT_RATIO: 64→48→36→28→20→18, with every step a keep. Then tested AR=14 (discard — too narrow) and AR=16 (discard). Settled on AR=18, then fine-tuned to AR=19 at exp44. This is S4.6's signature strength: methodical, patient architecture exploration.
Phase 3 — LR Reversal (exp35–56, 22 experiments, 3 keeps): Having changed the architecture, S4.6 realized the original LR boost was wrong. Walked MATRIX_LR back: 0.06→0.05→0.04 (reverting to stock). Also discovered SCALAR_LR=0.3. This self-correcting behavior — boosting a param, then undoing it when context changed — is unique to S4.6.
Phase 4 — Fine-Tuning Plateau (exp57–89, 33 experiments, 2 keeps): Explored WARMDOWN_RATIO (0.55 kept), UNEMBEDDING_LR (0.006 kept), betas (all discards), weight decay (all discards), HEAD_DIM=64 (discard), depth+1 (discard), various window patterns. Long plateau with diminishing returns.
Phase 5 — Window Pattern Discovery (exp87–91, 5 experiments, 2 keeps): Late-game breakthrough. Tested SSLL (discard), SSGL (crash), SLSL (discard), then found SLLL (keep at exp90). Combined with MATRIX_LR=0.035 for the final best at exp91.
Phase 6 — Final Sweep (exp92–99, 8 experiments, 0 keeps): Attempted MATRIX_LR variations, SCALAR_LR, WARMDOWN_RATIO, EMBEDDING_LR, UNEMBEDDING_LR — all discards. Optimum appears stable.
S4.6's defining characteristic is patient, systematic architecture search. The AR walk (64→48→36→28→20→18→19) tested every major step and fine-tuned the optimum. More remarkably, S4.6 reversed its own MATRIX_LR decision when the architecture changed — the only model to self-correct a previously successful change.
| Exp | MATRIX_LR | AR | val_bpb | Context |
|---|---|---|---|---|
| exp0 | 0.04 | 64 | 1.3859 | Stock default |
| exp2 | 0.06 | 64 | 1.3842 | Boosted (keep) |
| exp46 | 0.05 | 19 | 1.3245 | Walking back |
| exp47 | 0.04 | 19 | 1.3212 | Reverted to stock! |
| exp91 | 0.035 | 19 | 1.3093 | Below stock — architecture needed lower LR |
S4.6 understood that a narrower architecture (AR=19) needed gentler weight updates. This adaptive LR strategy is absent in both S4.0 and Haiku.
| Model | AR Walk | Final AR | Experiments Used |
|---|---|---|---|
| Sonnet 4.0 | Never explored | 64 (stock) | 0 |
| Sonnet 4.6 | 64→48→36→28→20→18→19 | 19 | 8 (6 keeps) |
| Haiku 4.5 | 64→48→32 | 32 | 2 (2 keeps) |
S4.6 explored AR most thoroughly and found the narrowest optimal value. Its AR=19 is close to the AR=21 it found in pre-optimized runs, confirming this is a consistent S4.6 preference.
4 crashes (4.0%):
| Exp | Parameter | Issue |
|---|---|---|
| exp3 | DEPTH + 1 | OOM or instability |
| exp16 | ASPECT_RATIO 64 → 80 | Model too wide |
| exp25 | TOTAL_BATCH_SIZE reduced | Numerical instability |
| exp88 | WINDOW_PATTERN SSGL | Invalid pattern crash |
Crashes span the full run (exp3–88), unlike Haiku (all in first 5). S4.6 occasionally takes risky exploratory steps.
| Exp | MFU | tok/sec | Peak Mem (GB) | Steps |
|---|---|---|---|---|
| exp0 (baseline AR=64) | 22.4% | 25,291 | 26.1 | 233 |
| exp29 (AR=28) | 13.9% | 45,272 | 15.5 | 414 |
| exp33 (AR=18) | 14.3% | 46,651 | 15.5 | 427 |
| exp47 (MATRIX_LR=0.04) | 15.6% | 50,616 | 15.5 | 465 |
| exp91 (final best) | 18.4% | 51,615 | 15.5 | 474 |
Throughput doubled (25K→52K tok/sec) and memory dropped 41% (26.1→15.5 GB) through AR optimization. Steps increased 2.0× (233→474). Compare: Haiku achieved 76K tok/sec and 697 steps via depth reduction.
# Sonnet 4.6 ClimbMix Best (exp91) — Stock Baseline
ASPECT_RATIO = 19 # was 64 (stock) — walked 64→48→36→28→20→18→19
HEAD_DIM = 128 # unchanged
WINDOW_PATTERN = "SLLL" # was "SSSL" — novel pattern
MATRIX_LR = 0.035 # was 0.04 — went UP to 0.06, then back DOWN below stock
EMBEDDING_LR = 0.6 # unchanged
UNEMBEDDING_LR = 0.006 # was 0.004
SCALAR_LR = 0.3 # was 0.5
WEIGHT_DECAY = 0.2 # unchanged
ADAM_BETAS = (0.8, 0.95) # unchanged (tried betas, no keeps)
WARMDOWN_RATIO = 0.55 # was 0.5
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3093 | mem: 15.5 GB | tok/sec: 51.6K | MFU: 18.4% | 474 stepsS4.6's run reveals a unique optimization personality: it adapts its strategy as context changes. The MATRIX_LR journey (stock → higher → reverted → below stock) shows that S4.6 doesn't just find good values — it re-evaluates past decisions when the landscape shifts.
This is fundamentally different from:
- S4.0, which never changed architecture and micro-tuned the same parameters endlessly
- Haiku, which binary-searched parameters in isolation until a late synergy emerged
S4.6's weakness was missing depth reduction entirely — a dimension that gave Haiku an extra 220+ gradient steps and the winning result.
Run complete (100/100 experiments). Sonnet 4.6 achieves val_bpb = 1.3093, 2nd place behind Haiku's 1.2953.
See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.0 Stock ClimbMix (Mar 26)