-
Notifications
You must be signed in to change notification settings - Fork 0
Opus 4.6 Stock ClimbMix Agent Run Mar 26 2026
Opus 4.6 stock-baseline run: Claude Opus 4.6 optimizing the climbmix-400b dataset from stock upstream defaults (AR=64, stock LRs). The fourth and final model in the fair-baseline ClimbMix comparison.
Key finding: The most expensive model finishes 3rd. Opus never discovered architecture changes despite trying both directions (AR=56 too timid, AR=68/80 catastrophic). With 13 crashes (13% — highest of any model) and no architecture innovation, Opus relied purely on optimizer tuning to reach 1.3569 — barely edging past Sonnet 4.0 (1.3588) but far behind architecture-discovering models Haiku (1.2953) and Sonnet 4.6 (1.3093).
Status: Complete (100/100 experiments)
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 100 (exp0–exp99) |
| Kept | 12 (12.0%) |
| Discarded | 75 (75.0%) |
| Crashed | 13 (13.0%) |
| Best val_bpb | 1.3569 (exp63) |
| Effective Baseline | 1.4098 (exp4 — exp0 crashed) |
| Total Improvement | −0.0529 (3.75%) |
| LLM Backend | Claude Opus 4.6 |
| Results | results/opus-4-6/climbmix/ |
Opus is the only model that crashed on stock defaults:
| Exp | Description | Status |
|---|---|---|
| exp0 | Baseline (no modifications) | CRASH |
| exp1 | "Fix crash by setting explicit conservative hyperparameters" | CRASH |
| exp2 | "Set minimal conservative hyperparameters with small batch size" | CRASH |
| exp3 | "Debug crash by using very minimal hardcoded hyperparameters with depth=4" | CRASH |
| exp4 | "Use dynamic defaults directly without any overrides" | ✅ 1.4098 |
Four consecutive crashes before Opus discovered that not overriding defaults was the fix. The irony: exp1-3 crashed because Opus's manual "fixes" broke the dynamic config system. Compare: S4.0, S4.6, and Haiku all ran the baseline perfectly on first try.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp4 | 1.4098 | — | Effective baseline (dynamic defaults) | Starting point |
| exp8 | 1.3898 | −1.42% | DEVICE_BATCH_SIZE=8 | Throughput optimization |
| exp29 | 1.3857 | −1.71% | MATRIX_LR 0.04 → 0.045 | LR walk begins |
| exp30 | 1.3820 | −1.97% | MATRIX_LR 0.045 → 0.05 | Continued walk |
| exp31 | 1.3799 | −2.12% | MATRIX_LR → 0.048, WARMDOWN_RATIO → 0.4 | First multi-param |
| exp34 | 1.3798 | −2.13% | WEIGHT_DECAY 0.2 → 0.15 | Reduced regularization |
| exp45 | 1.3764 | −2.37% | MATRIX_LR 0.048 → 0.05 | Revisiting higher LR |
| exp46 | 1.3761 | −2.39% | MATRIX_LR 0.05 → 0.052 | Continued walk |
| exp47 | 1.3710 | −2.75% | MATRIX_LR 0.052 → 0.054 | Best LR found |
| exp56 | 1.3685 | −2.93% | SCALAR_LR 0.5 → 0.4 | New dimension |
| exp58 | 1.3664 | −3.08% | SCALAR_LR 0.4 → 0.45 | Fine-tuned upward |
| exp63 | 1.3569 | −3.75% | EMBEDDING_LR 0.6 → 0.4 | Final best |
Phase 1 — Crash Recovery (exp0–4, 5 experiments, 1 keep): Four crashes before establishing a working baseline. Opus over-engineered "fixes" that broke the dynamic config. 5% of budget wasted before training even started.
Phase 2 — Broad Exploration (exp5–28, 24 experiments, 1 keep): Found DEVICE_BATCH_SIZE=8 (exp8). Then catastrophic detours — tried making the model bigger: depth=10 (val_bpb=2.0016), AR=80 (val_bpb=1.6011), depth=8+batch=16 (crash). Three more crashes from batch/total_batch size changes. Opus explored in the wrong direction on nearly every parameter.
Phase 3 — MATRIX_LR Walk (exp29–34, 6 experiments, 4 keeps): The productive phase begins. Systematic MATRIX_LR walk: 0.045→0.05→0.048. Combined with WARMDOWN_RATIO=0.4 and WEIGHT_DECAY=0.15. Four keeps in 6 experiments — Opus's best efficiency.
Phase 4 — Continued LR Optimization (exp35–55, 21 experiments, 3 keeps): Extended MATRIX_LR walk: 0.05→0.052→0.054 (3 more keeps). Tried window pattern change but accidentally set it to the same value ("SSSL"→"SSSL"). HEAD_DIM=64 was catastrophic (1.5327). β1=0.85 (wrong direction), β2=0.92 tested.
Phase 5 — LR Reduction Discovery (exp56–63, 8 experiments, 3 keeps): Pivot to reducing learning rates. Found SCALAR_LR=0.45 (exp58) and EMBEDDING_LR=0.4 (exp63) — the final best. Notable: Opus initially tried increasing these LRs (both failed), then reversed direction.
Phase 6 — Final 37 Experiments (exp64–99, 37 experiments, 0 keeps): The longest drought of any model. Opus explored every remaining dimension: EMBEDDING_LR micro-tuning (0.3, 0.35, 0.38, 0.42, 0.45 — all worse), β1 (0.75, 0.78 — worse), β2 (0.93, 0.94, 0.97 — worse), DEVICE_BATCH_SIZE (9, 10, 12 — all crashed), AR=56 (discard), AR=68 (catastrophic), window patterns (SSSS, SSLL — discards), MLP_RATIO (3.0, 3.5 — discards), FINAL_LR_FRAC (0.05 — discard). Nothing improved in the final 37% of experiments.
At experiment 82, Opus finally tried reducing ASPECT_RATIO (64→56). But 56 was too timid — val_bpb=1.4184, worse than baseline. Then at exp89, Opus went the other direction (64→68) — catastrophic at 1.5291. Opus concluded AR changes don't work and abandoned the idea.
The tragedy: if Opus had reduced AR to 32 (like Haiku) or 19 (like S4.6), it would have roughly doubled throughput to ~500+ steps and dramatically improved results. But testing only AR=56 (too close to 64 to matter) and AR=68 (wrong direction) gave Opus exactly the wrong signal about architecture's potential.
| Model | AR Values Tested | Found Improvement? |
|---|---|---|
| Haiku | 48, 32 | Yes — 2 keeps, massive improvement |
| S4.6 | 48, 36, 28, 20, 18, 19, 14, 16, 80 | Yes — 6 keeps |
| Opus | 56, 68, 80 | No — all worse |
| S4.0 | Never tested | N/A |
Opus tested 3 AR values but never went below 56 — the critical gap. Architecture reduction requires bold steps, not timid ones.
Opus takes small, methodical steps. The MATRIX_LR walk (0.04→0.045→0.05→0.052→0.054) is its cleanest pattern. But this incrementalism is fatal for architecture: AR=56 is too close to 64 to unlock the throughput gains that AR=32 or AR=19 provide.
Both Opus and S4.0 are "pure optimizer tuners" — neither effectively changed architecture. But they took opposite LR strategies:
| Parameter | S4.0 Direction | Opus Direction | Winner |
|---|---|---|---|
| MATRIX_LR | 0.04 → 0.0673 (+68%) | 0.04 → 0.054 (+35%) | S4.0 higher |
| EMBEDDING_LR | 0.6 → 0.8 (+33%) | 0.6 → 0.4 (−33%) | Opposite directions |
| SCALAR_LR | 0.5 (unchanged) | 0.5 → 0.45 (−10%) | Opus lower |
| β1 | 0.8 → 0.635 (−21%) | 0.8 (unchanged) | S4.0 walks β1 |
| WARMDOWN_RATIO | 0.5 → 0.3 (−40%) | 0.5 → 0.4 (−20%) | S4.0 more aggressive |
| WEIGHT_DECAY | 0.2 (unchanged) | 0.2 → 0.15 (−25%) | Opus reduces |
S4.0 goes higher on LRs and lower on momentum. Opus goes lower on most LRs. Different paths to nearly identical results (1.3569 vs 1.3588) — confirming the multi-optima landscape.
Opus showed adaptive behavior: initially tried increasing EMBEDDING_LR (0.6→1.0, discard at exp15) and SCALAR_LR (0.5→0.8, discard at exp21), then later discovered that decreasing them was the key (EMBEDDING_LR→0.4, SCALAR_LR→0.45). This reversal echoes S4.6's MATRIX_LR self-correction.
13 crashes (13.0%) — highest of any model by far:
| Category | Experiments | Count |
|---|---|---|
| Baseline over-engineering | exp0, 1, 2, 3 | 4 |
| Making model bigger | exp7, 12, 23 | 3 |
| Batch size changes | exp24, 25, 79, 80, 87, 88 | 6 |
| Model | Crash Rate | Crash Count |
|---|---|---|
| S4.0 | 0.0% | 0 |
| Haiku | 3.0% | 3 |
| S4.6 | 4.0% | 4 |
| Opus | 13.0% | 13 |
Opus crashes 4× more than any other model. The batch size sensitivity is unique — exp79 (BS=12), exp80 (BS=10), exp87 (BS=9) all crashed, while other models handled similar changes fine.
| Exp | MFU | tok/sec | Peak Mem (GB) | Steps |
|---|---|---|---|---|
| exp4 (baseline) | 20.2% | 22,984 | 24.8 | 211 |
| exp8 (DEVICE_BS=8) | 22.0% | 25,000 | 25.0 | 229 |
| exp47 (MATRIX_LR=0.054) | 22.9% | 25,930 | 25.0 | 239 |
| exp63 (final best) | 25.1% | 28,267 | 25.0 | 260 |
Memory essentially unchanged (24.8–25.0 GB) — no architecture changes. Throughput improved only 23% (23K→28K). Steps increased 23% (211→260). Compare:
| Model | Final Steps | Final tok/sec | Peak Memory |
|---|---|---|---|
| S4.0 | 251 | 27.3K | 26.1 GB |
| Opus | 260 | 28.3K | 25.0 GB |
| S4.6 | 474 | 51.6K | 15.5 GB |
| Haiku | 697 | 76.0K | 11.6 GB |
The two architecture-blind models (S4.0, Opus) cluster together. The two architecture-exploring models (S4.6, Haiku) cluster together. The gap between clusters is enormous.
# Opus 4.6 ClimbMix Best (exp63) — 3rd place
ASPECT_RATIO = 64 # UNCHANGED — tried 56 (too timid), 68/80 (catastrophic)
HEAD_DIM = 128 # unchanged (tried 64, catastrophic)
WINDOW_PATTERN = "SSSL" # UNCHANGED — tried SSSS, SSLL (discards)
DEPTH = default # UNCHANGED — increases all failed
DEVICE_BATCH_SIZE = 8 # changed from default
MATRIX_LR = 0.054 # was 0.04 — walked up in 5 keeps
EMBEDDING_LR = 0.4 # was 0.6 — reduced (initial increase failed)
SCALAR_LR = 0.45 # was 0.5 — reduced
WEIGHT_DECAY = 0.15 # was 0.2 — reduced
WARMDOWN_RATIO = 0.4 # was 0.5 — reduced
ADAM_BETAS = (0.8, 0.95) # unchanged (both directions tested, none kept)
FINAL_LR_FRAC = 0.0 # unchanged
# Result: 1.3569 | mem: 25.0 GB | tok/sec: 28.3K | MFU: 25.1% | 260 stepsOpus's 100 experiments prove the same lesson as S4.0: you cannot optimizer-tune your way past an architecture bottleneck. Both models achieved the highest MFU (24–25%) by perfectly utilizing the compute they had — but they had the fewest gradient steps (251–260) because they never reduced the model's compute footprint.
| Rank | Model | val_bpb | Steps | Strategy | Architecture? |
|---|---|---|---|---|---|
| 1st | Haiku | 1.2953 | 697 | Binary-search → late synergy | ✅ AR=32, depth−3, LLLL |
| 2nd | S4.6 | 1.3093 | 474 | Systematic explorer | ✅ AR=19, SLLL |
| 3rd | Opus | 1.3569 | 260 | Cautious iterative | ❌ Tried, too timid |
| 4th | S4.0 | 1.3588 | 251 | Conservative optimizer | ❌ Never tried |
The ranking perfectly correlates with gradient steps, not model capability or cost. Bigger LLM ≠ better optimizer. Smaller model ≠ worse optimizer. What matters is whether the LLM discovers that architecture changes dominate optimizer tuning in time-constrained training.
Run complete (100/100 experiments). Opus 4.6 finishes 3rd with val_bpb = 1.3569.
See also: Cross-LLM Comparison | Haiku 4.5 ClimbMix (Mar 25) | Sonnet 4.6 Stock ClimbMix (Mar 26) | Sonnet 4.0 Stock ClimbMix (Mar 26)