-
Notifications
You must be signed in to change notification settings - Fork 0
Sonnet 4.6 Cosmopedia v2 Agent Run Mar 24 2026
Third cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing Cosmopedia-v2 on the same hardware previously used by Claude Sonnet 4.0. This is the most surprising comparison yet because Sonnet 4.6 discovered ASPECT_RATIO=21 — breaking the universal AR=32 consensus that held across all five Sonnet 4.0 datasets.
Key finding: Sonnet 4.6 found a fundamentally different architecture (AR=21 vs AR=32) and beat Sonnet 4.0's best val_bpb by 0.6% (0.9549 vs 0.9606). With 16 keeps (vs 4), zero crashes (vs 2), and a systematic AR walk from 32→24→22→21, this run demonstrates that the "AR=32 is hardware-optimal" conclusion from Sonnet 4.0 was actually an artifact of the LLM's exploration strategy, not a true hardware constraint.
| Spec | Value |
|---|---|
| Chip | Apple M5 Max |
| Unified Memory | 64 GB |
| GPU Cores | 40 |
| Backend | MLX |
| Time Budget | 5 minutes per experiment |
| Metric | Value |
|---|---|
| Total Experiments | 101 (exp0–exp100) |
| Kept | 16 (16.0%) |
| Discarded | 84 (84.0%) |
| Crashed | 0 (0.0%) |
| Best val_bpb | 0.9549 (exp96) |
| Baseline val_bpb | 0.9806 (defaults) |
| Total Improvement | −0.0258 (2.63%) |
| LLM Backend | Claude Sonnet 4.6 |
| Results | results/sonnet-4-6/cosmopedia-v2/ |
| Metric | Sonnet 4.0 | Sonnet 4.6 |
|---|---|---|
| Baseline val_bpb | 0.9640 (pre-optimized) | 0.9806 (defaults) |
| Best val_bpb | 0.9606 | 0.9549 |
| Improvement | −0.35% | −2.63% |
| Keeps | 4 (3.9%) | 16 (16.0%) |
| Crashes | 2 (1.9%) | 0 (0.0%) |
Sonnet 4.0 started from a pre-optimized baseline (AR=32, inherited from prior runs), while Sonnet 4.6 started from reset defaults. Despite starting from a worse baseline, Sonnet 4.6 found a significantly better absolute result — the first time across any dataset comparison that the later model has clearly and unambiguously won.
| Exp | val_bpb | Δ from Baseline | Description | Key Change |
|---|---|---|---|---|
| exp0 | 0.9806 | — | Baseline (defaults) | Starting point |
| exp5 | 0.9803 | −0.03% | ADAM β2 0.975→0.95 | More responsive optimizer |
| exp8 | 0.9754 | −0.53% | WEIGHT_DECAY 0.2→0.1 | Lower regularization |
| exp11 | 0.9745 | −0.62% | EMBEDDING_LR 0.45→0.60 | Faster embedding learning |
| exp14 | 0.9716 | −0.92% | MATRIX_LR 0.053→0.062 | Higher matrix LR |
| exp18 | 0.9681 | −1.28% | WARMDOWN_RATIO 0.77→0.65 | Shorter warmdown |
| exp28 | 0.9662 | −1.47% | ASPECT_RATIO 32→24 | Architecture change |
| exp30 | 0.9635 | −1.75% | MATRIX_LR 0.062→0.068 | Re-tune for AR=24 |
| exp42 | 0.9629 | −1.81% | FINAL_LR_FRAC 0.1→0.02 | Lower final LR |
| exp59 | 0.9600 | −2.10% | ASPECT_RATIO 24→22 | Deeper architecture |
| exp60 | 0.9598 | −2.12% | ADAM β1 0.67→0.65 | Less momentum |
| exp61 | 0.9595 | −2.16% | ASPECT_RATIO 22→21 | Even deeper |
| exp65 | 0.9580 | −2.30% | EMBEDDING_LR 0.60→0.50 | Re-tune for AR=21 |
| exp69 | 0.9575 | −2.35% | SCALAR_LR 0.23→0.18 | Lower scalar LR |
| exp70 | 0.9556 | −2.56% | SCALAR_LR 0.18→0.13 | Even lower scalar LR |
| exp90 | 0.9553 | −2.58% | WARMDOWN_RATIO 0.65→0.66 | Fine-tune warmdown |
| exp96 | 0.9549 | −2.63% | ADAM β2 0.95→0.955 | Fine-tune β2 |
Phase 1 — Optimizer Tuning (exp1–exp8): Reduced β2 from 0.975 to 0.95 for faster adaptation, halved weight decay from 0.2 to 0.1. Two keeps established a better optimizer baseline.
Phase 2 — Learning Rate Rebalancing (exp9–exp18): Increased EMBEDDING_LR and MATRIX_LR, shortened WARMDOWN_RATIO from 0.77 to 0.65. Three more keeps, moving val_bpb from 0.9754 to 0.9681.
Phase 3 — Architecture Discovery (exp28–exp42): The pivotal phase. Reduced ASPECT_RATIO from 32→24 (keep), re-tuned MATRIX_LR for the new architecture, then reduced FINAL_LR_FRAC to 0.02. Three keeps, establishing the narrower architecture as superior.
Phase 4 — Architecture Walk (exp59–exp70): Systematically walked ASPECT_RATIO down: 24→22→21 (both keeps). Then re-tuned learning rates for the deeper model — EMBEDDING_LR down to 0.50, SCALAR_LR down through 0.18 to 0.13 (two consecutive keeps). Six keeps in this phase alone.
Phase 5 — Fine-Tuning (exp71–exp96): Exhaustive sweep of all parameters. WARMDOWN_RATIO 0.65→0.66 (tiny keep) and ADAM β2 0.95→0.955 (final keep). Most experiments in this phase confirmed the configuration was near-optimal.
This is the signature finding of this run. The systematic walk:
| AR | val_bpb | Steps | tok/sec | mem |
|---|---|---|---|---|
| 32 (default) | 0.9806 | 437 | 47.6K | 14.5 GB |
| 24 (exp28) | 0.9662 | 480 | 52.2K | 14.5 GB |
| 22 (exp59) | 0.9600 | 510 | 55.6K | 14.5 GB |
| 21 (exp61) | 0.9595 | 511 | 55.6K | 14.5 GB |
| 20 (exp58, discarded) | 0.9686 | 480 | 52.3K | 14.5 GB |
AR=20 was tried before 22 and 21 (at exp58, discarded). The sweet spot is AR=21–22, where the model gets 17% more gradient steps than AR=32 while remaining wider than the over-narrow AR=20. Sonnet 4.0 never explored below AR=32 on any dataset.
# Sonnet 4.6 Cosmopedia-v2 Best (exp96)
ASPECT_RATIO = 21 # was 32 — narrower, deeper model
MATRIX_LR = 0.068 # was 0.053
EMBEDDING_LR = 0.50 # was 0.45
SCALAR_LR = 0.13 # was 0.23 — dramatically lower
UNEMBEDDING_LR = 0.006 # unchanged
WEIGHT_DECAY = 0.1 # was 0.2
WARMDOWN_RATIO = 0.66 # was 0.77
FINAL_LR_FRAC = 0.02 # was 0.1
ADAM_BETAS = (0.65, 0.955) # was (0.67, 0.975)
# Result: 0.9549 | mem: 14.5 GB | tok/sec: 57.3K | MFU: 17.6% | 524 steps| Parameter | Sonnet 4.0 | Sonnet 4.6 | Same? |
|---|---|---|---|
| ASPECT_RATIO | 32 | 21 | Major divergence |
| MATRIX_LR | 0.0435 | 0.068 | ❌ (56% higher) |
| EMBEDDING_LR | 0.4 | 0.50 | ❌ |
| SCALAR_LR | 0.4 | 0.13 | ❌ (68% lower) |
| UNEMBEDDING_LR | 0.0033 | 0.006 | ❌ (82% higher) |
| WEIGHT_DECAY | 0.15 | 0.1 | ❌ |
| WARMDOWN_RATIO | 0.5 | 0.66 | ❌ |
| FINAL_LR_FRAC | 0.0 | 0.02 | ❌ |
| ADAM β1 | 0.8 | 0.65 | ❌ |
| ADAM β2 | 0.95 | 0.955 | ≈ |
Every single parameter differs. The ASPECT_RATIO divergence (32 vs 21) is the most dramatic — it represents a fundamentally different model architecture. Sonnet 4.6 traded model width for depth, getting a narrower but deeper network that runs more gradient steps in the time budget. The SCALAR_LR divergence (0.4 vs 0.13) is the second-largest — a 68% reduction.
See also: Cross-LLM Comparison | Cross-Dataset Comparison | Sonnet 4.0 Cosmopedia-v2 (Mar 20)