Skip to content

Sonnet 4.6 FineWeb Edu Agent Run Mar 22 2026

Jump to bottom
Dave Graham edited this page Mar 23, 2026 · 1 revision

Sonnet 4.6 FineWeb-Edu Agent Run — Mar 22, 2026 (M5 Max)

Overview

Second cross-generation LLM comparison run: Claude Sonnet 4.6 optimizing FineWeb-Edu on the same hardware previously used by Claude Sonnet 4.0. This is the most informative comparison yet because FineWeb-Edu was Sonnet 4.0's strongest dataset (19.3% keep rate, 17 keeps).

Key finding: Both models converge to essentially the same final val_bpb (1.3416 vs 1.3424), but via completely different optimization paths. Sonnet 4.6 started from a much better baseline (1.371 vs 1.409) — meaning the defaults improved between model generations — and still found 18 keeps. The final configs are substantially different, suggesting multiple near-equivalent optima in the educational text optimization landscape.

Hardware

Spec Value
Chip Apple M5 Max
Unified Memory 64 GB
GPU Cores 40
Backend MLX
Time Budget 5 minutes per experiment

Summary Statistics

Metric Value
Total Experiments 100 (exp0–exp100, exp42 missing)
Kept 18 (18.0%)
Discarded 80 (80.0%)
Crashed 1 (1.0%)
Best val_bpb 1.3416 (exp98)
Baseline val_bpb 1.3710 (defaults)
Total Improvement −0.0294 (2.14%)
LLM Backend Claude Sonnet 4.6
Results results/sonnet-4-6/fineweb-edu/

⚠️ Baseline Context

This run started from reset defaults, not the same defaults as the Sonnet 4.0 FineWeb-Edu run:

  • Sonnet 4.6 baseline: 1.3710 (from defaults — better defaults than S4.0)
  • Sonnet 4.0 baseline: 1.4088 (from defaults — original defaults)
  • Sonnet 4.0's absolute best: 1.3424 — very close to Sonnet 4.6's 1.3416
  • Sonnet 4.0 achieved 4.71% improvement vs Sonnet 4.6's 2.14% — but S4.0 had more room to work with

The different baselines reflect improved default hyperparameters between model generations. Both models converge to nearly identical absolute performance.

Cumulative Improvements (Kept Experiments)

Exp val_bpb Δ from Baseline Description Key Change
exp0 1.3710 — Baseline (defaults) Starting point
exp3 1.3675 −0.25% DEVICE_BATCH_SIZE / 2 More gradient steps
exp11 1.3645 −0.47% WARMDOWN_RATIO 0.63 → 0.75 Longer LR annealing
exp15 1.3628 −0.60% FINAL_LR_FRAC 0.0 → 0.1 Residual LR at end
exp18 1.3609 −0.74% MATRIX_LR 0.05 → 0.06 Higher matrix LR
exp31 1.3574 −0.99% ASPECT_RATIO → 24 Architecture change
exp32 1.3527 −1.33% ASPECT_RATIO → 32 Wider model
exp47 1.3527 −1.34% ADAM β1 0.71 → 0.68 Less momentum
exp51 1.3518 −1.40% SCALAR_LR 0.3 → 0.25 Lower scalar LR
exp52 1.3508 −1.47% MATRIX_LR 0.06 → 0.055 Fine-tune matrix LR
exp55 1.3504 −1.50% ADAM β2 0.95 → 0.96 Stabilize gradients
exp56 1.3497 −1.55% ADAM β2 0.96 → 0.97 More stable
exp57 1.3478 −1.69% ADAM β2 0.97 → 0.98 Even more stable
exp73 1.3463 −1.80% UNEMBEDDING_LR 0.004 → 0.006 Higher unembed LR
exp78 1.3439 −1.97% ADAM β2 0.98 → 0.975 Fine-tune β2
exp92 1.3438 −1.98% WARMDOWN_RATIO 0.75 → 0.77 Fine-tune warmdown
exp93 1.3425 −2.08% EMBEDDING_LR 0.4 → 0.45 Higher embed LR
exp94 1.3417 −2.14% SCALAR_LR 0.25 → 0.23 Fine-tune scalar LR
exp98 1.3416 −2.14% ADAM β1 0.68 → 0.67 Optimal momentum

Optimization Phases

Phase 1 — Initial Exploration (exp1–10, 10 experiments): Broad sweeps of batch size, LR values, weight decay, warmdown. Only exp3 (smaller device batch) kept. Established that more gradient steps help.

Phase 2 — LR Schedule Discovery (exp11–18, 8 experiments): Three consecutive keeps: WARMDOWN_RATIO=0.75 (exp11), FINAL_LR_FRAC=0.1 (exp15), MATRIX_LR=0.06 (exp18). Sonnet 4.6 recognized the LR schedule as the most productive optimization dimension.

Phase 3 — Architecture Search (exp19–33, 15 experiments): Tested depth changes, Adam betas, various LRs. Then discovered ASPECT_RATIO as a lever — tried 24, then 32 (two successive keeps). AR=32 gave the biggest single-experiment jump (−0.005 bpb).

Phase 4 — Re-optimization for Wide Model (exp34–46, 13 experiments): Tried re-tuning warmdown, betas, weight decay for AR=32. All discarded — the model's old hyperparams transferred well to the wider architecture.

Phase 5 — Systematic Optimizer Tuning (exp47–57, 11 experiments): Four consecutive keeps on Adam β2 (0.95→0.96→0.97→0.98). Also refined β1 to 0.68, scalar LR to 0.25, matrix LR to 0.055. Most productive phase by count.

Phase 6 — Deep Fine-tuning (exp58–78, 21 experiments): Exhaustive sweeps around the current best. Found UNEMBEDDING_LR=0.006 (exp73) and refined β2 to 0.975 (exp78). Increasingly small deltas.

Phase 7 — Final Polish (exp79–100, 22 experiments): Warmdown fine-tuned to 0.77, embedding LR to 0.45, scalar LR to 0.23, β1 to 0.67. Four final keeps squeezing the last ~0.002 bpb.

Hardware Utilization

Exp MFU tok/sec Peak Mem (GB) Steps
exp0 (baseline) 14.7% 47,872 15.5 438
exp3 15.1% 48,934 14.5 449
exp32 16.8% 54,593 14.5 500
exp57 17.5% 56,943 14.5 521
exp78 17.6% 56,905 14.5 522
exp98 (best) 17.9% 58,236 14.5 533

MFU rose 21.8%, throughput rose 21.6%, steps rose 21.7%. Memory dropped from 15.5 to 14.5 GB. Every optimization that improved val_bpb also improved hardware utilization.

Crash Analysis

1 crash (1.0%) — the lowest crash rate of any run:

Exp Parameter Issue
exp39 DEVICE_BATCH_SIZE / 3 Integer division issue

Sonnet 4.6 showed excellent crash avoidance on FineWeb-Edu, learning from the ClimbMix run's batch size failures.

Best Configuration

# Architecture
ASPECT_RATIO = 32
DEPTH = 8
HEAD_DIM = 128
WINDOW_PATTERN = "SSSL"
MLP_RATIO = 4.0

# Optimization (9 parameters changed from defaults)
DEVICE_BATCH_SIZE = default / 2     # More gradient steps
MATRIX_LR = 0.055                   # Default: 0.05
SCALAR_LR = 0.23                    # Default: 0.3
EMBEDDING_LR = 0.45                 # Default: 0.4
UNEMBEDDING_LR = 0.006              # Default: 0.004
ADAM_BETAS = (0.67, 0.975)          # Default: (0.71, 0.95)
WARMDOWN_RATIO = 0.77               # Default: 0.63
FINAL_LR_FRAC = 0.1                 # Default: 0.0
WEIGHT_DECAY = 0.2                  # Default (unchanged)

# Performance
# val_bpb: 1.3416 | mem: 14.5 GB | steps: 533 | MFU: 17.9%

Key Insight: Convergent Outcomes, Divergent Paths

The most striking finding is that Sonnet 4.0 and Sonnet 4.6 arrive at nearly identical final val_bpb (1.3424 vs 1.3416) via completely different configurations:

Parameter Sonnet 4.0 Best Sonnet 4.6 Best Same?
ASPECT_RATIO 32 32 ✅
MATRIX_LR 0.053 0.055 ≈
EMBEDDING_LR 0.59 0.45 ❌
SCALAR_LR 0.3 0.23 ❌
UNEMBEDDING_LR 0.003 0.006 ❌
WEIGHT_DECAY 0.12 0.2 ❌
WARMDOWN_RATIO 0.47 0.77 ❌
FINAL_LR_FRAC 0.0 0.1 ❌
ADAM β1 0.8 0.67 ❌
ADAM β2 0.95 0.975 ❌

Only architecture (AR=32) and matrix LR are shared. Every other parameter is substantially different. This proves that FineWeb-Edu's optimization landscape has multiple near-equivalent local optima — different combinations of warmdown schedule, momentum, and learning rate ratios can achieve the same performance.

This contrasts with ClimbMix, where the optimization landscape is relatively flat (general text datasets converge to similar configs regardless of starting point).

Full Experiment Log

All 100 experiments (click to expand)
Exp Description val_bpb Steps MFU Status
exp0 baseline (no modifications) 1.3710 438 14.7 baseline
exp1 WARMDOWN_RATIO 0.63→0.70 1.3799 415 13.9 discard
exp2 MATRIX_LR 0.05→0.07 1.3725 446 15.0 discard
exp3 DEVICE_BATCH_SIZE / 2 1.3675 449 15.1 keep
exp4 DEVICE_BATCH_SIZE / 4 1.3710 439 14.7 discard
exp5 MATRIX_LR 0.05→0.07 (w/ smaller batch) 1.3816 414 13.8 discard
exp6 SCALAR_LR 0.3→0.15 1.3790 408 13.7 discard
exp7 WEIGHT_DECAY 0.2→0.05 1.3807 415 13.9 discard
exp8 WARMDOWN_RATIO 0.63→0.45 1.3726 435 14.6 discard
exp9 EMBEDDING_LR 0.4→0.6 1.3712 440 14.8 discard
exp10 Increase DEPTH 1.4425 249 18.7 discard
exp11 WARMDOWN_RATIO 0.63→0.75 1.3645 466 15.7 keep
exp12 WARMDOWN_RATIO 0.75→0.85 1.3653 470 15.8 discard
exp13 ADAM β2 0.95→0.90 1.3680 466 15.7 discard
exp14 ADAM β1 0.71→0.85 1.3717 467 15.7 discard
exp15 FINAL_LR_FRAC 0.0→0.1 1.3628 459 15.4 keep
exp16 FINAL_LR_FRAC 0.1→0.05 1.3701 435 14.6 discard
exp17 ADAM β2 0.95→0.98 1.3730 425 14.2 discard
exp18 MATRIX_LR 0.05→0.06 1.3609 470 15.8 keep
exp19 EMBEDDING_LR 0.4→0.6 1.3643 458 15.4 discard
exp20 UNEMBEDDING_LR 0.004→0.008 1.3684 432 14.5 discard
exp21 ADAM β1 0.71→0.60 1.3735 428 14.3 discard
exp22 ADAM β1 0.71→0.80 1.3660 464 15.6 discard
exp23 WARMUP_RATIO 0.0→0.05 1.3858 429 14.4 discard
exp24 WEIGHT_DECAY 0.2→0.1 1.3780 410 13.7 discard
exp25 ADAM β1 0.71→0.75 1.3702 442 14.8 discard
exp26 SCALAR_LR 0.3→0.5 1.3753 428 14.3 discard
exp27 WEIGHT_DECAY 0.2→0.3 1.3733 430 14.4 discard
exp28 MATRIX_LR 0.06→0.07 1.3638 471 15.8 discard
exp29 HEAD_DIM 128→64 1.4147 366 12.2 discard
exp30 WINDOW_PATTERN change 1.3750 417 14.0 discard
exp31 ASPECT_RATIO → 24 1.3574 479 16.1 keep
exp32 ASPECT_RATIO → 32 1.3527 500 16.8 keep
exp33 ASPECT_RATIO → 40 1.3893 321 19.7 discard
exp34 WARMDOWN_RATIO 0.75→0.65 1.3553 487 16.4 discard
exp35 FINAL_LR_FRAC 0.1→0.15 1.3530 502 16.9 discard
exp36 FINAL_LR_FRAC 0.1→0.05 1.3844 391 13.1 discard
exp37 MATRIX_LR 0.06→0.05 1.4122 329 10.9 discard
exp38 ADAM β2 0.95→0.93 1.3974 360 12.0 discard
exp39 DEVICE_BATCH_SIZE / 3 — — — crash
exp40 EMBEDDING_LR 0.4→0.2 1.3893 385 12.9 discard
exp41 UNEMBEDDING_LR 0.004→0.04 1.4149 368 12.3 discard
exp43 WINDOW_PATTERN "SSSSL" 1.3842 392 13.1 discard
exp44 ADAM β1 0.71→0.65 1.3908 372 12.4 discard
exp45 WARMDOWN_RATIO 0.75→0.80 1.3532 499 16.8 discard
exp46 ADAM β2 0.95→0.97 1.3547 489 16.4 discard
exp47 ADAM β1 0.71→0.68 1.3527 502 16.9 keep
exp48 SCALAR_LR 0.3→0.4 1.3595 480 16.1 discard
exp49 ADAM β1 0.68→0.65 1.3595 475 16.0 discard
exp50 WEIGHT_DECAY 0.2→0.15 1.3551 485 16.3 discard
exp51 SCALAR_LR 0.3→0.25 1.3518 504 17.0 keep
exp52 MATRIX_LR 0.06→0.055 1.3508 506 17.0 keep
exp53 EMBEDDING_LR 0.4→0.3 1.3524 507 17.1 discard
exp54 WEIGHT_DECAY 0.2→0.25 1.3520 505 17.0 discard
exp55 ADAM β2 0.95→0.96 1.3504 506 17.0 keep
exp56 ADAM β2 0.96→0.97 1.3497 509 17.1 keep
exp57 ADAM β2 0.97→0.98 1.3478 521 17.5 keep
exp58 ADAM β2 0.98→0.99 1.3569 489 16.4 discard
exp59 WARMDOWN_RATIO 0.75→0.80 1.3569 484 16.3 discard
exp60 WARMDOWN_RATIO 0.75→0.70 1.3574 480 16.1 discard
exp61 EMBEDDING_LR 0.4→0.5 1.3538 511 17.2 discard
exp62 ADAM β1 0.68→0.70 1.3494 513 17.3 discard
exp63 ADAM β1 0.68→0.69 1.3494 513 17.2 discard
exp64 FINAL_LR_FRAC 0.1→0.05 1.3503 514 17.3 discard
exp65 FINAL_LR_FRAC 0.1→0.15 1.3500 515 17.3 discard
exp66 WEIGHT_DECAY 0.2→0.1 1.3494 515 17.3 discard
exp67 SCALAR_LR 0.25→0.20 1.3480 512 17.2 discard
exp68 SCALAR_LR 0.25→0.22 1.3493 509 17.1 discard
exp69 MATRIX_LR 0.055→0.050 1.3521 500 16.8 discard
exp70 EMBEDDING_LR 0.4→0.35 1.3584 478 16.1 discard
exp71 MATRIX_LR 0.055→0.058 1.3508 509 17.1 discard
exp72 UNEMBEDDING_LR 0.004→0.002 1.3739 513 17.3 discard
exp73 UNEMBEDDING_LR 0.004→0.006 1.3463 518 17.4 keep
exp74 UNEMBEDDING_LR 0.006→0.008 1.3491 512 17.2 discard
exp75 WEIGHT_DECAY 0.2→0.3 1.3488 514 17.3 discard
exp76 EMBEDDING_LR 0.4→0.45 1.3473 518 17.4 discard
exp77 SCALAR_LR 0.25→0.28 1.3489 515 17.3 discard
exp78 ADAM β2 0.98→0.975 1.3439 522 17.6 keep
exp79 WARMDOWN_RATIO 0.75→0.72 1.3508 493 16.6 discard
exp80 ADAM β2 0.975→0.972 1.3531 484 16.3 discard
exp81 ADAM β1 0.68→0.67 1.3467 512 17.2 discard
exp82 MATRIX_LR 0.053→0.057 1.3465 516 17.4 discard
exp83 WEIGHT_DECAY 0.2→0.15 1.3450 515 17.3 discard
exp84 MATRIX_LR 0.053→0.051 1.3443 517 17.4 discard
exp85 FINAL_LR_FRAC 0.1→0.12 1.3463 512 17.2 discard
exp86 EMBEDDING_LR 0.4→0.42 1.3459 512 17.2 discard
exp87 SCALAR_LR 0.25→0.23 1.3449 517 17.4 discard
exp88 ADAM β1 0.68→0.66 1.3454 515 17.3 discard
exp89 UNEMBEDDING_LR 0.006→0.007 1.3488 504 16.9 discard
exp90 ADAM β2 0.975→0.978 1.3511 494 16.6 discard
exp91 ADAM β2 0.975→0.971 1.3496 494 16.6 discard
exp92 WARMDOWN_RATIO 0.75→0.77 1.3438 522 17.6 keep
exp93 EMBEDDING_LR 0.4→0.45 1.3425 531 17.9 keep
exp94 SCALAR_LR 0.25→0.23 1.3417 532 17.9 keep
exp95 MATRIX_LR 0.053→0.050 1.3422 532 17.9 discard
exp96 MATRIX_LR 0.053→0.056 1.3429 532 17.9 discard
exp97 ADAM β1 0.68→0.69 1.3425 530 17.9 discard
exp98 ADAM β1 0.68→0.67 1.3416 533 17.9 keep
exp99 ADAM β1 0.67→0.66 1.3457 527 17.7 discard
exp100 FINAL_LR_FRAC 0.1→0.08 1.3482 511 17.2 discard

See also: Cross-LLM Comparison | FineWeb-Edu Sonnet 4.0 (Mar 17) | Cross-Dataset Comparison

Clone this wiki locally