Skip to content

Cross Dataset Comparison

Jump to bottom
Dave Graham edited this page Mar 20, 2026 · 11 revisions

Cross-Dataset Comparison — Deep Dive

Autonomous agent runs on the same hardware (Apple M5 Max, 64 GB) reveal how optimal configurations diverge across training datasets. Three datasets have been tested: climbmix (raw web crawl), FineWeb-Edu (educationally scored web text), and early results from cosmopedia-v2 (synthetic textbooks).

Cross-Dataset Comparison

Results Summary

Climbmix (Mar 19) FineWeb-Edu (Mar 17)
Best val_bpb 1.296 1.342
Baseline 1.297 (pre-optimized) 1.409 (defaults)
Improvement −0.08% −4.7%
Experiments 101 88
Keep rate 1.0% (1/101) 19.3% (17/88)
Crash rate 10.9% 2.3%

Note on baselines: The climbmix run started from a pre-optimized baseline (AR=32, MATRIX_LR=0.0435, EMBEDDING_LR=0.4, SCALAR_LR=0.4 — inherited from prior branch runs), while FineWeb-Edu started from full defaults. The low improvement rate for climbmix reflects that the configuration was already near-optimal, not that climbmix is easier to optimize.

Architecture Convergence

A key finding from the Mar 19 run: both datasets converge to AR=32. The climbmix branch accumulated AR=32 from prior optimization passes, and the Mar 19 run confirmed it couldn't improve from there. FineWeb-Edu independently discovered AR=32 as its biggest single improvement (exp34: −0.036 val_bpb).

Parameter Default Climbmix Optimal FineWeb-Edu Optimal
ASPECT_RATIO 64 32 32
Peak Memory 26.1 GB 15.5 GB 15.5 GB
tok/sec ~30K ~58K ~58K
Training Steps ~288 ~530 ~530

Both datasets benefit from a half-width model running 2× more gradient steps in the 5-minute budget. The AR=32 architecture appears to be a hardware-optimal configuration for the M5 Max at this model scale, regardless of data source.

Why AR=32 Works

The autoresearch training loop is time-budgeted (5 minutes). At AR=32, the model drops from ~50M to ~25M parameters, cutting memory from 26 GB to 15.5 GB and roughly doubling throughput. The extra gradient steps more than compensate for reduced per-step capacity. This tradeoff is hardware-specific — a GPU with more compute (e.g., H100) might favor wider models.

Hyperparameter Divergence

While architecture converged, optimization hyperparameters diverged significantly:

Parameter Default Climbmix FineWeb-Edu Divergence
MATRIX_LR 0.04 0.0435 (1.09×) 0.053 (1.33×) Both ↑, FineWeb more aggressive
SCALAR_LR 0.5 0.4 (0.80×) 0.3 (0.60×) Both ↓, FineWeb more aggressive
EMBEDDING_LR 0.6 0.4 (0.67×) 0.59 (0.98×) Opposite magnitudes
UNEMBEDDING_LR 0.004 0.0041 (1.03×) 0.003 (0.75×) Opposite directions
WEIGHT_DECAY 0.2 0.2 (1.00×) 0.12 (0.60×) Climbmix keeps default, FineWeb ↓
WARMDOWN_RATIO 0.5 0.5 (1.00×) 0.47 (0.94×) Climbmix keeps default, FineWeb ↓

Key Observations

EMBEDDING_LR is the biggest divergence: climbmix dropped it to 0.4 (33% reduction) while FineWeb-Edu barely touched it (0.59). This was climbmix's most impactful optimization — the agent found three consecutive keeps by pushing EMBEDDING_LR from 0.6 → 0.5 → 0.45 → 0.4 during prior runs on the branch. Educational text apparently needs near-default embedding learning rates, while web crawl benefits from more conservative embedding updates.

UNEMBEDDING_LR moves in opposite directions — up to 0.0041 on climbmix, down to 0.003 on FineWeb-Edu. The unembedding layer maps hidden states to token predictions. Climbmix's diverse vocabulary benefits from a slightly faster-learning output layer, while FineWeb-Edu's more structured text prefers a more conservative one.

WEIGHT_DECAY and WARMDOWN_RATIO stayed at defaults for climbmix but were reduced for FineWeb-Edu. The climbmix agent tested both extensively (8 weight decay experiments, 5 warmdown experiments) and rejected all changes — the defaults are truly optimal for this data. FineWeb-Edu's more regular structure benefits from slightly less regularization and a shorter warmdown.

Improvement Dynamics

Metric Climbmix (Mar 19) FineWeb-Edu (Mar 17)
Starting point Pre-optimized (AR=32) Full defaults
Total improvement −0.0011 (0.08%) −0.066 (4.7%)
Keep rate 1.0% 19.3%
Biggest single gain UNEMBEDDING_LR (+0.0001) AR 40→32 (−0.036)
Optimization character Exhausted — near plateau Steady descent with architecture breakthrough
Crash rate 10.9% (batch sizes) 2.3%

Climbmix: Near-Optimal Plateau

The Mar 19 climbmix run is the most definitive "nothing to find" result in the project. Across 101 experiments testing every parameter category — architecture, learning rates, regularization, Adam betas, batch sizes — only UNEMBEDDING_LR 0.004→0.0041 produced a measurable improvement (0.0011 val_bpb). The agent spent its entire budget confirming the configuration was already optimal.

The 10.9% crash rate (all from batch size changes) suggests the MLX backend has strict alignment requirements for batch dimensions that the agent hasn't learned to avoid.

FineWeb-Edu: Structured Descent

FineWeb-Edu's 17 kept experiments show a clear three-phase optimization:

  1. Phase 1 — Optimization tuning (exp11–14): SCALAR_LR, MATRIX_LR, UNEMBEDDING_LR, WARMDOWN adjustments
  2. Phase 2 — Architecture discovery (exp32–34): The critical AR 64→48→40→32 reduction
  3. Phase 3 — Fine-tuning (exp45–75): Weight decay reduction, embedding LR refinement, warmdown adjustment

Best Configurations Side-by-Side

Climbmix (Mar 19, 101 experiments)

ASPECT_RATIO = 32          # ↓ from 64 (prior runs)
MATRIX_LR = 0.0435         # ↑ slightly from 0.04
EMBEDDING_LR = 0.4         # ↓ significantly from 0.6
UNEMBEDDING_LR = 0.0041    # ↑ slightly from 0.004
SCALAR_LR = 0.4            # ↓ from 0.5
WEIGHT_DECAY = 0.2         # Default
WARMDOWN_RATIO = 0.5       # Default
# Peak memory: 15.5 GB | Steps: 530 | MFU: 17.9%

FineWeb-Edu (Mar 17, 88 experiments)

ASPECT_RATIO = 32          # ↓ from 64
MATRIX_LR = 0.053          # ↑ from 0.04
EMBEDDING_LR = 0.59        # ↓ slightly from 0.6
UNEMBEDDING_LR = 0.003     # ↓ from 0.004
SCALAR_LR = 0.3            # ↓ from 0.5
WEIGHT_DECAY = 0.12        # ↓ from 0.2
WARMDOWN_RATIO = 0.47      # ↓ slightly from 0.5
# Peak memory: 15.5 GB | Steps: ~530 | MFU: ~18%

Implications

  1. AR=32 is hardware-optimal on M5 Max. Both datasets independently converge to the same architecture. The 2× step throughput at half memory outweighs the reduced model capacity. This finding is likely hardware-specific and would not transfer to GPUs with different compute/memory ratios.

  2. Optimization hyperparameters are dataset-dependent. EMBEDDING_LR shows a 1.5× difference (0.4 vs 0.59), UNEMBEDDING_LR moves in opposite directions, and weight decay diverges (default vs −40%). Running one dataset's optimal config on another would leave performance on the table.

  3. Pre-optimized baselines are hard to improve. The climbmix run's 1% keep rate (vs FineWeb-Edu's 19%) confirms that once a configuration is near-optimal, exhaustive search yields diminishing returns. The 101 experiments mostly served to validate optimality rather than find improvements.

  4. Crash rate correlates with batch size exploration. Climbmix's 10.9% crash rate (vs FineWeb-Edu's 2.3%) comes entirely from DEVICE_BATCH_SIZE experiments. The agent could benefit from learning which batch sizes are hardware-compatible.

See individual run pages for full experiment logs: Climbmix (Mar 19) | FineWeb-Edu (Mar 17) | Climbmix (Mar 16, legacy)

Clone this wiki locally