Home
Apple Silicon port of karpathy/autoresearch with autonomous LLM-driven experiment optimization across multiple training datasets.
- TUI Dashboard — Real-time terminal dashboard for monitoring training runs
- Multi-Dataset Suite — Framework for running experiments across different training datasets
Autonomous agent runs on the same hardware (Apple M5 Max, 64 GB) reveal that optimal model configuration is dataset-dependent. The agent independently discovers different architectures, learning rates, and regularization strategies for each dataset.
| Climbmix (web crawl) | FineWeb-Edu (educational) | |
|---|---|---|
| Best val_bpb | 1.335 | 1.295 |
| Baseline | 1.353 | 1.409 |
| Improvement | -0.018 (1.3%) | -0.114 (8.1%) |
| Experiments | 81 | 101 |
| Keep rate | 11.1% | 18.8% |
| Crash rate | 3.7% | 2.0% |
The most striking finding: the same agent on the same hardware converges to completely different model architectures depending on the training data.
| Parameter | Default | Climbmix Optimal | FineWeb-Edu Optimal | Direction |
|---|---|---|---|---|
| ASPECT_RATIO | 64 | 64 (unchanged) | 32 (halved) | ← |
| Peak Memory | 26.1 GB | 26.1 GB | 15.5 GB | ← |
| tok/sec | ~30K | ~31K | ~62K | → |
| Training Steps | ~288 | ~289 | ~572 | → |
FineWeb-Edu benefits from a smaller model that runs 2× more gradient steps in the same 5-minute budget. Climbmix's diverse web crawl text needs the full model width.
Every tunable hyperparameter diverged between datasets, with some moving in opposite directions:
| Parameter | Default | Climbmix | FineWeb-Edu | Divergence |
|---|---|---|---|---|
| MATRIX_LR | 0.04 | 0.06 (1.50×) | 0.053 (1.33×) | Both ↑, different magnitude |
| SCALAR_LR | 0.5 | 0.5 (1.00×) | 0.3 (0.60×) | Climbmix keeps default, FineWeb ↓ |
| WEIGHT_DECAY | 0.2 | 0.02 (0.10×) | 0.12 (0.60×) | Both ↓, 6× difference |
| WARMDOWN_RATIO | 0.5 | 0.4 (0.80×) | 0.47 (0.94×) | Both ↓, different degree |
| UNEMBEDDING_LR | 0.004 | 0.0041 (1.03×) | 0.0028 (0.70×) | Opposite directions |
| EMBEDDING_LR | 0.6 | 0.585 (0.98×) | 0.59 (0.98×) | Both slightly ↓ |
Key insight: WEIGHT_DECAY shows a 6× difference (0.02 vs 0.12), and UNEMBEDDING_LR moves in opposite directions. These are not transferable across datasets.
| Metric | Climbmix | FineWeb-Edu |
|---|---|---|
| Biggest single gain | MATRIX_LR 0.04→0.06 (-0.013) | AR 40→32 (-0.036) |
| Late breakthrough? | No (diminishing returns) | Yes — exp89 broke a 20-experiment plateau |
| Key lever | Learning rate tuning | Architecture change + late LR discovery |
| Optimization character | Smooth, incremental | Two sharp phase transitions |
| Date | Chip | Dataset | Experiments | Best val_bpb | Branch |
|---|---|---|---|---|---|
| Mar 17, 2026 — FineWeb-Edu (101 experiments) | Apple M5 Max (64 GB) | FineWeb-Edu 10BT | 101 | 1.295 | feature/multi-dataset |
| Mar 16, 2026 — Climbmix (81 experiments) | Apple M5 Max (64 GB) | climbmix-400b | 81 | 1.335 | autoresearch/mar16-agent |
| Date | Chip | Best val_bpb | Branch |
|---|---|---|---|
| Mar 15, 2026 — M5 Max | Apple M5 Max (64 GB) | 1.320 | autoresearch/mar14-m5max |
| Mar 14, 2026 — M4 Pro | Apple M4 Pro (24 GB) | 1.429 | autoresearch/mar14 |
| Mar 11, 2026 — M1 Max | Apple M1 Max (64 GB) | 1.621 | autoresearch/mar11 |
-
karpathy/autoresearch PR #303 — "Evaluating Experiment Results at Scale" by Dean Sharon. Guide for noise floor estimation, Pareto efficiency, and reading results.tsv at scale. Adapted for Apple Silicon in
docs/evaluating-results.md.