Nanochat Speedrun (Modal)

Run Karpathy's nanochat speedrun on Modal with persistent storage for datasets, tokenizer artifacts, checkpoints, and compiled kernels.

Prerequisites

• Modal account + CLI configured:

pip install modal
modal setup

• H100 access on your Modal account.

Quick start

Smoke test (recommended first)

Runs 10 training steps on 1×H100 with evaluation disabled:

modal run speedrun-d12.py --task test

This seeds the volume with the tokenizer and a dataset shard, and validates the stack end-to-end. The full run skips re-downloading anything already on the volume.

Full speedrun

Runs the full speedrun on 8×H100:

modal run speedrun-d12.py

Train a different model size:

modal run speedrun-d12.py --model d32

Wipe a run and start fresh:

modal run speedrun-d12.py --force-restart

Model sizes

Flag	Params	Approx cost	Notes
--model d12	~100M	~$100	Default speedrun target
--model d26	~300M	~$300	Slightly surpasses GPT-2
--model d32	~560M	~$1,000+	Materially better reasoning

How it works

The Modal image pre-bakes the nanochat repo, Python 3.11 venv (via uv), and NVIDIA CUDA wheels at build time. LD_LIBRARY_PATH is set so PyTorch finds CUDA/NCCL/etc at runtime.

All data paths are controlled via NANOCHAT_BASE_DIR pointing to the persistent volume:

/vol
├── base_data/          # FineWeb-Edu parquet shards
├── tokenizer/          # tokenizer.pkl + token_bytes.pt
├── eval_bundle/        # ARC, MMLU, GSM8K, HumanEval benchmark data
├── runs/               # checkpoints + logs per run
└── cache/
    ├── torch_cache/    # torch.compile / Inductor kernels
    └── triton_cache/   # Triton autotuned kernels

CPU-only setup functions (ensure_tokenizer_on_volume, download_dataset) run before any GPU job so you don't pay for GPU time during data download.

On the first full run, torch.compile / Inductor may be quiet for several minutes while generating Triton kernels. A [HEARTBEAT] line prints every ~60s to confirm the process is alive. Subsequent runs load compiled kernels from the volume cache and start training immediately.

Checkpoints save periodically so a preempted run can resume without losing significant progress.

Inspecting outputs

Runs, checkpoints, and logs are saved to /vol/runs/<model>/. After a full run, report.md is written there with benchmark scores.

modal volume ls nanochat-persistent-storage /runs
modal volume get nanochat-persistent-storage /runs/d12 ./local_runs/d12

Cost estimation

Task	Estimated cost
Smoke test	< $2 (30-min timeout)
Full run (d12)	~$100 (~4 hours × 8×H100 @ ~$3/hr spot)
Full run (d26)	~$300 (~12 hours)
Full run (d32+)	~$1,000+

Troubleshooting

• "No dataset parquet files found" — Run the smoke test again (it calls download_dataset) and confirm the volume is mounted at /vol.

• "FileNotFoundError: /vol/tokenizer/tokenizer.pkl" — Run the smoke test again (it calls ensure_tokenizer_on_volume), or manually place tokenizer.pkl and token_bytes.pt under /vol/tokenizer.

• Wrong Python version in venv — Ensure UV_PYTHON=/usr/local/bin/python3.11 is set (already configured in speedrun-d12.py).

License

This runner script is provided as-is. nanochat itself is under its own license in the upstream repository.