supa beginna nanochat — walkthrough zero to tiny yap machine

[saranormous]

supa beginna nanochat

.---------------------------------------------------------------.
  |                   SUPA BEGINNA NANOCHAT                      |
  |                 zero to tiny yap machine                     |
  '---------------------------------------------------------------'
      text shards -> tokenizer -> base model -> chat model -> web ui

I wrote this while working through nanochat for the first time. The idea is to run every stage locally on a laptop before touching a GPU — so by the time you rent a box, you already know what each step does. The cloud section at the end is just the same thing on bigger hardware.

We'll start with a quick smoke test: ~10-15 min on a laptop, CPU only. After that there's a scaling section where you can throw progressively more compute at it and watch it get less dumb. The full runs/speedrun.sh on 8xH100 is ~2-3 hr and gives you actual GPT-2-grade results.

Tested against nanochat commit f068604 on macOS/Apple Silicon. If you hit NaN during SFT, jump to the troubleshooting section — there are two bugs in the bestfit packer that I spent a while tracking down.

How the pieces fit together

flowchart LR
    A["Text shards"] --> B["Tokenizer"]
    B --> C["Base model"]
    C --> D["Chat model (SFT)"]
    D --> E["CLI / Web UI"]

Loading

Each stage's output feeds the next. The tokenizer turns text into token IDs and back. Pretraining teaches the model to predict the next token on raw text. SFT (supervised fine-tuning) teaches it chat behavior from example conversations. Then you talk to it. Code stays in the repo, but runtime artifacts go into ~/.cache/nanochat by default (override with NANOCHAT_BASE_DIR):

NANOCHAT_BASE_DIR/
├── base_data_climbmix/    # downloaded parquet shards
├── tokenizer/             # tokenizer.pkl + token_bytes.pt
├── base_checkpoints/      # pretrained models
├── chatsft_checkpoints/   # chat models
├── eval_bundle/           # benchmark assets
└── report/                # markdown sections

Quick smoke test

End-to-end pipeline check on CPU. We'll use a temp directory so nothing touches the main cache.

Step 0: Set up the environment

Run:

# from the directory that contains your nanochat clone
cd nanochat
uv venv
uv sync --extra cpu
source .venv/bin/activate
export NANOCHAT_BASE_DIR=/tmp/nanochat-tutorial-smoke
mkdir -p "$NANOCHAT_BASE_DIR"

Success: uv sync finishes without errors and /tmp/nanochat-tutorial-smoke exists.

Step 1: Download training data

Run:

python -m nanochat.dataset -n 1

-n 1 grabs one shard.

Success: files land in $NANOCHAT_BASE_DIR/base_data_climbmix.

Step 2: Train a tiny tokenizer

Run:

python -m scripts.tok_train --max-chars=2000000 --doc-cap=2000 --vocab-size=2048
python -m scripts.tok_eval

Smaller than the default — optimizing for speed, not quality.

Success: tokenizer.pkl and token_bytes.pt appear in $NANOCHAT_BASE_DIR/tokenizer.

Step 3: Train a tiny base model

Run:

python -m scripts.base_train \
  --depth=2 \
  --aspect-ratio=32 \
  --head-dim=32 \
  --window-pattern=L \
  --max-seq-len=128 \
  --device-batch-size=1 \
  --total-batch-size=128 \
  --eval-every=-1 \
  --core-metric-every=-1 \
  --sample-every=-1 \
  --save-every=-1 \
  --num-iterations=2 \
  --run=dummy \
  --model-tag=smoke \
  --device-type=cpu

Two training steps. The first step is slow (setup/compile overhead).

Success: model_000002.pt appears in $NANOCHAT_BASE_DIR/base_checkpoints/smoke.

Step 4: Evaluate the base model

Run:

python -m scripts.base_eval \
  --eval bpb,sample \
  --model-tag smoke \
  --device-batch-size 1 \
  --split-tokens 512 \
  --device-type cpu

bpb (bits per byte — tokenizer-invariant loss metric) plus sample is enough to confirm loading and inference work.

Success: BPB numbers and gibberish samples. Two training steps won't produce language.

Step 5: Download the identity file for SFT

Run:

curl -L -o "$NANOCHAT_BASE_DIR/identity_conversations.jsonl" \
  https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl

Success: identity_conversations.jsonl appears in $NANOCHAT_BASE_DIR.

Step 6: Run one-step SFT

Run:

python -m scripts.chat_sft \
  --model-tag smoke \
  --max-seq-len 128 \
  --device-batch-size 1 \
  --total-batch-size 128 \
  --eval-every -1 \
  --chatcore-every -1 \
  --num-iterations 1 \
  --mmlu-epochs 0 \
  --gsm8k-epochs 0 \
  --load-optimizer 0 \
  --run dummy \
  --device-type cpu

Converts the base model into a chat model. First run downloads HuggingFace datasets, so it may stall briefly.

Success: model_000000.pt appears in $NANOCHAT_BASE_DIR/chatsft_checkpoints/smoke.

Step 7: Talk to it

Run:

python -m scripts.chat_cli --model-tag smoke --device-type cpu -p "Hello"

Success: chat_cli prints a reply. It will be nonsense — the model has barely trained. The chat path working is the point.

Step 8: Web server

Run (terminal 1):

python -m scripts.chat_web --model-tag smoke --device-type cpu --port 8001

Run (terminal 2):

curl http://127.0.0.1:8001/health

Success: curl returns {"status":"ok","ready":true,...}. Open http://127.0.0.1:8001/ for the UI.

Troubleshooting

uv sync fails — confirm you're in the repo root with uv installed. If the venv is half-created, rm -rf .venv && uv venv && uv sync --extra cpu.

First base-train step is slow — setup/compile overhead. Subsequent steps are faster.

chat_sft stalls — first run downloads HuggingFace datasets. Check disk/CPU activity.

chat_sft NaN at step 3-4 — this is a real bug in the bestfit packer. I tried every optimizer, LR, and dtype before finding it.

The root cause: F.cross_entropy(..., ignore_index=-1, reduction='mean') returns NaN when all targets are -1 (the padding marker), because it computes 0/0. The bestfit packer in chat_sft.py buffers 100 conversations and packs them into rows of max_seq_len + 1 tokens. With --max-seq-len=512, that's 513 tokens per row. 55.6% of identity_conversations.jsonl entries exceed 513 tokens. Once the buffer fills with conversations that can't fit, every row is pure padding, every batch is pure padding, and training produces NaN indefinitely.

Two fixes: (1) truncate conversations to row_capacity via max_tokens=row_capacity in render_conversation inside refill_buffer, and (2) skip all-padding batches with if (y == -1).all() before the forward pass. Only manifests at --max-seq-len=512; the smoke test never hits it. (Not yet PR'd — apply locally if needed.)

chat_cli can't find a model — check $NANOCHAT_BASE_DIR/chatsft_checkpoints/smoke exists and NANOCHAT_BASE_DIR is set.

What should exist on disk after the smoke test:

• $NANOCHAT_BASE_DIR/tokenizer/
• $NANOCHAT_BASE_DIR/base_checkpoints/smoke/
• $NANOCHAT_BASE_DIR/chatsft_checkpoints/smoke/

Notes on each stage

Tokenizer (nanochat/tokenizer.py). One object handles BPE vocabulary training and chat rendering (<|user_start|> / <|assistant_end|> markers). Change the tokenizer, retrain everything.

Base training (scripts/base_train.py, nanochat/gpt.py). --depth is the only size knob — embedding dim, head count, and MLP width are derived from it. In gpt.py, weights live in one dtype but the forward pass casts on the fly, so the same code runs on CPU (float32), MPS (float32), and CUDA (FP8/bfloat16) without branching.

Eval (scripts/base_eval.py). bpb gives you a number, core runs benchmarks, sample shows raw text. Locally, bpb + sample is enough.

SFT (scripts/chat_sft.py). Data switches from raw text to rendered conversations — SmolTalk, identity JSONL, MMLU, GSM8K, spelling tasks. Most hyperparameters inherit from the pretrained checkpoint. The bestfit packer is where the complexity hides, and where the NaN bug lives (see troubleshooting).

Inference (nanochat/engine.py). Prefill processes the prompt at once, then each new token is cheap via KV cache. Special tokens let the model call a sandboxed Python interpreter. The web server (scripts/chat_web.py) is ~200 lines of FastAPI.

Scaling

The smoke test model is braindead. Here's what happens as you throw compute at it:

Config	Params	Train steps	Base BPB	Time (M4 Max)	What it sounds like
smoke (depth=2)	491K	2	3.73	seconds	Random token fragments
depth=4, 1000 iter	36.7M	1000	1.77	~45 min CPU	Grammatical English, zero facts
depth=4, 3000 iter	36.7M	3000	1.56	~2.5 hr CPU	Topical but repetitive
depth=6, 5000 base + 1500 SFT	73M	5000+1500	1.17 (0.72 after SFT)	~55 min MPS	Knows Paris is France's capital. First working chat model.
speedrun.sh (depth=24)	~1.4B	6612+483	~2.3	~2.5 hr on 8xH100	Actual GPT-2-grade conversation

At depth=4 it speaks English. At depth=6 it knows Paris exists. The diff is just 30 minutes of MPS time!!

Setup

Unset the smoke test directory and grab more data:

unset NANOCHAT_BASE_DIR   # scaling runs use the default ~/.cache/nanochat
python -m nanochat.dataset -n 8
python -m scripts.tok_train

All configs below use --window-pattern=L --head-dim=64. If you run out of RAM, halve --device-batch-size. Times are from an M4 Max MacBook Pro. The depth=4 runs use --device-type=cpu; depth=6 switches to --device-type=mps (Apple Silicon GPU).

Python buffers stdout when piped. Use PYTHONUNBUFFERED=1 and tee to a log file:

PYTHONUNBUFFERED=1 python -m scripts.base_train ... 2>&1 | tee train.log

depth=4: the shape of language without the content

Run:

python -m scripts.base_train \
  --depth=4 --head-dim=64 --window-pattern=L \
  --max-seq-len=512 --device-batch-size=4 --total-batch-size=16384 \
  --eval-every=200 --eval-tokens=524288 \
  --core-metric-every=-1 --sample-every=-1 --save-every=500 \
  --num-iterations=1000 --run=d4-1k \
  --device-type=cpu

At depth=4 (36.7M params), the model learns English grammar but no facts. At 1000 iterations (~45 min CPU, BPB 1.77), it produces things like:

"The capital of France is" → a great way to make a sense of life. The most common way of the
"The planets of the solar system are:" → The energy of the energy of the energy of the energy

Success: checkpoint in ~/.cache/nanochat/base_checkpoints/d4-1k/.

Triple the training to 3000 iterations (--num-iterations=3000, ~2.5 hr, BPB 1.56) and it starts feeling topical — "The solar system is a solar system that is a solar system" is wrong, but it's reaching for concepts in the zone. Same architecture, more compression. Goooo lil model.

SFT diverges at depth=4 with --max-seq-len=512 — see the troubleshooting section for why.

depth=6 on MPS

Same architecture as runs/runcpu.sh. On Apple Silicon, MPS is a massive speedup — hours on CPU becomes 43 minutes on MPS.

Run:

# Base training: depth=6, 5000 iterations (Karpathy's runcpu.sh recipe)
PYTHONUNBUFFERED=1 python -m scripts.base_train \
  --depth=6 --head-dim=64 --window-pattern=L \
  --max-seq-len=512 --device-batch-size=32 --total-batch-size=16384 \
  --eval-every=200 --eval-tokens=524288 \
  --core-metric-every=-1 --sample-every=-1 --save-every=1000 \
  --num-iterations=5000 --run=d6-base \
  --device-type=mps 2>&1 | tee base_train.log

73M params, 43 min on MPS (M4 Max), final val_bpb: 1.17. On CPU the same run would take several hours.

Run:

# SFT: 1500 iterations (picks up the latest base checkpoint automatically)
PYTHONUNBUFFERED=1 python -m scripts.chat_sft \
  --model-tag=d6-base \
  --max-seq-len=512 --device-batch-size=32 --total-batch-size=16384 \
  --eval-every=200 --eval-tokens=524288 \
  --chatcore-every=-1 \
  --num-iterations=1500 --run=d6-sft \
  --device-type=mps 2>&1 | tee sft_train.log

SFT: ~12 min on MPS. SFT val_bpb starts at 0.97 (lower than the base 1.17 because SFT data is easier to predict than raw web text) and drops to 0.72. Loss starts around 3.0 and settles near 2.0.

Run:

python -m scripts.chat_cli --model-tag d6-sft

Device type auto-detects (CUDA > MPS > CPU), so no flag needed.

User: What is the capital of France?
Assistant: The capital of France is Paris. Paris is a city known for its
beautiful landmarks, cultural institutions, and iconic landmarks. It is a
beautiful city known for its historical landmarks, cultural experiences,
and cultural attractions. Paris is the largest city in Paris, where its
vibrant colors and cultural patterns are often associated with its cultural
and social norms. [continues repeating...]

Knows Paris. Gets stuck in a repetition loop after two sentences, but the core fact is there and it responds as an assistant, not a text-completion engine.

No MPS? Run bash runs/runcpu.sh (several hours) or jump to a GPU below.

File map

scripts/    what the repo does (train, eval, chat, serve)
nanochat/   the reusable machinery (model, tokenizer, engine, dataloader)
tasks/      chat/task datasets (smoltalk, mmlu, gsm8k, etc.)
runs/       opinionated recipes (speedrun.sh, runcpu.sh)

Key files: runs/speedrun.sh (reference pipeline), scripts/base_train.py (pretraining), nanochat/gpt.py (model), scripts/chat_sft.py (SFT), nanochat/engine.py (inference).

8×H100 speedrun

Same pipeline on 8×H100 GPUs via bash runs/speedrun.sh. 2 hours 40 minutes, ~$64.

depth=24 (~1.38B params), FP8 precision, Flash Attention, 6,612 pretraining steps, 483 SFT steps.

The loss curves

Pretraining starts at 10.4 (random weights guessing uniformly over the vocabulary) and drops to 2.32 over 6,612 steps. SFT starts at 1.3 — the model already knows language, it just needs to learn the chat format — and drops to 0.79 in 483 steps.

The zoomed view shows the power-law descent after warmup. Notice how the loss barely moves after step 4000. You could stop there and save 40% of the compute. But the SFT loss starts lower when you don't — and that matters for final quality.

GPU utilization

Pretraining MFU (model FLOPS utilization — what fraction of the GPU's theoretical math throughput you're actually using): 59.4%, near the practical max for H100s. SFT drops to 50.3% because chat conversations are shorter and more variable than uniform 2048-token text chunks, so the GPUs spend more time waiting on padding.

~970K tokens/sec, steady. The dips are eval checkpoints — the GPUs pause to run benchmarks.

What the model learned

After pretraining (CORE: 0.269) it can complete text coherently:

> The capital of France is Paris. It is the largest city in France...
> The chemical symbol of gold is Au. Gold is a chemical element...
> The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and

But it can't chat — it just continues text. SFT fixes that.

After SFT (ChatCORE: 0.375):

Benchmark	Score
ARC-Easy	64.8%
ARC-Challenge	51.9%
MMLU	37.6%
GSM8K	9.6%
HumanEval	11.0%
SpellingBee	98.8%

Factual questions and basic reasoning work. Math and code need more parameters and data.

Reproducing this

Run:

# On an 8×H100 machine:
git clone https://github.com/karpathy/nanochat.git
cd nanochat
bash runs/speedrun.sh

Lambda GPU VM

A GPU VM just makes it faster. The repo wants one Linux machine with local disk — no separate training platform needed.

Two shapes: a single H100/A100, or 8xH100 for runs/speedrun.sh. Ubuntu, enough disk, SSH key, screen/tmux.

Getting on the VM

Run:

ssh -i "<SSH-KEY-FILE-PATH>" ubuntu@<INSTANCE-IP>
git clone https://github.com/karpathy/nanochat.git
cd nanochat
screen -L -Logfile runs/setup.log -S nanochat

Optionally wandb login — if you skip it, the scripts fall back to dummy logging.

Full speedrun (8xH100)

Run:

WANDB_RUN=speedrun bash runs/speedrun.sh

Handles everything: env setup, data download, tokenizer, pretraining, eval, SFT, chat eval, report.md.

Single GPU

Same pipeline, smaller model. torchrun is PyTorch's distributed launcher — even with one GPU, nanochat expects it because the training scripts use distributed primitives internally. The -- separates torchrun args from script args.

Run:

uv venv
source .venv/bin/activate
uv sync --extra gpu
python -m nanochat.dataset -n 8
python -m scripts.tok_train
python -m scripts.tok_eval

Success: tokenizer.pkl in ~/.cache/nanochat/tokenizer/.

Run:

torchrun --standalone --nproc_per_node=1 -m scripts.base_train -- \
  --depth=12 --device-batch-size=4 --run=lambda-smaller

depth=12 is ~84M params. The script uses gradient accumulation automatically to maintain the effective batch size. If you OOM, halve --device-batch-size or drop to --depth=6.

Success: checkpoints in ~/.cache/nanochat/base_checkpoints/lambda-smaller/.

Run:

torchrun --standalone --nproc_per_node=1 -m scripts.base_eval -- \
  --eval bpb,sample --device-batch-size=4 --run=lambda-smaller

Success: BPB numbers and text samples.

Run:

torchrun --standalone --nproc_per_node=1 -m scripts.chat_sft -- \
  --device-batch-size=4 --run=lambda-smaller

Success: chat checkpoint in ~/.cache/nanochat/chatsft_checkpoints/lambda-smaller/.

After training

Run:

source .venv/bin/activate
python -m scripts.chat_web --run lambda-smaller

Use the machine's public IP (not localhost), port must be open. You should see base_data_climbmix/, tokenizer/, base_checkpoints/, chatsft_checkpoints/, and report/ under ~/.cache/nanochat.

Overnight

Verify the first stage started, know how to reattach (screen -r nanochat), know where checkpoints go. Stop the VM when done — GPUs don't care that you're sleeping.

Identity infusion

Once you have a working chat model, you can SFT it further on your own data to give it a specific voice. The nanochat identity guide covers the mechanics. I ran five rounds of this with my own writing and the short version is: 160 conversations grounded in real long-form writing outperformed 12,787 scraped tweets. Structure and reasoning matter more than volume.

One practical note: use --load-optimizer 0 when switching datasets during SFT. Without it, stale Adam momentum causes NaN in the first few steps.

Code agents

I used Claude Code throughout this project and the most useful pattern was pointing it at a failure and letting it trace the code. The NaN bug is a good example: SFT was dying at step 3-4 regardless of optimizer, learning rate, or dtype. I'd tried every knob. I pointed Claude Code at the bestfit packer in chat_sft.py and asked it to trace what happens when conversations are longer than max_seq_len. It found the buffer deadlock — 55.6% of conversations exceed 513 tokens, the 100-entry buffer fills with unfittable conversations, and every subsequent batch is pure padding. F.cross_entropy on all-padding targets computes 0/0 → NaN. Two bugs, one root cause, and I'd been blaming the optimizer for hours.

The general pattern: run a stage, hit something unexpected, and ask a code agent to trace the data flow rather than guessing at flags. "What happens to a 600-token conversation in the bestfit packer when max_seq_len=512?" is a better prompt than "why is my training NaN."

[svlandeg]

Great write up! FYI, I think the nan bug should be fixed by #486.

[Seqaeon]

Hello, is there a table for the minimum val_bpp at each depths ? is it available?

[christophergeyer]

Thank you! Appreciated this walkthrough.

I had Claude Code follow this discussion and run the full pipeline end-to-end on a RunPod pod with 2×H100 PCIe GPUs: data download → tokenizer → base pretrain (d24, 100 iters) → SFT → chat eval.

The whole run came in around $20 and 3.5 hours.

To keep the run inexpensive, I capped pretraining at 100 iterations, so the reported val_bpb is an early checkpoint rather than a fully trained result.

To answer the question above: base val_bpb at d24/100 iterations landed at 1.334, and ChatCORE reached 0.226 after SFT.

I also wrapped each stage with roar, a lineage tracer I've been working on. It automatically captured the dependency graph from training shards → tokenizer → base checkpoint → SFT model without modifying any of the nanochat code.

Full session DAG:
https://glaas.ai/dag/965b9b5e9476e77f26625e7f050dfd3ad115af4419352bea8923c7f117e7856e

Happy to share the full metrics table if useful.