v1.1.3: What 2c and 4c would need

The notes of two side topics now say what their results would need to improve, and what the papers they follow measured.

• 2c, latent attention: DeepSeek-V2's ablations on MMLU (7B dense models: full multi-head 45.2%, grouped-query 41.2%, multi-query 37.9%; latent attention ahead of full multi-head in mixtures of experts of 16B and 250B, with 14% and 4% of its cache). Latent attention can only gain where sharing keys costs something measurable, which this laptop's model size does not reach. A fused kernel such as FlashMLA is the way out of its slow decoding of a single sequence.
• 4c, continuous batching: the story model's running batch would need a prefill cheaper than a step per request, batched or chunked as in Sarathi-Serve. Batch-invariant kernels (He, 2025) make greedy replies independent of the batch. The gains Orca (36.9 times FasterTransformer's throughput at the same latency), vLLM (2 to 4 times) and Sarathi-Serve (2.6 to 3.7 times vLLM's request rate) measured.

The films are unchanged and attached again, all 28.

v1.1.2: Continuous batching

A new side topic, 4c: continuous batching. Stage 4's batches start and end together: a row whose reply has ended waits for the longest one, and a new request waits for the whole batch. A running batch lets requests join and leave between any two steps, as every inference server does. Every row is at its own position, with its own rotary angles, its own mask over the cache and its own temperature and top_p. The server of stage 10 now answers requests that arrive together in one running batch.

• Qwen2.5-0.5B on 256 GSM8K problems. The replies are 294 tokens long on average and up to 512.
  • With 16 slots and every request at once, a running batch ends them all in 89.6 seconds, where batches that end together take 123.5.
  • With requests arriving at random, 4 a second, a request waits 7.6 seconds for its first token instead of 27.3.
  • With 64 slots, that wait is 0.07 seconds instead of 7.5.
• The 14M story model on 512 requests. Its replies are alike in length, and on this model a forward pass costs about the same for 1 row as for 64. The prefill of every request on its own then eats the gain. With 64 slots and every request at once, the running batch ends later, after 9.8 seconds instead of 8.2. Arriving at random, a request still gets its first token sooner: 0.01 seconds instead of 0.69.
• Greedy replies in bfloat16. 158 of Qwen's 256 greedy replies come out different somewhere when batched otherwise, because a sum's rounding depends on the shape of the batch. All 512 of the story model's come out the same. In 32-bit floats on the CPU, the test gets exactly the reply of each request alone.

New code: minllm/batching.py (Batcher, sample_rows), positions per row in Transformer.hidden, bench_batching.py, and serve.py --slots --context.

The films of every chapter are attached, 28 of them.

v1.1.1: Latent attention

A new side topic, 2c: latent attention. DeepSeek's multi-head latent attention caches a short latent per token in place of keys and values, projects every head's key and value up from it, and at decoding folds the up-projection into the query, so the cache is never projected up. A rotary key of its own carries the position. Stage 3's 14M model was trained with it and with three kinds of key/value heads, for the 6,000 steps of the reference run.

• Loss. Per token and layer, the cache holds 768 numbers with 6 key/value heads, 256 with stage 2's 2, 128 with 1, and 160 with a latent of 128 and a rotary key of 32. Their validation losses:
  • 6 key/value heads: 1.455
  • 2, stage 2's: 1.465, and 1.461 with another seed
  • 1 key/value head: 1.464
  • latent 128 + 32: 1.464
  • latent 64 + 32: 1.478, and latent 32 + 16: 1.490
• Speed alone. For one sequence, decoding against the latent writes 179 tokens per second, against 509 with stage 2's attention.
• Speed in a batch. With 1,024 sequences at once, it writes 43,800 tokens per second, 1.6 times as many as stage 2's attention. The smallest latent writes 68,300. Projected up at every step, the latent of 128 writes 12,800.
• Why. Stage 2's attention copies shared keys out to every head, so one key/value head is hardly faster than six. The latent path reads its cache once for all heads.

New code: LatentAttention in minllm/model.py, train.py --kv-latent --rope-dim, and bench_attention.py.

The films of every chapter are attached, 27 of them.

v1.1.0: DPO on math problems

A new side topic, 8b: DPO on math problems. Stage 8's DPO runs on the 0.5B model, and each pair is two of its own replies to a GSM8K training problem: a right one, preferred, and a wrong one, as judged by stage 9's verifier. The model is stage 6's fine-tuned one, which self-training (9a) and GRPO (stage 9) also start from.

• Pairs. 3,840 replies to 480 problems, 32.9% of them right, gave 551 pairs from the 331 problems that had both right and wrong replies.
• No setting gains. Five DPO runs were scored on the 1,319 test problems, against 36.6% at the start:
  • learning rate 2e-6: 35.7%
  • learning rate 1e-5: 35.2%
  • learning rate 5e-5, one pass: 34.6%
  • learning rate 5e-5, two passes: 29.2%
  • learning rate 5e-5 with the right reply's own loss added: 33.8%
• Both replies can fall. At learning rate 5e-5, after two passes, the wrong replies were 91 nats less likely than at the start, and even the right ones were 21.5 nats less likely. Adding the right reply's own loss keeps the right replies up.
• Against GRPO. GRPO sampled as many replies, but drew them as it trained, and reached 42.5%.

New code: minllm/dpo.py, make_preferences.py --task gsm8k, and analyze_dpo.py. dpo.py now carries the 0.5B model.

The films of every chapter are attached, 26 of them.

v1.0.0: the tutorial, complete

The tutorial is complete: ten stages from raw text to a served model, and sixteen side topics that hang off the stage they change. Every number and sample in the notes and films comes from a run saved in runs/, and every note's "Run it" works from a fresh clone, in chain order.

What changed since 0.5.11. The README names the laptop every number was measured on, a MacBook Pro with an M5 Pro and 64 GB of memory. It also says what to expect on another machine: other times, and other last digits, since the same sums are added in another order. Even on this laptop, a greedy reply in 16-bit floats depends on the batch it was generated in. Stage 7 measured the published 0.5B model at 48.6% and 46.6% on the same 500 GSM8K problems.

The films of every chapter are attached, 25 of them.

v0.5.11: the tutorial from a fresh clone

The tutorial was run end to end from a fresh clone, every note's "Run it" in chain order at reduced size. This release fixes what that run found.

Your results no longer overwrite the published ones. Scripts write to out/runs/, under the same names as in runs/, and the analyses read only your own results. runs/ stays the record the notes quote, and MINLLM_RUNS=runs writes to it. Before, a run of the tutorial overwrote the record. Analyses silently mixed a learner's results with the published ones. 03c's sweep skipped every run as done before.

Every "Run it" works from an empty checkout.

• Prerequisites stated: 2b says it comes after 3b. 4a, 4b, 7b and 9c start with the downloads and earlier outputs they need.
• Missing commands added: 7a, 8, 9, 9a and 10a list the evaluations their tables read, among them stage 9's two repair runs.
• Chain order: analysis scripts run in chain order, leaving out by name the models that later stages make.
• Script fixes: bench_moe.py runs with ten steps or fewer; long_context.py prints no empty bands for a short window.
• Smaller fixes:
  • a paired comparison of runs of different sizes says so;
  • small downloads show their size in KB;
  • 2a no longer lists one training command twice.

The README says where results go, which training scripts overwrite their models, and that the first test run downloads GSM8K.

The films of every chapter are attached, 25 of them.

v0.5.10: every quoted number from a saved run

Every number and sample the notes quote now comes from a file under runs/ that a script wrote.

One conclusion reversed. 7a said the model "does not know less when it writes". On the 828 questions whose written reply could be read, the letter is right 54.2% of the time and the written reply 48.3%: 5.9 points less (interval 2.5 to 9.2). Writing costs on the questions the model finishes too.

Reproduced exactly:

• the reward model's four epochs, in 4.9 minutes;
• the export and quantization checks of 2a, 10 and 10a;
• the reply lengths of stages 7 and 9 (920, 211 and 268 characters);
• the tokenizer's 15.7M chunks, 19,880 distinct ones, and 97% of words a single token;
• the words task's 436,157 examples;
• stage 3's bird story and stage 6's mole replies;
• the published model at 320 tokens, now written by the script itself, with the same scores on all 1,319 problems.

Changed to the new run's values:

• stage 3's 300-step check: 2.4977 and 2.4960;
• stage 3's benchmark over three more runs: the factor 3.77 to 3.85;
• the robot story, 3a's distilled story, and two of stage 4's three temperature samples;
• the memory pool of stage 6, measured with the new --keep-pool: 2.0 GB of tensors, a pool of 22.3 GB after 50 steps;
• MLX's speed: 1,854 tokens per second with mlx-lm 0.32.

Cut: two asides with no measurement behind them, a bfloat16 matrix multiplication's speed and a step of 4,096 tokens.

Also: stage 10's map has a row for every side topic, and the spelling is American throughout.

The films of every chapter are attached, 25 of them.

v0.5.9: the published model's tool calls, as Qwen tokenizes them

Two open items from the v0.5.8 review, measured.

6a: the published model rarely calls the tool. 6a told the published model about the calculator with the <tool_call> markers of its instructions spelled out as characters, where Qwen's own tokenizer makes them single tokens. Written Qwen's way (tools.encode(..., system_markers=True), evaluate_tools.py --system-markers):

• Calls: it calls in 5.9% of the problems, against 0.5%; 16 of its 260 calls are malformed.
• Score: 41.5% against 40.2%, a difference within noise.
• Cost of the tool's description: 5.3 points against no tool (interval 2.7 to 8.1), not 6.6.

The note, its table, figure and film, 9b's exercise and the README now give this run first. The default encoding is unchanged, so the fine-tuned model of 6a is still evaluated as it was trained.

Stage 7: the LoRA model at 320 tokens scores 36.5%, the number in stage 7's table that had no saved file. It is now runs/eval/gsm8k-sft-gsm8k-lora-320.json, with its command in "Run it".

The films of every chapter are attached, 25 of them.

v0.5.8: corrections from a review

Corrections from a review of every note against its saved runs, code and references.

Two change how earlier results should be read:

• The intervals do not cover the arithmetic. On the laptop's GPU in 16-bit floats, a greedy reply depends on the batch it was generated in: the published 0.5B model solved 48.6% and 46.6% of the same 500 GSM8K problems in two runs. The paired intervals used throughout cover which problems were drawn, not this. Stage 7 now says so, and a difference of a point or two should be read with it in mind.
• 6a's prompt for the published model spells the tool-call markers out as characters, where Qwen's own tokenizer writes them as single tokens. The published model's 0.5% call rate may partly reflect this; the note now says so.

Also:

• Numbers corrected against their files: 2b's gain from a passage read before, 3c's misses of the law, 9a's steps per round, 10's file size.
• Overstated claims qualified: 8a's refusals of safe requests, 7b's judges and long stories, 9a's shrinking gains.
• Commands in "Run it" now reproduce the saved runs of 4a, 7 and 7b.
• Citations corrected: DeepSeek-R1-Zero for the replies that grew longer, R1's training stages, STaR's rounds, the GPT-4 version of TinyStories.
• Outside tools brought up to date: vLLM on macOS, mlx_lm's decode loop, SGLang in TGI's place, FP8 training.
• Side topics: stages 3, 6, 7 and 8 now name theirs.
• 9a's film says later rounds add less, as the note does.
• scripts/dpo.py: the --beta help now says a larger beta holds the policy tighter.

The films of every chapter are attached, 25 of them.

v0.5.7: speculative decoding

A new side topic of stage 4: 4b, speculative decoding.

A cheap draft guesses the next tokens and the model checks them all in one pass, keeping those it agrees with: with the rule of Leviathan et al. and Chen et al., the output is distributed exactly as the model's own. Measured one reply at a time on this laptop's GPU: drafted by the 0.5B model, Qwen2.5-7B writes 38.2 tokens a second against 16.9 alone, 2.26 times as fast; drafted by the 1.5B model, 1.76 times. Qwen2.5-1.5B gains only 1.18 times from the 0.5B draft, a third of its size, and the story models of 3a are slower with any draft: a round of draft and check has to cost less than it yields. Prompt lookup, which drafts from the text itself and costs almost nothing, makes the 1.5B and 7B models about 1.45 times as fast on GSM8K, where a solution copies the numbers of its problem.

In 16-bit floats on the GPU, greedy replies with a draft part from the model's own at near ties, at nearly the same token whatever the draft; in 32-bit floats on the GPU, and on the CPU, every reply measured is the model's own.

Also: speculative.generate now ends a reply before its stop token, as inference.generate does; before, a reply with a draft looked one token longer than the model's own. scripts/bench_speculative.py takes --dtype, and scripts/analyze_speculative.py summarizes the measurements.

The films of every chapter are attached, 25 of them.