v1.1.2: Continuous batching

A new side topic, 4c: continuous batching. Stage 4's batches start and end together: a row whose reply has ended waits for the longest one, and a new request waits for the whole batch. A running batch lets requests join and leave between any two steps, as every inference server does. Every row is at its own position, with its own rotary angles, its own mask over the cache and its own temperature and top_p. The server of stage 10 now answers requests that arrive together in one running batch.

• Qwen2.5-0.5B on 256 GSM8K problems. The replies are 294 tokens long on average and up to 512.
  • With 16 slots and every request at once, a running batch ends them all in 89.6 seconds, where batches that end together take 123.5.
  • With requests arriving at random, 4 a second, a request waits 7.6 seconds for its first token instead of 27.3.
  • With 64 slots, that wait is 0.07 seconds instead of 7.5.
• The 14M story model on 512 requests. Its replies are alike in length, and on this model a forward pass costs about the same for 1 row as for 64. The prefill of every request on its own then eats the gain. With 64 slots and every request at once, the running batch ends later, after 9.8 seconds instead of 8.2. Arriving at random, a request still gets its first token sooner: 0.01 seconds instead of 0.69.
• Greedy replies in bfloat16. 158 of Qwen's 256 greedy replies come out different somewhere when batched otherwise, because a sum's rounding depends on the shape of the batch. All 512 of the story model's come out the same. In 32-bit floats on the CPU, the test gets exactly the reply of each request alone.

New code: minllm/batching.py (Batcher, sample_rows), positions per row in Transformer.hidden, bench_batching.py, and serve.py --slots --context.

The films of every chapter are attached, 28 of them.