Repository navigation
The notes of two side topics now say what their results would need to improve, and what the papers they follow measured.
- 2c, latent attention: DeepSeek-V2's ablations on MMLU (7B dense models: full multi-head 45.2%, grouped-query 41.2%, multi-query 37.9%; latent attention ahead of full multi-head in mixtures of experts of 16B and 250B, with 14% and 4% of its cache). Latent attention can only gain where sharing keys costs something measurable, which this laptop's model size does not reach. A fused kernel such as FlashMLA is the way out of its slow decoding of a single sequence.
- 4c, continuous batching: the story model's running batch would need a prefill cheaper than a step per request, batched or chunked as in Sarathi-Serve. Batch-invariant kernels (He, 2025) make greedy replies independent of the batch. The gains Orca (36.9 times FasterTransformer's throughput at the same latency), vLLM (2 to 4 times) and Sarathi-Serve (2.6 to 3.7 times vLLM's request rate) measured.
The films are unchanged and attached again, all 28.