v0.4.12

A Metal decode runs 1.4 to 1.8 times faster because the matvec stopped launching a block four times wider than the row it was reducing. Trying to remove the reduction underneath it as well is worth nothing, which is how this release also establishes that the matvec is no longer where a small model's decode goes.

Changed

• The matvec launches 32 threads a row on Metal rather than 128. TILE had been 128 since the kernel was written and it was never a decision about the matvec, it is the width every other kernel launches at. On an M4 the output head of a small model, 49152 rows of 576, was running at a tenth of what the same bytes read with no arithmetic on them. End to end that is 1.4 times on SmolLM2 135M, 1.8 on Qwen 2.5 0.5B and 1.5 on Llama 3.1 8B, same binary either side of the one constant. The logit corpus passes all thirteen cases on both widths, with every figure moving in the fifth or sixth significant digit because the additions happen in a different order.
• The same sweep now has its answer on CUDA. A 4090 at load 0.06 finds the same pattern, the head 1.85 times faster at 32 and the wide control preferring 128, but end to end it is worth about one per cent, so CUDA keeps 128.
• A shuffle reduction under the narrower block was written, measured and reverted. It is a wash on both small models and it changes the order the partial sums are added in, which is not worth paying for nothing.

What the negative results establish

Narrowing the block was worth 1.4 to 1.8 times and removing the reduction under it is worth zero, so what 128 threads a row was costing was never the reduction. It was the threads that arrived at a 576 column row with nothing in them.

On Metal, which runs one path whatever the model, decode time a token against bytes a token fits a slope of 29.4 ms a GiB and an intercept of 13.6 ms a token that does not move with the model. A thirty layer token is around 450 launches at the 19.6 us measured, which is 8.8 ms, so most of that intercept is submission and Metal decode on a small model is launch bound.

CUDA is a different picture and the first version of these notes got it wrong. Models under a gibibyte of layer weights decode through the fused kernel there, so SmolLM2 and Qwen are fused and the 8B is not, and a line through all three has an intercept that means nothing. With the path pinned, a fused SmolLM2 token is 2.16 ms against 5.58 unfused, and the fused path is already one launch a layer, so its submission is about 0.15 ms and its weights about 0.22 ms. The remaining 1.9 ms is the fused kernel's own grid and barriers. On the 8B fused and unfused are within 5 per cent, 12.4 against 13.0, because the bus is the rest.

So kernel tuning is finished as a way to move a small model on either backend, and what is in the way is different on each: submission on Metal, and the fused kernel's grid on CUDA.

What's Changed

• Launch the Metal matvec at 32 threads a row, and release 0.4.12 by @tamnd in #229

Full Changelog: v0.4.11...v0.4.12