v0.4.4

Three of the kernels a layer launches were doing one flop per element, and every one of them can be done by the kernel in front of it for free.

A matvec thread block owns one output row and has that row in a register when its reduction finishes. Anything that reads only that row, and nothing else the same launch is still writing, is work the block can finish itself. The projection bias, the residual add and the gate half of a gated MLP are all exactly that, so all three are epilogues now rather than launches. A layer goes from fifteen launches to twelve on a model without qkv bias and from eighteen to twelve on one with it.

Decode on an RTX 4090, eight interleaved repetitions: SmolLM2 135M q8_0 from 282.2 tokens per second to 310.8, Qwen 2.5 0.5B q4_K_M from 244.9 to 309.2, Llama 3.1 8B q4_K_M from 99.8 to 101.4. The 8B barely moves because at 5.6 GiB a token its launches were already hidden behind its weight reads.

It is exact, so the logit corpus is byte identical to 0.4.3 on both backends and no tolerance moved.

Full changelog: https://github.com/tamnd/molla/blob/v0.4.4/CHANGELOG.md

What's Changed

• Fold the bias, the residual add and the glu into the matvec epilogue by @tamnd in #194
• Release 0.4.4 by @tamnd in #195

Full Changelog: v0.4.3...v0.4.4