v0.4.10

A decode runs a layer in one launch on CUDA. A Llama shaped layer used to be twelve launches and a thirty layer token 363 of them, and it is 33 now. On a 4090 at a 512 token prompt and 128 generated, SmolLM2 135M goes from 264.5 to 492.3 tokens a second and Qwen 2.5 0.5B from 304.8 to 367.8, both against llama.cpp at 888.9 and 767.9 and Ollama at 682.0 and 490.8 in the same sitting.

It is off for a large model and off on Metal, and both of those are measurements rather than caution. Llama 3.1 8B decodes at 81.8 tokens a second fused against 93.7 unfused, because a few hundred blocks is narrower than the block a row the unfused matvec launches and a model that reads five gibibytes a token is bound by that rather than by submission.

See CHANGELOG.md for the full entry.

What's Changed

• Say why a repack failed instead of what the file it did not replace looks like by @tamnd in #221
• One kernel a layer by @tamnd in #200
• Retake the gpc tables on the fused build and cut 0.4.10 by @tamnd in #223

Full Changelog: v0.4.9...v0.4.10