v0.4.3

The decode matvec spent more of its time turning integers into floats than multiplying them, and there is a way to do that with no conversion instruction in it.

Float32(n) for a small integer compiles to a convert, and on NVIDIA a convert issues on the same unit as transcendentals at a quarter of the rate of a multiply. A byte of two four bit weights costs four of them, which is worth sixteen multiplies before any multiplying has happened, and the kernel only does four. A float32 whose exponent is 23 has a mantissa step of exactly one, so the representable numbers between 2^23 and 2^24 are the integers and their bit patterns are 0x4B000000 plus the integer. Nothing carries out of the mantissa, so the addition is an or. Or the weight in, subtract 2^23 back off, and the conversion is an integer or and a float subtract, both full rate on both vendors.

Decode on Llama 3.1 8B q4_K_M goes from 72.9 tokens per second to 99.7 on an RTX 4090 and from 2.0 to 2.9 on an Apple M4, against llama.cpp's 163.7 on the same card and file. The byte wide types do not move and were not expected to, because a signed byte load already sign extends in the hardware.

It is exact rather than close, so the whole logit corpus is unchanged in every digit on both backends and no tolerance moved.

Full notes in CHANGELOG.md.

What's Changed

• Turn a quant into a float with a bit pattern instead of a convert by @tamnd in #192
• Release 0.4.3 by @tamnd in #193

Full Changelog: v0.4.2...v0.4.3