dequant: how fast can a NEON kernel unpack 4-bit weights to floats?

A quantized model stores its weights packed at 4 bits; before any of them can be multiplied they must be dequantized back to floats. Since the roofline study showed decode is bound by streaming those weights, the unpack sits directly on the critical path. This implements Q4_0 dequantization - scale * (nibble - 8), 32 weights per block sharing one f16 scale - two ways, a scalar reference and a NEON kernel that unpacks a whole 16-byte block per pass, and measures throughput and exactness across weight-matrix sizes. Standalone C++ (ARM NEON), no server.

Pre-registration

Three predictions were committed to git (PREREG.md) before the benchmark: (1) the NEON dequant is more than 2x the scalar throughput; (2) it is bit-exact against scalar (no approximation); (3) the speedup is consistent across matrix size. All three held.

Result

 weights          scalar Mw/s   neon Mw/s   speedup   exact
 2.4M (1536x1536)     4868        12918      2.7x      yes
 13.8M (8960x1536)    4807        12768      2.7x      yes
 233M (1536x151936)   4569        12456      2.7x      yes

1. NEON dequant is 2.7x the scalar throughput, consistently. Across every matrix size, unpacking a 16-byte block at once (mask the low nibbles, shift the high nibbles, subtract 8, widen to 32-bit, multiply by the scale) reaches ~12.8 billion weights per second versus ~4.8 for the one-value-at-a-time scalar loop.

2. It is bit-exact - the speedup costs no accuracy. The maximum absolute difference from the scalar result is exactly zero at every size, because dequantization is integer unpacking times a shared scale, with no approximation. This is the opposite of the sibling softmax kernel, whose NEON speedup came from an approximate exp; here the 2.7x is free.

3. Dequant is small but on the critical path. At 12.8 billion weights/second, unpacking a 1.5B model's weights costs about 0.12 ms - a couple of percent of a decode step, which is dominated by the memory read and the matrix multiply. It is not the bottleneck, but it is paid on every weight the model touches, so a 2.7x cheaper unpack is a direct, free saving on the hot path.

The one-line finding: unpacking 4-bit weights to floats vectorizes to 2.7x with a NEON kernel at zero accuracy cost - a free speedup on an operation every weight passes through, in contrast to the softmax kernel where the NEON win was bought with an approximation.

Reproduce

./reproduce.sh 200 12           # REP RUNS
./scripts/gate.sh               # C++ -Werror build + tests, asm-check vector unpack, ruff, mypy, verify
make asm                        # count the vector unpack/convert ops in the NEON kernel

tools/verify.py recomputes the speedup and the bit-exactness with its own median and asserts the

2x speedup, the zero max-abs-diff, and the consistency across sizes - no shared code with the analysis.

Limitations and falsifiers

• One ISA (ARM NEON on Apple silicon), one compiler (-O3 -march=native), the Q4_0 block format, single-threaded, three matrix sizes. The speedup magnitude is implementation-specific; the bit- exactness is a property of the format (integer unpack times a scale).
• This measures the dequant kernel in isolation; end-to-end decode also streams the packed weights and runs the matrix multiply, which the roofline study covers.
• Falsifier: if the NEON speedup were under 2x, or the max-abs-diff nonzero, the findings would be wrong. It is 2.7x and exactly zero at every size.

MIT licensed. Ground truth is the scalar dequant; correctness is exact equality against it.