neonkernel: the throughput-accuracy frontier of a quantized decode GEMV

Token generation is dominated by matrix-vector products y = W x, one per weight matrix per layer. In a quantized model W and x are 8-bit integers with dequant scales, and the inner product accumulates in int32. This measures the same GEMV four ways - scalar int8, hand-written NEON SDOT int8, scalar f32, hand-written NEON FMA f32 - across four realistic decode shapes (attention and FFN projections at two model widths), and traces the throughput (GFLOP/s) against the accuracy (relative-L2 of the int8 result versus an f64 reference).

Everything is standalone C++ (ARM NEON), -O3 -march=native, no model server involved. Data is deterministic so accuracy is bit-reproducible; each cell is timed over 12 runs for bootstrap CIs.

Pre-registration

Three predictions were committed to git (PREREG.md) before the authoritative run: (1) quantization is the dominant throughput lever (int8 SDOT >= 3x the best f32 kernel); (2) that speedup costs < 1% relative-L2; (3) the hand-vectorization premium over the optimizing compiler is type-dependent - larger for f32 than int8. All three held.

Result

Median GFLOP/s (95% CI) and int8 accuracy, on the 1.5B FFN shape (M=8960, K=1536), representative of all four:

 kernel       GFLOP/s         rel_L2 vs f64
 scalar_f32     4.5           1.5e-07
 neon_f32      13.1           1.5e-07
 scalar_i8     62.8           4.2e-04
 sdot_i8       83.7           4.2e-04

Across all four shapes (medians):

1. Quantization is the throughput lever: 6.4x. The int8 SDOT GEMV runs at 6.4x the throughput of the fastest f32 kernel, because it moves one-quarter the bytes and the SDOT instruction folds 16 int8 multiply-accumulates into one op. The cost is a 0.045% relative-L2 error - absmax int8 quantization of a dot product is essentially free in accuracy at this width.

2. Hand-writing the kernel beats the compiler, but how much depends on the type. For f32 the hand-NEON kernel is 3.1x the scalar loop; for int8 the hand-SDOT kernel is only 1.46x the scalar loop. The asymmetry is not luck - it is a language-semantics fact confirmed in the emitted assembly:

   • The scalar f32 reduction compiles to scalar fmadd only; the compiler will not reassociate a floating-point sum without -ffast-math, so it cannot vectorize the reduction. The hand-NEON kernel opts into that reassociation explicitly and gets the vector fmla.
   • The scalar int8 reduction compiles to vector smlal/smull widening multiply-accumulates - integer addition is associative, so the compiler already auto-vectorizes it. Hand-writing SDOT still helps (SDOT is denser than widening MACs) but only modestly.

The one-line finding: the big win is quantization (6.4x at 0.045% error); hand-vectorization pays most exactly where the compiler is forbidden to vectorize by floating-point semantics (f32 reductions, 3.1x) and least where it is already free to (associative integer accumulation, 1.46x).

Reproduce

./reproduce.sh 2000 12          # REP RUNS
./scripts/gate.sh               # C++ -Werror build + tests, asm-check SDOT, ruff, mypy, pytest, verify
make asm                        # print the SDOT instruction count in the hand kernel

tools/verify.py is an independent recompute (its own median, no shared code) that re-derives the quantization lever, the accuracy, and the type-dependent hand-vectorization premium from the raw benchmark rows.

Limitations and falsifiers

• One ISA (ARM NEON on Apple silicon), one compiler (-O3 -march=native), one quant scheme (symmetric absmax int8 with per-row/per-vector scales), single-threaded, four GEMV shapes. Not a claim about other ISAs, block-quant formats (e.g. Q4_K), GPUs, or multi-threaded throughput.
• GFLOP/s here is single-core kernel throughput on data sized to fit the shapes; it is not end-to-end tokens/sec. The point is the relative ordering of kernels and its mechanism, which the assembly substantiates.
• Accuracy is for a single GEMV; error accumulation across a full network is out of scope.
• Falsifier: if the scalar int8 loop did not auto-vectorize (no smlal/sdot in its asm), the type-dependence explanation for prediction 3 would be wrong. It does (checked in the gate).
• Falsifier: if int8 relative-L2 exceeded 1%, the "nearly free" accuracy claim would be wrong.

MIT licensed. Ground truth is an f64 reference GEMV; no model-quality judgement involved.