gemvthreads: does the CPU quantized matmul scale with cores, or hit the memory wall?

The roofline study found token decode is memory-bandwidth-bound - so one might expect the matrix- vector product at its core to stop scaling once a few threads saturate memory. This tests that directly: it runs the fused Q4_0 GEMV across thread counts, for a weight matrix that fits in cache and one that does not, and reports throughput, parallel efficiency, and the implied memory bandwidth against the machine's measured ~216 GB/s ceiling. Standalone C++ (ARM NEON).

Pre-registration

Three predictions were committed to git (PREREG.md) before the benchmark: (1) the CPU GEMV scales near-linearly with cores; (2) it is compute-bound, not memory-bound (implied bandwidth well below the ceiling); (3) the cache-resident and memory-resident matrices scale the same - the decisive test. All three held.

Result

 cache-resident (8960x1536)              memory-resident (90000x1536, ~130 MB)
 threads  GFLOP/s  eff   implied GB/s    threads  GFLOP/s  eff   implied GB/s
 1         12.1   100%     3.4           1         11.8   100%     3.3
 2         23.4    97%     6.6           2         23.1    97%     6.5
 4         42.9    89%    12.1           4         42.9    90%    12.1
 8         83.7    87%    23.5           8         83.5    88%    23.5
 14       117.6    70%    33.1           14       116.3    70%    32.7

1. The CPU quantized matmul scales near-linearly with cores. About 88% parallel efficiency at 8 threads and still 70% at full width - it uses the whole machine well, the opposite of a kernel that flatlines against a shared bottleneck.

2. It is compute-bound, not memory-bandwidth-bound. Even at full thread count the implied memory bandwidth reaches only ~33 GB/s - about a seventh of the machine's ~216 GB/s ceiling. The per-core NEON dequant-and-multiply throughput (~12 GFLOP/s) is the limit, not memory.

3. The decisive test: cache-resident and memory-resident scale identically (9.8x each at full width). A memory-bound kernel would scale worse once the 130 MB of weights no longer fit in cache and must stream from RAM; this one does not care whether the weights are cached. Memory is not the bottleneck.

The one-line finding: the same quantized matrix-vector product that is memory-bandwidth-bound on the GPU (where compute is cheap enough to saturate memory) is compute-bound on the CPU (where the NEON dequant-plus-FMA is the bottleneck), so it scales near-linearly across cores and never approaches the memory wall. A roofline's bound is a property of the processor's compute-to-bandwidth ratio, not of the operation alone.

Reproduce

./reproduce.sh 8               # RUNS
./scripts/gate.sh              # C++ -Werror build + tests, ruff, mypy, independent verify

tools/verify.py recomputes the thread scaling, the implied bandwidth versus the ceiling, and the cache-vs-memory comparison with its own median, and re-asserts the near-linear scaling, the compute-boundedness, and the identical scaling - no shared code with the analysis.

Limitations and falsifiers

• One ISA (ARM NEON on Apple silicon), one compiler, the Q4_0 format, f32 activations, up to the machine's hardware-concurrency, row-partitioned threading. The absolute GFLOP/s and the crossover to memory-bound at higher thread counts are machine-specific; the finding is that on this CPU the operation is compute-bound over the whole thread range, evidenced by the identical cache/memory scaling.
• The 216 GB/s ceiling is the sibling roofline study's measured multi-threaded STREAM triad; the conclusion (implied bandwidth is a small fraction of it) is robust to the exact value.
• Falsifier: if the memory-resident matrix had scaled worse than the cache-resident one, the operation would be memory-bound; they scale identically (9.8x each).

MIT licensed. The oracle is monotonic-clock throughput; the threaded result is tested equal to the single-threaded one; no LLM judgement.