Token generation reads every model weight once per token to do a single multiply-add each - two
FLOPs per weight. That is an arithmetic intensity of 2*params / weight_bytes, which for a 4-bit
model is only 2-3 FLOP/byte. A roofline says an operation that low-intensity is limited by memory
bandwidth, not compute. This measures the bandwidth ceiling (a multi-threaded STREAM triad in C++),
the decode rate at several batch sizes (from the server), and places decode on the roofline. Two
model sizes (1.5B and 0.5B, Q4_K_M).
Three predictions were committed to git (PREREG.md) before the measurement: (1) single-token
decode is memory-bandwidth-bound (intensity far below the ridge); (2) a single stream already uses a
large fraction (>40%) of peak bandwidth; (3) batching amortizes the weight read and raises aggregate
throughput but stays memory-bound. All three held.
Measured memory-bandwidth ceiling (STREAM triad): 221 GB/s multi-thread, 124 GB/s single-thread. Roofline ridge at a nominal 2000 GFLOP/s compute peak: AI = 9.0 FLOP/byte.
model AI(1 tok) batch agg_tok/s GFLOP/s per-token-read GB/s % of peak BW
1.5B 3.12 1 151 466 149 67%
2 143 442 141 64%
4 207 639 205 92%
0.5B 1.99 1 245 240 120 54%
2 403 395 198 90%
4 554 542 272 123%
-
Single-token decode is memory-bandwidth-bound. Both models have an arithmetic intensity of 2-3 FLOP/byte, far below the ~9 ridge, so they sit deep on the memory-bound (diagonal) part of the roofline. A model would need ~9 FLOP/byte to become compute-bound; decode has a third of that.
-
A single stream already runs near the memory ceiling. One decode stream implies 54-67% of the measured peak bandwidth - decode is not idling on memory, it is close to saturating it with a single request.
-
Batching amortizes the weight read - and the numbers prove it. Aggregate throughput rises with batch size (0.5B 245 -> 554 tok/s, 1.5B 151 -> 207). At batch 4 the 0.5B's per-token-read bandwidth computes to 123% of the physical ceiling - impossible if each token re-read the weights, so the weights must be read once per step and shared across the batch. That sharing raises the effective arithmetic intensity and walks the operating point right along the roofline
- toward, but not reaching, the compute ceiling. The gain saturates as the bandwidth ceiling is hit.
The one-line finding: decode is a memory-bandwidth problem, not a compute problem. A single stream already uses over half the memory bandwidth; batching helps only by amortizing weight reads against that same wall, and the smaller model - fewer bytes per token - both starts further from the ceiling and batches better. This is the mechanism behind the batching tax and the flat single-stream throughput seen at the request level.
./reproduce.sh 8081 8082 # PORT_15B PORT_05B
./scripts/gate.sh # C++ -Werror build, ruff, mypy --strict, pytest, verify
tools/verify.py is an independent recompute (its own arithmetic-intensity and batch-scaling
derivation, no shared code) that re-asserts the memory-bound conclusion, the near-ceiling single
stream, and the amortization evidence from the raw stream and decode rows.
- One backend (llama.cpp on Apple GPU), one machine, Q4_K_M, two model sizes, batch up to 4. The bandwidth ceiling is a CPU STREAM triad; the GPU decode achieves an effective bandwidth in the same range, which is why STREAM is a valid scale reference, but it is not a direct GPU-bandwidth measurement.
- Parameter counts are the nominal published values; weight bytes are the exact on-disk sizes. The memory-bound conclusion is robust to the compute-peak assumption because the intensity (2-3) is far below the ridge for any plausible peak.
- Falsifier: if the arithmetic intensity were above the ridge, decode would be compute-bound; it is 2-3 vs a ridge of 9.
- Falsifier: if batching gave no aggregate gain, the amortization claim would be wrong; it gives 1.4x-2.3x and drives the per-token-read bandwidth past the physical ceiling.
MIT licensed. The oracle is exact arithmetic plus monotonic-clock timing; no model-quality judgement.