Evaluation harnesses ask the server for token logprobs - for calibration, perplexity, answer scoring, reward modelling. Does asking for them slow generation down, and does asking for more of them cost more? This measures decode per-token latency with n_probs = 0 (plain) versus requesting the top-k logprobs, sweeping k, on two model sizes (1.5B and 0.5B, Q4_K_M), greedy.
Three predictions were committed to git (PREREG.md) before the sweep: (1) requesting logprobs adds
a per-token overhead; (2) the overhead grows with k; (3) it is larger for the smaller model.
Predictions 1 and 3 held; prediction 2 was falsified - the overhead is a fixed step, nearly
independent of k.
model baseline k=5 k=20 k=100 k=300
1.5B 6.174 ms 1.063x 1.063x 1.066x 1.073x
0.5B 3.686 ms 1.101x 1.106x 1.103x 1.126x
(times relative to plain decoding)
-
Requesting logprobs costs a fixed per-token overhead: +6% (1.5B), +10% (0.5B). It is real - the k=5 overhead is 1.063x [95% CI 1.049, 1.078] for the 1.5B and 1.101x [1.088, 1.112] for the 0.5B - and it is paid the moment you ask for any logprobs at all.
-
The overhead is flat in k - asking for more logprobs is essentially free. Going from the top-5 to the top-300 adds only +1.0% (1.5B) and +2.6% (0.5B) on top of the fixed step; the overhead varies just 1-2% across the whole k range. The expensive part - the softmax over the ~150k-token vocabulary - happens on every decode step regardless of whether logprobs are returned; extracting 5 versus 300 top values from it is negligible. So the pre-registered "cost grows with k" is wrong: it is a fixed cost of the request, not a per-logprob cost.
-
The overhead is larger for the smaller model (+10% vs +6%). The extraction and serialization is a roughly fixed cost, and it is a bigger fraction of the 0.5B's cheaper forward pass - the same fixed-cost-over-cheaper-baseline pattern that shows up whenever a small per-token add-on meets a faster model.
The one-line finding: logprobs cost a flat ~6-10% at the point of asking, not per logprob - so an eval that already requests logprobs loses nothing by requesting the full top-k instead of a handful, and a latency-sensitive path loses ~6-10% the moment it turns logprobs on.
./reproduce.sh 8001 8002 # PORT_15B PORT_05B (the ports your servers run on)
./scripts/gate.sh # ruff, mypy --strict, pytest, ASCII, independent verify
tools/verify.py recomputes the per-k overhead with its own median and asserts the fixed real cost,
the flatness (5->300 adds far less than the initial step), and the cross-model ordering - no shared
code with the analysis.
- One backend (llama.cpp on Apple GPU), one build, Q4_K_M, two sizes, greedy, k up to 300, n=100 tokens x 6 reps. The absolute percentage is backend-specific; the shape (fixed step, flat in k) is the claim.
- Per-token latency is the server's own predicted_per_token_ms; the study measures generation latency, not the size of the returned payload.
- Falsifier: if the k=300 overhead were much larger than the k=5 overhead, the cost would be per-logprob and finding 2 wrong. It is within 1-3%.
- Falsifier: if the smaller model had the smaller overhead, finding 3 would be wrong; it is +10% vs +6%.
MIT licensed. The oracle is exact server timing; no model-quality judgement involved.