determinism: is quantized inference bit-reproducible, and when is it not?

If you rerun the same prompt through a local llama.cpp server, do you get the same numbers? Not just the same text - the same logprobs, bit for bit? This matters whenever logprobs are logged and later compared: calibration, perplexity, reward scoring, answer ranking, regression tests. This audits reproducibility at the LOGIT level (token id + full-precision logprob + the top-k vector) across conditions widely believed to perturb numerics, on two model sizes (1.5B and 0.5B, Q4_K_M).

The oracle is exact stream equality, hashed. Crucially, the audit includes POSITIVE CONTROLS - different prompt, different seed, different temperature - which MUST diverge, so an "everything is identical" result cannot be a broken comparator quietly returning true.

Pre-registration

Four predictions were committed to git (PREREG.md) before the authoritative audit: (1) serial repeats are logit-identical; (2) concurrency / cross-batch / cache reuse perturb the logits; (3) the greedy token stream survives that perturbation (same text, different logprobs); (4) seeded sampling does not survive it. All four held.

Result

Each condition compares a logit stream to a fixed solo baseline. logit-ident = full stream bit-identical; token-ident = only the sampled token ids match. Pooled over both models:

 condition               logit-ident      token-ident
 repeat_greedy            60/60 (100%)     60/60 (100%)   <- serial, incl. seeded repeats
 concurrent_greedy_c2      2/4  ( 50%)      4/4  (100%)   <- threshold
 concurrent_greedy_c4      0/8  (  0%)      8/8  (100%)
 concurrent_greedy_c8      0/16 (  0%)     16/16 (100%)
 concurrent_seeded_c6      2/12 ( 17%)      4/12 ( 33%)   <- sampling flips
 cross_batch               0/2  (  0%)      2/2  (100%)
 cache_reuse               0/2  (  0%)      2/2  (100%)
 controls (prompt/seed/temp)  0/6 diverge at both levels  <- comparator detects real differences

1. Serial reproducibility is perfect. Repeated identical calls are 60/60 logit-identical. Given a fixed prompt, params, and seed, run solo, the numbers are exactly repeatable.

2. Any execution-context change perturbs the logits - but not the greedy tokens. Concurrency at C>=4, sharing a batch with different prompts, and prefix-cache reuse are 0/28 logit-identical yet 28/28 token-identical. The floating-point reduction order depends on batch composition and scheduling, and cache reuse reloads KV rather than recomputing it, so the logits move in the low bits - but the top token's margin almost always exceeds that jitter, so the decoded text is unchanged. "Same text" is not "same logprobs".

3. Stochastic sampling is the exception. Under concurrency, seeded temperature sampling reads the whole jittered distribution, so even the sampled tokens diverge: only 4/12 token-identical. A fixed seed is not sufficient for reproducibility under load.

4. There is a concurrency threshold. C=2 is still (partly) bit-identical; C>=4 never is.

The practical rule: greedy-text evaluation is reproducible under load; anything that reads logprobs (calibration, perplexity, reward, ranking) or samples is NOT reproducible across concurrency or cache state - serialize it, or pin the execution context, if you need the numbers to match.

Reproduce

./reproduce.sh 8081 8082        # PORT_15B PORT_05B
./scripts/gate.sh               # ruff, mypy --strict, pytest, ASCII, independent verify

tools/verify.py recomputes identity independently from the stored stream hashes (not the recorded divergence fields) and re-asserts all four findings, sharing no logic with the audit or analyze.

Limitations and falsifiers

• One backend (llama.cpp on Apple GPU, -fa), one build, Q4_K_M, two model sizes, concurrency up to 8. Not a claim about other backends, quant formats, or CPU-only execution, where the reduction order and thus the reproducibility boundary may differ.
• The audit measures reproducibility relative to a solo baseline; it does not claim the logits are "wrong" under load, only that they are not bit-identical.
• Falsifier: if the positive controls had NOT diverged, the comparator would be vacuous and the reproducibility rows meaningless. They diverge 6/6, at both the token and logit level.
• Falsifier: if greedy tokens had changed under concurrency (not just logprobs), finding 2 would be wrong. They do not (28/28 token-identical).

MIT licensed. The oracle is exact stream equality; no model-quality judgement involved.