latencytail: what does each concurrent request cost a local inference server?

A local llama.cpp server decodes one token at a time. When several clients stream at once, where does the latency go - into per-token jitter, into a slower steady rate, or into waiting? This measures the full inter-token latency distribution, time-to-first-token (TTFT), and end-to-end latency as a function of the number of concurrent request streams, closed-loop (concurrency C = exactly C requests in flight), for two model sizes (1.5B and 0.5B, Q4_K_M). The server runs 4 parallel slots.

Client is a C++ libcurl load generator using true OS-thread concurrency (no interpreter lock), so it does not manufacture the server-side latency it is trying to measure. Each token's arrival is timestamped with a monotonic-raw clock. 7 concurrency levels x 2 models x 3 shuffled repeats; warmup discarded; >=12000 inter-token samples per cell; bootstrap CIs on quantiles.

Pre-registration

Three predictions were committed to git (PREREG.md) before the authoritative run: (1) inter-token jitter stays flat and low (p99/p50 < 1.5) at every concurrency; (2) a batching tax - per-token latency grows to the slot count then saturates; (3) a queueing cliff - TTFT jumps once concurrency exceeds the slots. A pilot probe had suggested all three.

Prediction 1 was falsified, and in the interesting way. The higher-powered run shows jitter is low everywhere except exactly at the slot-saturation point, where it spikes. So the picture is three regimes, not two. Predictions 2 and 3 held.

Result: three regimes of concurrency cost

qwen2.5-1.5B

 C   gap_p50  gap_p99  gap_p999  jitter(p99/p50)  ttft_p50  ttft_p99  e2e_p50
 1     6.15     7.55      8.29        1.23           27.2      30.2     418
 2     9.47    11.56     37.68        1.22           38.1      68.1     656
 3    16.29    20.66     56.89        1.27           65.2     105.2    1113
 4    18.35    41.06     67.47        2.24           60.9     105.9    1253
 6    19.14    27.75     57.69        1.45          698.5    1471.3    2009
 8    18.38    20.73     21.71        1.13         1304.2    1368.9    2474
 12   18.34    21.11     24.76        1.15         2529.1    2646.0    3699

qwen2.5-0.5B

 C   gap_p50  gap_p99  gap_p999  jitter(p99/p50)  ttft_p50  ttft_p99  e2e_p50
 1     3.74     4.97      5.41        1.33           12.6      14.2     251
 2     8.57    13.14     17.51        1.53           23.4      37.2     574
 3     8.74    11.39     27.75        1.30           32.2      43.3     587
 4     6.69    17.84     30.58        2.67           30.9      54.2     471
 6     7.06     8.93     23.79        1.27          253.5     548.7     711
 8     6.75     8.04      8.96        1.19          489.2     526.1     919
 12    6.75     8.16      9.07        1.21          948.5     993.2    1375

(all times in ms; gap = inter-token latency)

1. Below saturation (C < slots): a batching tax. Median inter-token latency grows with concurrency - 1.5B 6.2 -> 18.4 ms from C=1 to C=4, 0.5B 3.7 -> 6.7 ms - because the server decodes the active requests as one batch and each step does more work. It then saturates: from C=4 to C=12 the per-token median barely moves. (Aggregate throughput is roughly flat, so the batch buys little here.)

2. At saturation (C = slots): a jitter spike. Exactly at C=4 - the slot count - inter-token p99/p50 jumps to 2.24 (1.5B) and 2.67 (0.5B), versus ~1.2-1.3 everywhere else. This is the one concurrency where decode is not smooth. The batch is full but nothing is queued yet, so its composition churns: requests of different remaining lengths finish and new ones join mid-batch, and each reshuffle costs a longer decode step. The spike appears in both models at exactly the slot count, across all three repeats - it is a property of the saturation boundary, not noise.

3. Past saturation (C > slots): a queueing cliff. Once concurrency exceeds 4, per-token latency and jitter both flatten (the active batch is pinned at 4), and the entire additional cost moves into waiting for a slot: median TTFT climbs 27 -> 2529 ms (1.5B, 93x) and 13 -> 949 ms (0.5B, 75x) from C=1 to C=12, while a token, once flowing, arrives as smoothly as at C=1.

The one-line finding: decode is deterministic under load except at the saturation knee; below it you pay a per-token batching tax, above it you pay a steep queueing latency, and the tail is worst not at maximum load but exactly where the slots fill.

Reproduce

./reproduce.sh 8081 8082 3      # PORT_15B PORT_05B REPEATS
./scripts/gate.sh               # C++ build (-Werror) + tests, ruff, mypy --strict, pytest, verify

tools/verify.py is an independent recompute (its own plain-Python quantile, no shared code with analyze.py or the C++ harness) that re-derives the three regimes from the raw per-request rows.

Limitations and falsifiers

• One server backend, one build, 4 slots, two model sizes, closed-loop (fixed in-flight concurrency), n_predict=64. Not a claim about other backends, slot counts, or open-loop (Poisson) arrivals - though the slot-saturation mechanism should generalize wherever a fixed batch width is the scheduling unit.
• The client shares the machine with the server; client threads are I/O-bound (blocked on recv), and the near-identical solo inter-token latency measured client-side (6.15 ms) and server-side (6.3 ms from the server's own timings) shows the client is not the bottleneck.
• Slot count is read from the server configuration, not inferred; the "saturation = slot count" claim is anchored to that known value.
• Falsifier: if the C=4 jitter spike did not reproduce across repeats or appeared at a concurrency other than the slot count, regime 2 would be an artifact. It reproduced in both models at exactly C=4.
• Falsifier: if median TTFT past the slot count did not climb steeply, regime 3 would be wrong.

MIT licensed. Ground truth is monotonic-clock wall time; no model-quality judgement involved.