A Kubernetes CPU limit is enforced by the Linux CFS bandwidth controller: the
container gets limit x 100ms of CPU per 100 ms period, and once it spends that
quota it is throttled - descheduled until the next period. Operators read a limit
as an average ("400m = 0.4 cores"), but CFS enforces it per 100 ms window. A
request that needs a short CPU burst cannot finish inside one period under a tight
limit, so it is throttled repeatedly and its wall latency balloons - while the
node's CPU dashboard shows it nearly idle. This measures the effect directly on a
real cluster and reads the throttling straight from the cgroup.
Real single-node Kubernetes (minikube, cgroup v2, kernel 6.10). A server that spends 60 ms of CPU per request and returns its wall time; a single-stream client with a gap between requests, so the pod draws only a trickle of the node's CPU.
Four predictions were committed to git (PREREG.md) before the run: (P1)
throttling happens at utilization far below the limit; (P2) tail latency inflates
as the limit tightens; (P3) the inflation is throttling, not queueing; (P4) it is
a per-period artifact, not a CPU shortage. P2, P3, and P4 held decisively. P1 was
falsified for single-threaded work - and the reason is itself informative (below).
60 ms of CPU work per request. Node has 4 cores. 40 requests per limit.
CPU limit p50 p99 p99 inflation pod util util/ throttled periods
(ms) (ms) (p99/work) (cores) limit ms/req throttled
100m 502 584 9.73x 0.08 0.82 337 69%
200m 221 282 4.70x 0.14 0.68 116 52%
400m 90 120 1.99x 0.20 0.50 29 33%
800m 60 60 1.00x 0.22 0.28 0 0%
1600m 60 60 1.00x 0.22 0.14 0 0%
(p99 is over 40 requests per limit - the near-max tail. The pod-util column is the burst pod's own cgroup usage; the node has 4 cores.)
-
A CPU limit inflates tail latency, sharply, as it tightens. (P2, held - the headline.) A request that costs 60 ms of CPU has a p99 wall latency of 584 ms at a 100m limit - a 9.7x inflation - falling monotonically to 282 ms (4.7x), 120 ms (2.0x), and finally 60 ms (1.0x, no inflation) as the limit loosens. The inflation grows roughly as
work / limit: at 100m the burst gets 10 ms of CPU per period and stalls for the other 90, over and over. -
The inflation is CFS throttling, and it vanishes when the burst fits a period. (P3, held.) The cgroup's throttled time per request - 337 ms, 116 ms, 29 ms, 0, 0 - tracks the latency inflation exactly. At 800m the per-period quota is 80 ms, which fits the 60 ms burst in a single period, so throttling drops to zero and latency falls to the raw work time. Nothing about the request changed; only whether its burst fit inside one 100 ms window.
-
The pod is denied idle CPU on its own node. (P4, held.) The burst pod uses only 0.08-0.22 of the node's 4 cores across every run (under 6% of the node) - the CPU the limit denies it is sitting idle on the same node. The tail is not CPU contention or queueing; it is purely the per-period quota cap. This is the trap: the pod's CPU usage looks tiny while its requests are throttled and p99 is 10x inflated.
-
P1 (throttled far below the limit) was falsified for single-threaded work - and that is a finding. We predicted throttling while utilization sits far below the limit. Instead, wherever throttling occurs the utilization is close to the limit (0.82 of the 100m limit, 0.68 of 200m, 0.50 of 400m); the only limits with utilization far below them (800m: 0.28, 1600m: 0.14) are the ones that do not throttle. A single thread can use at most one core, so when capped below one core it simply runs at its limit continuously. The classic "throttled at low utilization" trap requires multi-threaded fan-out - many threads briefly demanding many cores and exhausting the quota in a fraction of the period - which a single request stream does not exhibit.
On a real Kubernetes cluster a CPU limit inflates a 60 ms-CPU request's p99 wall latency up to 9.7x (584 ms) purely through CFS per-period throttling, with the inflation tracking the cgroup's throttled time and vanishing the moment the limit lets the burst fit one 100 ms period - all while the pod uses under 6% of the 4-core node, so the CPU it is denied is idle on the same node - which makes a CPU limit a tail-latency hazard for bursty, latency-sensitive services even when there is abundant CPU to spare.
./reproduce.sh # analyze + independently verify from committed runs (no cluster)
results/runs.jsonl records every request's wall latency and the cgroup cpu.stat
delta per limit. tools/verify.py re-reads it, recomputes the p99, inflation,
utilization, and throttle metrics with its own arithmetic (no shared code with
src or analyze.py), and re-asserts every threshold - including P1's honest
falsification. To regenerate on a real cluster: python tools/run_bench.py with
kubectl pointed at a running cgroup-v2 Kubernetes cluster (see manifests/).
- One cluster (minikube, single node, cgroup v2), one single-threaded CPU-burst microbenchmark, one request stream, the default 100 ms CFS period (Kubernetes does not expose the period). CPU limits only, not requests. The loosest limit is 1600m (in place of the pre-registered "unlimited"); both exceed the 600m quota-fits-the-burst crossover, so the null end is unaffected.
- The pod-utilization figures are the burst pod's own cgroup usage; the study does not measure node-wide CPU, so the claim is "the pod is denied CPU that is idle on its node," not a node-utilization measurement. p99 is over 40 requests per limit (a near-max tail on a bimodal distribution, not a finely estimated percentile).
- P1 falsified (reported as-is): single-threaded work cannot demand more than one core, so it cannot be throttled while its utilization is far below a sub-core limit; the sub-limit-utilization trap needs multi-threaded fan-out. The measured throttling here is real and drives the tail, but its utilization-relative form differs from the multi-threaded case.
- Falsifier that did not fire (P2/P3): had the limit not throttled bursts, p99 would be flat near the 60 ms work time; instead it is 9.7x inflated at 100m and falls to 1.0x exactly when the quota fits the burst.
MIT licensed. Latency is the server's own wall time; throttling is read from the cgroup. No LLM judgement.