capsim

Capacity planning and autoscale simulator. Predicts replica needs from traffic using a queueing-theory model, drives preemptive scaling on a real Kubernetes cluster, and compares the prediction against reactive HPA scaling under load.

How it works

k6 scenarios ──► echo service (CPU-bound /work endpoint)
                    │  /metrics
                    ▼
               Prometheus ◄── kube-state-metrics (replica counts)
                    │
                    ▼
               predictor ── M/M/c sizing + rate forecast ──► /predict, capsim_predicted_replicas
                    │
                    ▼
                 scaler ── patches HPA minReplicas floor ahead of load
                    │
                    ▼
              echo HPA (CPU 60%) — still reacts on top of the floor

• predictor polls Prometheus every 5s: observed arrival rate λ and mean service time E[S]. It forecasts λ 30s ahead (one pod-startup horizon) by linear regression over the last 2 minutes, then sizes an M/M/c system for max(λ_now, λ_forecast) against a p95 latency SLO and a utilization ceiling.
• scaler reads the prediction and raises the HPA's minReplicas to the predicted count. Scale-ups apply immediately; scale-downs hold for 60s. The HPA keeps reacting to CPU above the floor, so predictive and reactive control compose rather than fight. In reactive mode the floor is pinned at the base value — that is the comparison baseline.

Why M/M/c

The workload is c parallel pods, each handling one CPU-bound request at a time (Poisson-ish arrivals from many independent VUs, ~exponential-enough service). M/M/c gives a closed-form answer to exactly the capacity question: given arrival rate λ and service rate μ, how many servers c keep waiting time acceptable? Erlang-C yields the queueing probability

P_wait = ErlangC(c, a),  a = λ/μ

and because the M/M/c waiting tail is exponential,

P(W > t) = P_wait · e^-(cμ-λ)t   ⇒   p95 wait = ln(P_wait/0.05)/(cμ-λ)

the predictor picks the smallest c with p95_wait + E[S] ≤ SLO and λ/(cμ) ≤ 0.75. This captures the non-linearity that reactive CPU scaling misses: near saturation, latency explodes with queueing long before average CPU reaches 100%.

Two deliberate approximations:

1. Service here is a fixed CPU burn (closer to M/D/c). M/M/c assumes more variable service, so it over-provisions slightly — the safe direction.
2. Under saturation the measured service time inflates (requests queue on the pod's CPU), which inflates the model's replica ask. Also conservative: the model scales out hard exactly when the system is drowning.

The regression forecast is what buys lead time: on a ramp, the projected λ crosses capacity thresholds ~30s before the observed λ does, which is roughly what pod startup + metrics propagation costs.

Quick start

Requires: docker, kind, kubectl, helm, k6, go.

make cluster-up          # kind cluster + metrics-server (HPA needs it)
make deploy-echo         # sample workload + HPA (NodePort 30080)
make deploy-monitoring   # Prometheus :30090, Grafana :30030 (dashboard provisioned)
make deploy-predictor    # M/M/c predictor
make deploy-scaler       # preemptive scaler (predictive mode)
make build-cli

Running experiments

./bin/capsim run spike            # reactive baseline, then predictive, side by side
./bin/capsim run diurnal -mode predictive
./bin/capsim run ramp -mode reactive

Scenarios: steady, spike (12× jump), ramp (linear climb), diurnal (compressed day cycle). Each run samples Prometheus every 5s and writes reports/<scenario>-<mode>-<timestamp>.csv with rps, p95, error ratio, predicted replicas, applied floor, HPA desired, and ready replicas, then prints a summary. Watch it live in Grafana: http://localhost:30030 (dashboard "capsim — predicted vs actual").

Mode can also be flipped manually: make mode-reactive / make mode-predictive.

Comparing predictive vs reactive

capsim run <scenario> -mode both produces the comparison: it runs the reactive baseline (HPA alone), waits for scale-down, runs the same scenario predictively, and prints both summaries. Render a chart from the two CSVs:

python3 scripts/plot_report.py spike \
  reports/spike-reactive-<ts>.csv reports/spike-predictive-<ts>.csv \
  -o docs/img/spike-comparison.png

Getting a valid experiment

Three design traps will silently produce a meaningless comparison — this repo walked into all three, and the current setup exists because of them:

1. Fixed CPU work, not wall-clock work. The /work endpoint burns a calibrated number of CPU iterations. An earlier version spun until a wall-clock deadline — under contention each request silently did less work, so the service could never be overloaded and every scaling policy looked fine.
2. Open-model load. The k6 scenarios use ramping-arrival-rate, not VU loops. Closed-loop VUs slow down when latency rises, shedding exactly the overload the experiment is supposed to measure (and violating M/M/c's assumption that arrivals are independent of service).
3. Fit the experiment to the machine. Peak scenario demand (~6 cores) must fit inside the kind node's CPU budget, or the host — not the scaler — sets the latency ceiling. Same for the HPA baseline: CPU request must equal the limit (Guaranteed QoS), otherwise utilization-of-request reads 500% and the reactive baseline scales unrealistically fast.

On the spike scenario the mechanism to look for in the CSVs: the predictive run's floor jumps to the full M/M/c ask within ~10s of the rate rising (one predictor tick after the 30s rate window catches the jump), while the reactive HPA walks up in multiplicative steps per 15s metric cycle. The histogram also inflates the measured service time during the crunch, which pushes the M/M/c ask harder — conservative exactly when it should be.

Layout

• services/echo/ — workload: /work?ms=N burns N ms of CPU per request (the service-time knob), Prometheus metrics
• predictor/ — Erlang-C sizing, OLS rate forecast, thin Prometheus client
• scaler/ — HPA floor patcher (minimal raw REST client, RBAC: get/patch one HPA)
• loadgen/k6/ — scenario definitions
• deploy/ — kind config, manifests, helm values
• dashboards/ — Grafana dashboard JSON (provisioned via configmap)
• cmd/ — capsim (experiment driver), predictor, scaler daemons