Event-driven benchmark of dynamic batch size controllers for LLM serving, measuring the throughput vs TTFT SLO trade-off under variable load.
Author: Joao Felipe De Souza Year: 2026
GPU serving systems typically use a fixed batch size chosen at deployment time. Under variable load this is suboptimal at both extremes: a small batch size leaves the GPU underutilized during peak demand, while a large batch size inflates tail latency during quiet periods.
This benchmark asks:
- Can a dynamic batch size controller adapt in real time to balance GPU utilization and per-request TTFT?
- Which controller type is most effective in which workload regime?
- Is there a controller that dominates all others on the Pareto frontier?
No single policy dominates all metrics. The policies form a Pareto frontier with two clear extremes:
- latency_target: lowest SLO violations, lowest throughput
- bang_bang: highest throughput, moderate SLO
In steady_heavy workloads, bang_bang achieves 3.10 req/s versus 1.65 req/s for fixed_16, matching queue_reactive while maintaining similar SLO rates.
In ramp_up workloads, bang_bang achieves 3.88 req/s versus 2.45 req/s for fixed_16, with lower p99 TTFT than queue_reactive.
Under the tested feedback dynamics, util_aware and hybrid reduce batch size aggressively and rarely recover it. Their average batch size stays near the configured minimum floor, behaving similarly to fixed_4. Utilization alone is an insufficient feedback signal.
In bursty workloads, latency_target achieves 3-4% SLO violation rate versus 65% for fixed_16. The cost is 0.63 vs 1.69 req/s throughput.
fixed_16 is dominated in throughput by queue_reactive and bang_bang in every workload, and dominated in SLO by latency_target in most.
fixed_4 / fixed_16 / fixed_64: Static baselines.
queue_reactive: Increases batch when queue grows, decreases when idle.
latency_target: Decreases batch when p99 TTFT exceeds target, increases when TTFT is well below target.
util_aware: Targets a GPU utilization band.
hybrid: Weighted combination of queue, latency, and utilization signals.
bang_bang: Aggressive scale-up under queue pressure or high TTFT, conservative scale-down only when both signals confirm easing. The asymmetry (fast up, slow down) is the key design choice.
- diurnal: sinusoidal 10x peak variation
- bursty_spikes: sudden 20x arrival spikes
- steady_heavy: constant high load near saturation
- ramp_up: linear increase from light to heavy over 30 min
- mixed_length: very short and very long requests mixed
- throughput_rps
- p50_ttft_ms, p95_ttft_ms, p99_ttft_ms
- slo_violation_rate
- avg_gpu_util
- avg_batch_size, min_batch_size, max_batch_size, batch_size_std
- avg_queue_depth, p95_queue_depth
From the project directory:
cd ~/dev/dynamic-batch-size-controller
source venv/bin/activate
python -u run.py
results/summary_v13.csv
results/pairwise_v13.csv
plots/
dynamic-batch-size-controller/
|-- src/
| |-- __init__.py
| |-- config.py
| |-- workload.py
| |-- controller.py
| |-- simulator.py
| |-- bench.py
| |-- analysis.py
|-- results/
|-- plots/
|-- run.py
|-- SUMMARY.txt
|-- DESIGN.md
|-- LICENSE
|-- README.md
|-- requirements.txt
|-- .gitignore
Dynamic batch size control resolves the throughput vs latency SLO trade-off that fixed batch sizes cannot address. A bang-bang controller with hysteresis is the best choice for throughput-optimized systems. A latency-targeting controller is best for systems with tight TTFT SLOs. Utilization-aware control collapses to minimum batch size under the tested feedback dynamics and requires adaptive recovery to be effective.
MIT License. See LICENSE.