Benchmarks

Benchmarking Guide

Use a real LLM backend (--mock off) and test at least three repeated runs after model cache warm-up. Capture:

Measurement	How to collect
CPU / GPU type	nvidia-smi, torch device properties
Number of replicas	--devices values
Batch size	--batch-size
Input/output limits	--max-input-tokens, --max-new-tokens
Initialization latency	Model load value printed by parent
Run throughput	Rows/s and approximate output tokens/s
GPU peak memory	nvidia-smi or PyTorch profiling (not built in)
Per-worker imbalance	Worker i processed N rows

Compare cuda:0 and cuda:0 cuda:1 on the same dataset, and report both wall-clock time and model-loading overhead. Scaling is not guaranteed. The project has no automated benchmark suite or preverified multi-GPU results.