Evaluation Benchmark Plan for vLLM v0.15.1

[susiejojo]

Data Collection Matrix

vLLM version: v0.15.1
Load level: Fixed sub-saturation rate per experiment
Status (2026-03-31): 39 active experiments

✅ Dataset Update (2026-03-31): Statistics updated based on actual collected data from vllm_data/ground_truth/experiments.json.

Dimensions

Models (6)

Model	Type	Params	Min TP (H100)
meta-llama/Llama-3.1-8B-Instruct	Dense	8B	TP1
Qwen/Qwen3-14B	Dense	14B	TP1
codellama/CodeLlama-34b-Instruct-hf	Dense	34B	TP2
meta-llama/Llama-2-70b-hf	Dense	70B	TP4
mistralai/Mixtral-8x7B-v0.1	MoE	47B (13B active)	TP2
mistralai/Mixtral-8x22B-Instruct-v0.1	MoE	141B (39B active)	TP8

Hardware (3)

GPU	VRAM	Interconnect	FP8 native
L40S	48 GB	PCIe	Yes
A100-80GB SXM	80 GB	NVLink	No (emulated)
H100-80GB SXM	80 GB	NVLink	Yes

Workloads (6)

General, General-Lite, Codegen, Roleplay, Reasoning, Reasoning-Lite

Note: General-Lite and Reasoning-Lite use reduced request rates (6 RPS and 4 RPS respectively) to avoid saturation on slower hardware or configurations.

vLLM Config Knobs (5)

Knob	Sweep values	Default
max_num_batched_tokens	1024, 2048, 8192	2048
cpu_offloading	Disabled, Enabled	Disabled
gpu_memory_utilization	0.9, 0.95	0.9
TP	Default + 1 higher (model-dependent)	Min viable
DP (--enable-expert-parallel)	1, 2 (MoE only; EP = TP×DP)	1

---

Dataset Statistics

Overall

• Total active experiments: 39
• Hardware distribution: H100 (33), A100-80GB (4), L40S (2)
• Configuration split: 24 default configs, 17 varied configs

By Model

Model	Experiments	Coverage
Llama-3.1-8b	11	Full: 3 workloads × H100 + config sweeps + hardware variation
Mixtral-8x7B	10	Full: 3 workloads × H100 + config sweeps + EP sweep
Qwen3-14B	6	3 workloads × H100 + hardware variation
Codellama-34b	4	2 workloads × H100 + hardware variation
Llama-2-70b	4	2 workloads × H100 + hardware variation
Mixtral-8x22B	4	3 workloads + reasoning-lite × H100

By Hardware

Hardware	Experiments	Models Covered
H100	33	All 6 models
A100-80GB	4	4 models (Llama-3.1-8b, Qwen3-14B, Codellama-34b, Llama-2-70b)
L40S	2	2 models (Llama-3.1-8b, Qwen3-14B)

By Workload

Workload	Experiments	Rate (RPS)
general	15	20
general-lite	10	6
codegen	6	20
roleplay	6	20
reasoning	1	4
reasoning-lite	1	4

Configuration Coverage

• CPU offload: 3 experiments (Enabled vs Disabled)
• Max batched tokens: 4 experiments (1024, 2048, 8192)
• Tensor parallelism: 2 experiments (TP sweep)
• GPU memory: 2 experiments (0.9 vs 0.95)
• Expert parallelism: 1 experiment (DP=2, EP=4)
• Default configs: 25 experiments

---

Complete Experiment Matrix

#	Model	Precision	HW	Workload	mbt	cpu_offload	gpu_mem	TP	DP	Notes
2	Codellama-34b	FP16	H100	codegen	2048	Enabled	0.9	2	—
3	Codellama-34b	FP16	H100	roleplay	2048	Enabled	0.9	2	—
6	Llama-2-70b	FP16	H100	codegen	2048	Enabled	0.9	4	—
7	Llama-2-70b	FP16	H100	roleplay	2048	Enabled	0.9	4	—
9	Mixtral-8x7B	FP16	H100	general	2048	Enabled	0.9	2	1
10	Mixtral-8x7B	FP16	H100	codegen	2048	Enabled	0.9	2	1
11	Mixtral-8x7B	FP16	H100	roleplay	2048	Enabled	0.9	2	1
13	Qwen3-14B	FP16	H100	general	2048	Disabled	0.9	1	—
14	Qwen3-14B	FP16	H100	codegen	2048	Disabled	0.9	1	—
15	Qwen3-14B	FP16	H100	roleplay	2048	Disabled	0.9	1	—
16	Llama-3.1-8b	FP16	H100	general	2048	Disabled	0.9	1	—
18	Llama-3.1-8b	FP16	H100	codegen	2048	Disabled	0.9	1	—
19	Llama-3.1-8b	FP16	H100	roleplay	2048	Disabled	0.9	1	—
22	Llama-3.1-8b	FP16	H100	general	1024	Disabled	0.9	1	—	mbt sweep
23	Llama-3.1-8b	FP16	H100	general	8192	Disabled	0.9	1	—	mbt sweep
24	Llama-3.1-8b	FP16	H100	general	2048	Enabled	0.9	1	—	cpu_offload sweep
25	Llama-3.1-8b	FP16	H100	general	2048	Disabled	0.95	1	—	gpu_mem sweep
26	Llama-3.1-8b	FP16	H100	general	2048	Disabled	0.9	2	—	TP sweep (2x default)
27	Mixtral-8x7B	FP16	H100	general-lite	1024	Disabled	0.9	2	1	mbt sweep
28	Mixtral-8x7B	FP16	H100	general	8192	Disabled	0.9	2	1	mbt sweep
29	Mixtral-8x7B	FP16	H100	general	2048	Enabled	0.9	2	1	cpu_offload sweep
30	Mixtral-8x7B	FP16	H100	general	2048	Disabled	0.95	2	1	gpu_mem sweep
31	Mixtral-8x7B	FP16	H100	general	2048	Disabled	0.9	4	1	TP sweep (2x default)
32	Mixtral-8x7B	FP16	H100	general	2048	Disabled	0.9	2	2	EP sweep: --enable-expert-parallel, EP=4 (4 GPUs)
36	Qwen3-14B	FP16	A100-80GB	general-lite	2048	Disabled	0.9	1	—	General-Lite (6 rps) for safe rate; decode 19.039 ms/token
38	Llama-3.1-8b	FP16	A100-80GB	general-lite	2048	Disabled	0.9	1	—	General-Lite (6 rps) for safe rate
40	Codellama-34b	FP16	A100-80GB	general-lite	2048	Disabled	0.9	2	—	General-Lite (6 rps) for safe rate; decode 23.598 ms/token
41	Llama-2-70b	FP16	A100-80GB	general-lite	2048	Disabled	0.9	4	—	General-Lite (6 rps) for safe rate; decode 28.783 ms/token
46	Llama-3.1-8b	FP16	H100	reasoning	2048	Disabled	0.9	1	—
49	Mixtral-8x22B	FP16	H100	general	2048	Disabled	0.9	8	1
50	Mixtral-8x22B	FP16	H100	codegen	2048	Disabled	0.9	8	1
51	Mixtral-8x22B	FP16	H100	roleplay	2048	Disabled	0.9	8	1
53	Mixtral-8x22B	FP16	H100	reasoning-lite	2048	Disabled	0.9	8	1
54	Qwen3-14B	FP16	L40S	general-lite	2048	Disabled	0.9	1	—
55	Llama-3.1-8b	FP16	L40S	general-lite	2048	Disabled	0.9	1	—
56	Mixtral-8x7B	FP16	H100	general	2048	Disabled	0.9	2	1	Default-config baseline
57	Codellama-34b	FP16	H100	general-lite	2048	Disabled	0.9	2	—	Default-config baseline; General-Lite (6 rps) for safe rate
58	Llama-2-70b	FP16	H100	general-lite	2048	Disabled	0.9	4	—	Default-config baseline; General-Lite (6 rps) for safe rate
59	Qwen3-14B	FP16	H100	general-lite	2048	Disabled	0.9	1	—	Matches A100/L40S workload for hardware comparison

---

Dataset Quality

Saturation Analysis

• ✅ All 39 experiments operated successfully with zero failed requests
• ✅ All experiments are healthy and suitable for simulator validation
• TTFT p99 ranges from 0.06s to 5.02s across all experiments
• Median TTFT is healthy (< 0.2s) across all experiments, including those with elevated p99

Note: Exp 54 (Qwen3-14B/L40S/general-lite) shows TTFT p99 of 5.02s with median of 0.185s. While the tail latency is elevated, the experiment successfully completed with no failures and provides valuable data on hardware portability to L40S GPUs.

Strengths

• ✅ Zero failed requests across all experiments
• ✅ 100% successful operation - all 39 experiments are healthy
• ✅ Strong H100 coverage (33 experiments)
• ✅ Balanced default vs varied configs (22 default, 17 varied)
• ✅ 6 production-scale models spanning 8B to 141B parameters
• ✅ Both dense and MoE architectures represented

Limitations

• ⚠️ Limited A100/L40S coverage (6 experiments = 14.6%)
• ⚠️ Minimal reasoning workload data (2 experiments)
• ⚠️ No multi-node configurations

---

Notes

• FP8 on A100-80GB: A100 lacks native FP8 compute — weights stored in FP8 but upcasted for computation. Expect slower inference vs H100.
• L40S single-node constraint: L40S nodes have 2 GPUs per node. Calibration uses single-node deployments (max TP=2). Usable VRAM at TP=2: 86.4 GB.
• Expert parallelism in vLLM: EP is not a standalone flag. Set --tensor-parallel-size, --data-parallel-size, and --enable-expert-parallel. EP degree = TP × DP. MoE expert weights are distributed across all TP×DP GPUs; dense/attention layers use TP sharding within each DP replica.
• Config sweep model selection: 2 models — Llama-3.1-8b (small dense) and Mixtral-8x7B (medium MoE). Config sensitivity doesn't require large models.
• Reasoning workload limited: Reasoning workloads (934+1448 tokens, 4 rps) are difficult to run reliably on stock vLLM v0.15.1. Only 2 experiments included.
• General-Lite for slower configs: Several experiments use General-Lite (6 RPS) instead of General (20 RPS) to avoid saturation on slower hardware (A100, L40S) or with cpu_offload enabled.

[susiejojo]

Rate Calibration (Pre-Experiment Step)

The workload profiles use fixed rates across all models (General: 8→20 rps, Codegen: 5→10 rps, Roleplay: 6 rps, Reasoning: 4 rps). Larger models on weaker hardware may saturate at these rates. A quick calibration step ensures sub-saturation for every experiment.

Procedure: Before running the full matrix, send one streaming request (~500 input tokens, 250 max output tokens, temperature=0) per unique model×hardware×TP config. Record wall-clock timestamps for each SSE token event, skip the first token (includes TTFT/prefill latency), and compute:

decode_ms_per_token = (t_last - t_second) / (n - 2) * 1000

From this, derive the safe rate:

safe_rps = 0.5 × min(R_mbt, R_seq)

where:
  R_mbt = mbt × 1000 / (avg_total_tokens × decode_ms_per_token)
  R_seq = max_num_seqs × 1000 / (avg_output_tokens × decode_ms_per_token)

The 0.5 multiplier provides a 50% safety margin for variance, scheduling overhead, and prefill/decode interleaving. R_mbt captures the throughput limit from max_num_batched_tokens; R_seq captures the limit from max_num_seqs (128). For each experiment, set the workload rate to min(profile_rate, safe_rps).

Why decode_ms_per_token is workload-independent: For sequences under 4K tokens, decode latency is dominated by model weight loading (GPU memory bandwidth), not KV-cache attention. This makes decode_ms_per_token essentially independent of workload (input/output token counts) and vLLM knobs like max_num_batched_tokens, cpu_offload_gb, or gpu_memory_utilization. Those parameters affect batch capacity but not per-token decode speed. Workload differences are handled analytically through avg_total_tokens and avg_output_tokens.