AIPerf — Release 0.12.0
Summary
AIPerf 0.12.0 is one of the largest releases to date, spanning agentic workloads, adaptive scaling, streaming metrics, and broad accuracy/analysis tooling. The headline is AgentX v1.0 — a new agentic replay stack covering WEKA, agentic replay traces, and DAG-shaped workloads. A new Anthropic Messages API endpoint (/v1/messages) joins the endpoint catalog, and a new accumulator engine wires streaming metrics through the records pipeline. Adaptive scaling arrives via single-run adaptive scaling for Dynamo Test Framework, adaptive scale variables for controls and SLA metrics, and multiple warmup and profiling phases in a single run. Interactive tooling grows with an aiperf chat command, an aiperf analyze CLI plus swim-lane viewer, and a turn-messages viewer with a vendored fzstd decompressor. Accuracy gains a MMLU-Pro benchmark and reasoning-model MMLU fixes; a new spec-decode per-request acceptance record plus vLLM adapter lands, along with a client-observed Decode Duration metric for streamed generation. Dataset coverage expands with Exgentic v1/v2, baseten_trace faithful trace replay, and a verbatim-payload replay fast path backed by a content-addressed mmap dataset cache. Ops gets dynamic QPS control via JSON, optional session-affinity and X-Dynamo-Session-ID headers for external routers, and an OpenAI Responses context reset + structured payload extraction. This release drops Python 3.10 support and adds Windows-on-ARM install/run support. Extensive fixes cover accuracy graders (LCB, MMLU, GSM8K path), tokenizer aliases (DeepSeek-V3.2-Exp), video synthesis, OpenAI Responses token double-counting, timing, dataset validation, and orchestrator-v1 gating. Late cherry-picks to release/0.12.0 patch ffmpeg CVE-2026-8461, batch-encode ShareGPT to fix a 300s configure timeout, keep shared prefix blocks full for synthesized traces, and fix the multi-run detailed aggregation JSONL fallback.
Breaking changes
- Minimum Python is now 3.11. Python 3.10 support has been dropped (#1107).
Key highlights
- Agentic workloads: AgentX v1.0 — WEKA, agentic replay, and DAG-shaped workloads (#1165).
- New endpoint: Anthropic Messages API /v1/messages (#731).
- Metrics: Accumulator engine with records-pipeline wiring (#1102); client-observed Decode Duration metric for streamed generation (#1147).
- Adaptive scaling: Single-run adaptive scaling for the Dynamo Test Framework (#1058); adaptive scale variables for controls and SLA metrics (#1074); multiple warmup and profiling phases in a single run (#1150).
- Accuracy: MMLU-Pro benchmark + reasoning-model MMLU fixes (#1140); grade reasoning models on answer content only, not CoT preamble (#1161).
- Spec-decode: Per-request acceptance record + vLLM adapter (#1167).
- Analysis tooling: aiperf analyze CLI + swim-lane viewer (#1116); turn-messages viewer with vendored fzstd decompressor (#1115); interactive aiperf chat command (#1076).
- Datasets: Exgentic v1/v2 replay (#1064, #1078); faithful trace replay via baseten_trace (#1129); verbatim payload replay fast path + content-addressed mmap dataset cache (#1104).
- Routing & control: Dynamic QPS control via JSON (#1067); opt-in X-Dynamo-Session-ID header from correlation ID (#1151); opt-in session-affinity headers for external routers (#1094).
- Platform: Drops Python 3.10 (#1107); adds Windows-on-ARM install/run (#1068).
Features and enhancements
Agentic workloads and DAG
- AgentX v1.0: WEKA, agentic replay, and DAG-shaped workloads — #1165 (@ajcasagrande).
Endpoints and routing
- Anthropic Messages API endpoint /v1/messages — #731 (@ajcasagrande).
- Image UUID cache reuse for image endpoints — #869 (@furionw).
- OpenAI Responses: context reset and structured payload extraction — #1097 (@ajcasagrande).
- Opt-in X-Dynamo-Session-ID header from correlation ID — #1151 (@ajcasagrande).
- Opt-in session-affinity headers for external routers — #1094 (@ajcasagrande).
Metrics and telemetry
- Accumulator engine with records-pipeline wiring — #1102 (@ajcasagrande).
- Client-observed Decode Duration metric for streamed generation — #1147 (@sxuff).
- Server-metrics: warmup-phase export, realtime robustness, and per-worker logging — #1096 (@ajcasagrande).
- Namespace NVIDIA telemetry metrics (refactor) — #976 (@FrankD412).
Adaptive scaling and control
- Single-run Adaptive Scaling for Dynamo Test Framework — #1058 (@ilana-n).
- Adaptive scale variables for controls and SLA metrics — #1074 (@ilana-n).
- Multiple warmup and profiling phases in a single run — #1150 (@ilana-n).
- Dynamic QPS Control via JSON — #1067 (@kfir4444).
Accuracy benchmarks
- MMLU-Pro benchmark + fix MMLU for reasoning models — #1140 (@ajcasagrande).
Spec-decode
- Per-request acceptance record + vLLM adapter — #1167 (@matthewkotila).
CLI and analysis
- Interactive aiperf chat command for quick perf sanity checks — #1076 (@matthewkotila).
- Swim-lane viewer + aiperf analyze CLI — #1116 (@ajcasagrande).
- Turn-messages viewer with vendored fzstd decompressor — #1115 (@ajcasagrande).
Datasets and traces
- Exgentic public dataset replay — #1064 (@ishandhanani).
- Exgentic v2 replay — #1078 (@ishandhanani).
- Faithful trace replay using baseten_trace — #1129 (@ShounakRay).
- Verbatim payload replay fast path + content-addressed mmap dataset cache — #1104 (@ajcasagrande).
- Hoist leading system turn into conversation system prompt — #1061 (@hatemfaheem).
Exporters, timing, and analysis pipeline
- Console export artifact, width pinning, and metric gating — #1100 (@ajcasagrande).
- Periodic time-based flush for buffered JSONL writers — #1099 (@ajcasagrande).
- Warmup progress heartbeat and empty-drain guard — #1121 (@ajcasagrande).
Mock server
- Optional mock server inference auth — #986 (@ajcasagrande).
Windows support
- Install + run AIPerf on Windows-on-ARM (markers, lazy native imports, win-arm CI) — #1068 (@lkomali).
Bug fixes and robustness
| Change | PR |
|---|---|
| Accuracy: update GSM8K dataset name to openai/gsm8k | #1075 |
| Accuracy: allow LCB grader to spawn from daemon record processor | #1082 |
| Accuracy: preflight grader deps so missing lighteval fails cleanly | #1083 |
| Accuracy: grade LCB codegen in an out-of-process worker (#1145) | #1175 |
| Accuracy: grade reasoning models on answer content only, not CoT preamble | #1161 |
| Config: honor CLI --server-metrics format overrides | #1073 |
| Config: read the v2 phase field 'rate' in descriptors and exporters | #1093 |
| Config: use readable per-run dir names for nested scenario sweeps | #1080 |
| Records: gate record processing on dataset configuration | #1079 |
| W&B: prevent argv metadata credential leaks | #1081 |
| Server metrics: parse OpenMetrics exposition for the vLLM Rust frontend | #1060 |
| HTTP: fallback when OS rejects large HTTP socket buffers | #994 |
| UI: render timeslice latency charts correctly | #1105 |
| Video: write synthetic WebM to a seekable file so it carries container duration | #1149 |
| Dataset: restore --isl-block-size override for trace datasets | #1159 |
| Dataset: apply --synthesis-max-osl cap in loader overrides | #1122 |
| Dataset: harden inputs.json and raw-payload loader input validation | #1120 |
| Validators: harden orchestrator-v1 gate against malformed DAG datasets | #1103 |
| Endpoints: count OpenAI Responses streaming output tokens once, not twice | #1134 |
| Tokenizer: register deepseek_v32 config alias for DeepSeek-V3.2-Exp | #1069 |
| Config: restore profile image source CLI flag | #975 |
| Timing: carry has_forks onto continuation turns | #1123 |
| Controller: reap orphaned service processes; SIGUSR1 stack dumps | #1095 |
| CLI runner: fix export outputs JSON CLI flag | #1138 |
| Docs: stabilize generated cli-options across cyclopts default rendering | #1148 |
| CI: repair windows-latest build (concurrency assert + UTF-8 plugin-artifact I/O) | #1043 |
Cherry-picks to release/0.12.0
- Security: bump ffmpeg to 8.1.2 for CVE-2026-8461 — #1214 (@nv-nmailhot).
- Synthesize: keep shared prefix blocks full so synthesized traces replay (cherry-pick of #1208) — #1219 (@lkomali).
- Dataset: batch-encode ShareGPT to fix 300s configuration timeout — #1216 (@FrankD412).
- CLI runner: fix multi-run detailed aggregation JSONL fallback — #1217 (@ilana-n).
- Adaptive scale: harden edge cases — #1171 (@ilana-n).
Performance
- Parse completed responses once in workers — #1172 (@jthomson04).
- Fast-path single-field SSE data — #1173 (@jthomson04).
Refactors
- Records: remove legacy results processors; route by record type — #1144 (@ajcasagrande).
- Models: move turns off the record wire; add root_correlation_id — #1118 (@ajcasagrande).
- Cleanup: remove dead environment knobs and MessageType members — #1124 (@ajcasagrande).
- Telemetry: namespace NVIDIA telemetry metrics — #976 (@FrankD412).
Documentation
Dependencies, chore, and tooling
- Bump aiperf version to 0.12.0 — #1193 (@nv-nmailhot).
- Update attributions for 0.11.0 — #1128 (@nv-nmailhot).
- Bump distroless base image to 3.13-v4.0.8-dev — #1110 (@nv-nmailhot).
- Fern: upgrade to 5.57.0 and switch to multi-source — #1156 (@debermudez).
- Fern: bump 5.57.0 → 5.75.8 — #1158 (@aschilling-nv).
- Replace --tokenizer gpt2 with --tokenizer builtin in docs and tests — #1157 (@ajcasagrande).
- Drop file/function-size ergonomics checks and BLE001 / PLR0915 / PLR0912 — #1091 (@ajcasagrande).
- Hardcode AIPerf team in linear-issue skill — #1160 (@debermudez).
Testing and CI
- Fix two windows/CI flake families in tests — #1108 (@ajcasagrande).
- Register the looptime pytest marker — #1098 (@ajcasagrande).
- Stabilize flaky user-centric per-user-gap timing test — #1114 (@matthewkotila).
- Stop _init_worker tests from leaking a raised root log level — #1126 (@matthewkotila).
- Limit pytest auto workers by CPU capacity — #980 (@ajcasagrande).
- Raise unit workflow timeout — #1139 (@ajcasagrande).
- Add v2 regression guards and supersession markers — #1119 (@ajcasagrande).
- Restrict Velonix workflows to trusted branch pushes — #1177 (@saturley-hall).
- Fake amdsmi bindings to exercise AMD telemetry on non-AMD hosts — #1014 (@FrankD412).
- Guard unregistered accuracy tags on the display path — #1176 (@debermudez).
New contributors
- @ishandhanani — first contribution in #1064.
- @majunze2001 — first contribution in #1060.
- @sqjian — first contribution in #1105.
- @ShounakRay — first contribution in #1129.
- @sxuff — first contribution in #1147.
- @kfir4444 — first contribution in #1067.
- @aschilling-nv — first contribution in #1158.
- @waynel96 — first contribution in #1069.
- @hatemfaheem — first contribution in #1061.
- @jthomson04 — first contribution in #1172.
Full changelog
Full changelog: v0.11.0…v0.12.0