Repository navigation
Overview
GuideLLM v0.8.1 is a major release with several significant new features and improvements that enhance usability.
To get started, install with:
pip install guidellm[recommended]==0.8.1Or from source with:
pip install 'guidellm[recommended] @ git+https://github.com/vllm-project/guidellm.git'@v0.8.1Highlights
Specifying profile and constraint sub-benchmark parameters
It has been possible to specify sub-benchmark (strategy) parameters directly in the --profile body on the CLI or in a config file. This is not the intended operational mode, and may not be supported in the future. For example, '{"kind":"concurrent","streams":[1,2,3]}' to specify three concurrent strategies with 1, 2, and 3 request streams.
In v0.8.1, GuideLLM will issue a deprecation warning when this happens.
Instead, specify the profile kind and profile parameters on --profile and specify the strategy-specific parameters with --override profile.<name> <values>, like --override profile.streams 1,2,3. This improves clarity and avoids the need for error-prone serialized JSON on the command line.
"compile" benchmark status
In the live console updates, you can see each sub-benchmark progress through a series of states, "pending" to "warmup" to "active" to "cooldown", to "completed". Due to post processing of statistics, there can be a significant delay between completing "cooldown" and the final "completed". This adds a new "compile" state to represent the compilation of benchmark statistics. Normally it will be short.
Knee detection
This new profile generates a series of concurrent benchmark strategies, somewhat like the sweep profile, focused on finding a maximum (knee) in the output token throughput. See the documentation for more information at https://vllm-project.github.io/guidellm/0.8.1/guides/knee_detection/
Metrics-based warning system
Complicated or large synthetic datasets, especially large WEKA datasets, can take substantial time to prepare for benchmarking. In extreme cases, benchmarking may begin with insufficient data for the configured concurrency, and measured throughput may be unexpectedly low.
This new system provides a mechanism to generate warnings based on runtime statistics. Default rules provide warnings (on the console and in the JSON output) when
- The mean request generation delay is more than 1% of the mean time to first token (TTFT);
- When the 95th percentile of conversation first-turn dispatch delay is greater than 0.2 seconds later than scheduled;
- When a trace dataset has not been fully loaded
See https://vllm-project.github.io/guidellm/0.8.1/guides/warnings for more information on building and specifying your own warning rules.
Change Log
Features
- Add deprecations for list based rate/constraint by @sjmonson in #1213
- Add a new benchmark status to indicate post-processing state by @sjmonson in #1236
- Feature: knee detection with adaptive concurrency runs by @Rehan164 in #1199
- Added metrics-based configurable warning system. by @jaredoconnell in #1246
Bug fixes
- Merge close timestamps in one pass in rate and concurrency stats by @himanshu1573 in #1212
- Start request timing when the request is sent by @QHarshil in #1240
- fix: encode NumPy and PyTorch media arrays by @git-jxj in #1241
- fix: console log omits prime-parameter on sub-benchmarks by @dfeddema in #1260
Documentation
- docs: load local JSONL examples with the file deserializer by @git-jxj in #1214
- docs: fix the benchmark report reload example by @git-jxj in #1215
- docs: fix the reasoning history backend example by @git-jxj in #1216
CI & environment
Dependency updates
- build(deps): Bump urllib3 from 2.7.0 to 2.8.0 by @dependabot[bot] in #1209
- build(deps): Bump virtualenv from 21.5.1 to 21.7.13 by @dependabot[bot] in #1252
- build(deps): Bump multidict from 6.7.0 to 6.9.1 by @dependabot[bot] in #1255
- build(deps): Bump fsspec from 2025.10.0 to 2026.6.0 by @dependabot[bot] in #1256
New Contributors
Full Changelog: v0.8.0...v0.8.1
