Skip to content

GuideLLM v0.8.0

Choose a tag to compare

View all tags
@dbutenhof dbutenhof released this 30 Sep 19:49
· 21 commits to main since this release
Immutable release. Only release title and notes can be modified.
v0.8.0
4545d7c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Overview

GuideLLM v0.8.0 is a major release with several significant new features and improvements that enhance usability.

To get started, install with:

pip install guidellm[recommended]==0.8.0

Or from source with:

pip install 'guidellm[recommended] @ git+https://github.com/vllm-project/guidellm.git'@v0.8.0

Highlights

WEKA and OTEL replay

GuideLLM can now replay WEKA and OTEL trace datasets, including subagent branches and tool calling. See https://vllm-project.github.io/guidellm/0.8.0/guides/trace_replay/

Backend support

  • websockets backend now tracks round-trip metrics
  • vLLM in-process batch processing

Configuration (scenario) files

The built-in configuration files (scenarios) have been replaced with more modern and relevant configurations.

  • rhaiis/concurrent-100k-1k-equal.json -- very large context
  • rhaiis/concurrent-1k-1k-equal.json -- prefill/decode balanced
  • rhaiis/concurrent-1k-1k-multiturn.json --- Multi-turn with prefix caching
  • rhaiis/concurrent-2ki-128-equal.json -- Prefill-heavy
  • rhaiis/concurrent-512-2ki-norm.json -- Generation-heavy
  • rhaiis/concurrent-8k-1k-equal.json -- Large context
  • rhaiis/concurrent-8k-800-norm.json -- Large context with sequence length

Goodput

GuideLLM can now measure and report performance against a target SLO. The SLO is set using `--metrics '{"kind":"generative","slo":{"ttft_ms":200,"tpot_ms":50}}'. The "Goodput Attainment" will be reported in the console output and in the JSON/CSV benchmark files by percentage and per-second mean.

The Goodput profile will search for the highest load that can meet the SLO specified by --metrics parameters at some target attainment level. For example,

$ guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=goodput,target_attainment=0.99 \
  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
  --metrics '{"kind":"generative","slo":{"ttft_ms":200,"tpot_ms":50}}' \
  --constraint kind=max_duration,seconds=120

Minimum requests constraint

The new min_requests acts like max_requests in every way except that it continues to queue and run new requests after its request count is reached and only stops the benchmark when the count number of requests fully complete. This gives it the advantage of max_seconds where the full request rate is maintained at the end of the benchmark as opposed to max_requests which tails off.

Chinese documentation

Contributor @tangming1996 has begun a project to translate GuideLLM documentation into Chinese, beginning with the "Getting started" section. See https://vllm-project.github.io/guidellm/0.8.0/zh/

New HTML output

New HTML output (--output kind=html) is a single file that's more easily transportable.

Using SQL datasets

You can now use a SQLite3 database as a data source by specifying the SQLAlchemy URI and a SQL query string: for example, --data kind=db_file,uri=sqlite:///prompts.db,load_kwargs.sql="SELECT text FROM samples"

Change Log

Features

Bug fixes

Documentation

CI & environment

Dependency updates

  • Bump aiohttp from 3.14.1 to 3.14.3 by @dependabot[bot] in #995
  • Bump h2 from 4.3.0 to 4.4.1 by @dependabot[bot] in #1011
  • [GitHub Actions]: Bump redhat-actions/push-to-registry from 2.8 to 3 by @dependabot[bot] in #1037
  • [GitHub Actions]: Bump redhat-actions/buildah-build from 2.13 to 3 by @dependabot[bot] in #1036
  • [GitHub Actions]: Bump redhat-actions/buildah-build from 3.0.2 to 3.1.0 by @dependabot[bot] in #1062
  • build(deps): Bump transformers from 5.5.0 to 5.10.1 by @dependabot[bot] in #1076
  • [GitHub Actions]: Bump docker/setup-qemu-action from 4.2.0 to 4.3.0 by @dependabot[bot] in #1129
  • build(deps): Bump anyio from 4.12.0 to 4.14.2 by @dependabot[bot] in #1162
  • Bump pygments v2.19.2 -> v2.21.0 by @sjmonson in #1174
  • [GitHub Actions]: Bump docker/setup-qemu-action from 4.3.0 to 4.4.0 by @dependabot[bot] in #1178

New Contributors

Full Changelog: v0.7.4...v0.8.0

Contributors

maryamtahhan, sjmonson, and 24 other contributors
Assets 3
Loading