Skip to content

Conversation

@git-jxj
Copy link
Contributor

@git-jxj git-jxj commented Oct 23, 2025

Summary

This PR fixes two critical bugs that cause the guidellm benchmark command to crash when using the --output-formats csv option. These changes ensure that CSV reports are generated reliably and correctly.

Details

  1. Fix AttributeError**:
    In src/guidellm/benchmark/output.py, a hasattr check has been added to the _get_benchmark_status_metrics_stats function.
    This check ensures that the code only attempts to access status attributes (e.g., successful) on metric objects that actually possess them (like StatusDistributionSummary). It safely skips composite metrics such as GenerativeTextMetricsSummary, thus preventing the AttributeError.

  2. Fix KeyError**:
    The data flow for benchmark compilation has been refactored to ensure the original data configuration (args.data) is correctly propagated.
    The method signatures for Benchmarker.run (in benchmarker.py) and GenerativeBenchmark.compile (in schemas.py) were updated to accept a data parameter.
    args.data is now passed in during the benchmarker.run call within benchmark_generative_text (in entrypoints.py).
    As a result, the benchmark.benchmarker.requests field in the final GenerativeBenchmark object now directly stores the raw data configuration, which resolves the KeyError in output.py.

Test Plan

guidellm benchmark
--target "http://x.x.x.x:x/"
--processor "Qwen/Qwen3-8B-FP8"
--rate-type "concurrent"
--rate 1
--max-requests 10
--data "prompt_tokens=32,output_tokens=32,samples=1"
--output-formats csv

Related Issues

#434

  • Resolves #

  • "I certify that all code in this PR is my own, except as noted below."

Use of AI

  • Includes AI-assisted code completion
  • Includes code generated by an AI application
  • Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

Signed-off-by: xinjun.jiang <xinjun.jiang@daocloud.io>
Copy link
Collaborator

@markurtz markurtz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@markurtz markurtz merged commit b05b0e7 into vllm-project:main Oct 23, 2025
11 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants