Skip to content

Add remove_from_body to OpenAIHTTPBackend #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 24, 2025
Merged

Conversation

jackcook
Copy link
Contributor

This PR adds the remove_from_body parameter to OpenAIHTTPBackend, which allows users to remove keys from the body of requests made to the OpenAI server. This is needed for us to profile tokasaurus, which does not support the max_completion_tokens or stream parameters.

jackcook and others added 3 commits June 12, 2025 16:12

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

📦 Build Artifacts Available
The build artifacts (.whl and .tar.gz) have been successfully generated and are available for download: https://github.com/neuralmagic/guidellm/actions/runs/15807642173/artifacts/3378060351.
They will be retained for up to 30 days.

@markurtz markurtz requested review from Copilot and markurtz and removed request for Copilot June 24, 2025 12:51

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces the remove_from_body parameter to OpenAIHTTPBackend to allow users to specify keys that should be removed from the request payload, addressing compatibility with tokasaurus.

  • Added extra_body and remove_from_body parameters to the init method with corresponding docstring updates
  • Refactored payload generation in _completions_payload to incorporate the removal of specified keys
  • Renamed the _body helper to _extra_body for clarity in its purpose
Comments suppressed due to low confidence (1)

src/guidellm/backend/openai.py:466

  • [nitpick] Consider adding an inline comment here to explain the rationale for removing keys from the payload, providing context for future maintainers.
        if self.remove_from_body:

📦 Build Artifacts Available
The build artifacts (.whl and .tar.gz) have been successfully generated and are available for download: https://github.com/neuralmagic/guidellm/actions/runs/15850937815/artifacts/3392030461.
They will be retained for up to 30 days.

@markurtz markurtz merged commit 3c33c39 into vllm-project:main Jun 24, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants