Skip to content

[Refactor] guidellm.backend.openai performance improvements #340

@markurtz

Description

@markurtz

Improve performance on top of the current rewrite to remove overheads such as too many async updates, too many decodes, and n^2 string chaining for gathering tokens

Metadata

Metadata

Labels

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions