support new api of chat/compilation api

[talregev]

Is your feature request related to a problem? Please describe.
guidellm cannot connect with transformers serve.
transformers serve use new openai api. it found on
http://localhost:8000/v1/chat/completions
with new msg body:

{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role":"user","content":"Who is the SpaceX CEO?"}],
    "max_tokens": 20,
    "stream": false
 }

Describe the solution you'd like
guidellm will be implemented the new openai api such as v1/chat/completions.

Describe alternatives you've considered
I consider to translate the msgs between of the shelf programs, but it another layer and it give more complexity.

Additional context

[tukwila]

i tried such api with msg body and it works; pls refer to PR: https://github.com/vllm-project/guidellm/pull/328/files

[talregev]

guidellm doesn't work with the new api.
I tried guidellm v0.3.0.

Here the setup. I am using cpu, but you don't have to.

Setup transformers server:
Extra steps that don't mention here, is to get access to the meta-llama/Llama-3.2-1B-Instruct model.
Also give your hugging face token to transformers to download the model.

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers[serving]
  transformers serve \
    --device cpu \
    --host 0.0.0.0 \
    --port 8000 \
    --force-model meta-llama/Llama-3.2-1B-Instruct &

guidellm setup

pip install guidellm
            guidellm benchmark \
              --target "http://localhost:8000" \
              --rate-type concurrent \
              --rate 1 \
              --max-requests 24 \
              --data "prompt_tokens=128,output_tokens=64" 

[tukwila]

guidellm doesn't work with the new api. I tried guidellm v0.3.0.

Here the setup. I am using cpu, but you don't have to.

Setup transformers server: Extra steps that don't mention here, is to get access to the meta-llama/Llama-3.2-1B-Instruct model. Also give your hugging face token to transformers to download the model.

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers[serving]
transformers serve
--device cpu
--host 0.0.0.0
--port 8000
--force-model meta-llama/Llama-3.2-1B-Instruct &
guidellm setup

pip install guidellm
guidellm benchmark
--target "http://localhost:8000"
--rate-type concurrent
--rate 1
--max-requests 24
--data "prompt_tokens=128,outlut_tokens=64"

Yes, i reproduce your problem in local transformers server. And i found the problem is as such:

OpenAIHTTPBackend request with headers: {'Content-Type': 'application/json'} and params: {} and payload: {'messages': [{'role': 'user', 'content': 'Test connection'}], 'model': 'meta-llama/Llama-3.2-1B-Instruct', 'stream': True, 'stream_options': {'include_usage': True}, 'max_tokens': 1, 'max_completion_tokens': 1, 'stop': None, 'ignore_eos': True} failed: Client error '422 Unprocessable Entity' for url 'http://localhost:8000/v1/chat/completions'

transformers doesn't support params: 'stop': None, 'ignore_eos': True

so when i disable:

guidellm can do benchmark test successfully:

but i don't know how to fix this. @sjmonson Could you give me some idea?

[sjmonson]

Officially in the OpenAI chat-completion spec there is no way to control the minimum number of output tokens. This is one of the reasons we default to the legacy completion endpoint. You can pass --backend_args "remove_from_body=['stop','ignore_eos']" to remove the the fields from the request. However without enforcing a strict output length your benchmark results will be very inconstant (basically not recommended).

Alternatively use a real-world dataset with semi-predictable responses from the model server and no max_tokens. This is not perfect either, the response length will still vary and do not compare the performance of different models this way as it will not be apples-to-apples.

Also here is a shameless plug for vLLM which supports the legacy completions endpoint and chat/completions with these options (GuideLLM is a subproject of vLLM).