[Usage]: How to batch requests to chat models with OpenAI server?

[sidjha1]

Your current environment

...

How would you like to use vllm

I am serving a chat model (e.g. Llama 3 70B Instruct) using the OpenAI compatible server. However the v1/chat/completions endpoint does not take a batch. In contrast, the v1/completions endpoint takes a batch. Is the proper way to support batching to use multiple threads to send requests to v1/chat/completions or should I apply the chat template myself and then send it to the v1/completions endpoint? Thanks!

[simon-mo]

Either approaches works, the performance should be similar as there's one single engine performing batching under the hood.

[sidjha1]

Great, thanks @simon-mo!