Ability to batch prompts in Chat Completion like for the Completion endpoint.

[h-gharbi]

Describe the feature or improvement you're requesting

In the Completion endpoint we can batch input prompts in one API request. It would be nice to have the same feature for the Chat Completion.

Additional context

No response

[sfarrell5123]

it already has index in the response.

either batch on the chat endpoint

or allow us gpt-35-turbo on the complete endpoint - so we can batch there.

[logankilpatrick]

Thanks for the feedback! It is something the team was thinking about but has not had time for.

[logankilpatrick]

I am going to close for now since this repo is for the Python SDK not API feedback but will follow up if we end up adding this.

[JakeWinter]

Tried to use async requests in python, sometimes it stalls out, sometimes it works well

[morenobonaventura]

this is an alternative solution:
https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py

[SantoshGuptaML]

this is an alternative solution: https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py

This doesn't seem to be quite fully done for the chat completions endpoint, I get an error with the tokenizer. I created a version that is working

https://github.com/openai/openai-cookbook/blob/987d8cb2585424bad1594b5e50787ef221d5cc3d/examples/api_request_parallel_processor.py#L353

[FilippoBoni1921]

I am facing the same issue. I think that giving the possibility to batch the request was a really helpful feature for completion that has been removed with chat.completion. I think that would have been better to spend some time on it before releasing.
Anyway I think quite some time has passed and still we do not have a concrete alternative. Apart from the introduction of the "system message" I do not see a lot of advantages of chat.completions over simple completion.
Also I have one question: why completions has been being labelled as legacy for such a long time but not eliminated, nor fully substituted?
I am really looking forward to an alternative for batch processing if someone can suggest. I hope to receive an answer even after so so much time from the opening of the issue