[Feature]: Rate limit aware batch completion calls

[ishaan-jaff]

The Feature

pass in a bunch of batch completions and a rate limit. the requests should not fail

Motivation, pitch

user request

Twitter / LinkedIn details

No response

[krrishdholakia]

Related issues

• [Bug]: Have fallback work with batch_completions  #507
• [Feature]: Support Caching with Batch Completions  #508

[krrishdholakia]

@ishaan-jaff are the related issues in-scope for this feature?

[ishaan-jaff]

as a v0 of this sticking to just this: https://docs.litellm.ai/docs/rate_limit_manager

[vilmar-hillow]

Is there current/planned support for rate limit aware batch completion calls? Or any workaround/tweaks to the current batch_completion method call to slow down/space out the rate of messages being sent?

[krrishdholakia]

@vilmar-hillow the recommended approach here is to use Router for handling the reliability logic, and allow users to write their own parallel calling logic - https://docs.litellm.ai/docs/routing

reason: seems like the batch completion logic is just multi-threaded / parallel async calls to completion().

Is there a reason you want litellm to handle this?

[vilmar-hillow]

Thanks! I'll explore the router. The use case is simple - batch processing for experimentation, sending as many requests as possible to a single model given the set TPM/RPM, and batch_completion seems usable only for non-rate-limited cases. Then I saw this closed feature specifically talking about this feature, so I thought it would be a good place to ask :)

[krrishdholakia]

@CLARKBENHAM caught a relevant issue around making the router better for this.

We've added the testing for this. Will work on fixing issues it raises. If there's something specific, will make it a separate issue for tracking.