-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: async timeout error on proxy #1278
Comments
User is using:
|
we have a deployed instance of the proxy here, we can run load testing against it: https://litellm-api.up.railway.app/ and try to repro |
seems related to this: openai/openai-python#821 I suspect it has something to do with the Azure/OpenAI client Initialization |
"It's like consistently around 4 hours" - user "[7:51 AM, 12/30/2023] Okay happened, again, checking logs WARNINGWarning: the config option 'server.enableCORS=false' is not compatible with 'server.enableXsrfProtection=true'. |
Pretty sure it's an azure-openai client issue. @ishaan-jaff open to thoughts on how to repro this within a shorter timeperiod. Can play around with min timeouts, and see how to better test this. |
This seems pretty similar to what we're facing: openai/openai-python#769 (comment) |
Another relevant issue - openai/openai-python#874 (comment) |
Seems like this - #1270, could be a solution for the async timeout problem. We can just cache clients for ~1hr, and reinitialize them if they're invalid. |
have this be a controllable param, for users who are seeing higher traffic, and need to re-initalize clients more frequently (there's obviously a trade-off here). This is also a patch, until openai has a more permanent fix out for how they handle connections (seems to be some sort of underlying bug with connections not being closed properly). |
this is now added. |
What happened?
Hey, it seems after a few hours, the python server starts timing out for every call, then when it's restarted, it's fine but then starts timing out all the calls again.
Relevant log output
Twitter / LinkedIn details
No response
The text was updated successfully, but these errors were encountered: