[Bug]: async timeout error on proxy

[krrishdholakia]

What happened?

Hey, it seems after a few hours, the python server starts timing out for every call, then when it's restarted, it's fine but then starts timing out all the calls again.

Relevant log output

ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err
ERROR    raise APITimeoutError(request=request) from err

Twitter / LinkedIn details

No response

[krrishdholakia]

User is using:

• litellm proxy
• azure chat completions

[ishaan-jaff]

we have a deployed instance of the proxy here, we can run load testing against it: https://litellm-api.up.railway.app/ and try to repro

[ishaan-jaff]

seems related to this: openai/openai-python#821

I suspect it has something to do with the Azure/OpenAI client Initialization

[krrishdholakia]

"It's like consistently around 4 hours" - user

"[7:51 AM, 12/30/2023] Okay happened, again, checking logs
[7:53 AM, 12/30/2023] No errors
[7:53 AM, 12/30/2023] Only warning is

WARNINGWarning: the config option 'server.enableCORS=false' is not compatible with 'server.enableXsrfProtection=true'.
[7:54 AM, 12/30/2023] any subsequent calls to gpt-3.5-turbo (azure) hung. calls to gpt-4 (openai) were fine
[7:55 AM, 12/30/2023] After restart, all is good"

[krrishdholakia]

Pretty sure it's an azure-openai client issue.

@ishaan-jaff open to thoughts on how to repro this within a shorter timeperiod. Can play around with min timeouts, and see how to better test this.

[krrishdholakia]

This seems pretty similar to what we're facing: openai/openai-python#769 (comment)

[krrishdholakia]

Seems like this might be related to connections not being properly closed, I don't think bumping the number is a good solution, as it'll just show up a little later on -

[krrishdholakia]

Another relevant issue - openai/openai-python#874 (comment)

[krrishdholakia]

Seems like this - #1270, could be a solution for the async timeout problem.

We can just cache clients for ~1hr, and reinitialize them if they're invalid.

[krrishdholakia]

have this be a controllable param, for users who are seeing higher traffic, and need to re-initalize clients more frequently (there's obviously a trade-off here).

This is also a patch, until openai has a more permanent fix out for how they handle connections (seems to be some sort of underlying bug with connections not being closed properly).

[krrishdholakia]

this is now added.