Open AI Clients starts timing out after some time in production

[AshuBik]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

A weird issue that I am facing in production, after the code is deployed and it has been 24-26 hours the any calls made using the above client start timing out. Usually taking out a redeployment helps.

This pattern makes it feel like the underlying HTTP connections are not being released or refreshed correctly, which may be causing a connection pool exhaustion.

Any best practices or recommendations on how to tackle this ?

My configuration

openai.api_key = os.environ.get("OPENAI_API_KEY")

os.environ.setdefault("OPENAI_LOG", "debug")

client = AsyncOpenAI(
    http_client=DefaultAioHttpClient(
        limits=httpx.Limits(max_connections=1000, max_keepalive_connections=100)
    ),
)

I am streaming and once i am done with streaming I am closing the stream as well

response = return await chat_client.chat.completions.create(**params)

## Later on 
if response:
    await response.close()

name = "openai"
version = "1.99.9"

To Reproduce

• NA since it it not easily replicable in local.

Code snippets

OS

Debian Linux

Python version

3.10.6

Library version

1.99.9

[AshuBik]

The timeout trace

"OpenAi error [TimeoutError]::  Traceback (most recent call last):
  File "/app/app/openai/v2/api/chat.py", line 167, in get_completion_response
    completion = await get_completion(chat_client, model, params)
  File "/app/app/openai/v2/api/chat.py", line 119, in get_completion
    return await chat_client.chat.completions.create(**params)
  File "/usr/local/lib/python3.10/site-packages/openai/resources/chat/completions/completions.py", line 2585, in create
    return await self._post(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1794, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1529, in request
    response = await self._client.send(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx_aiohttp/transport.py", line 151, in handle_async_request
    response = await self.client.request(
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1488, in __aenter__
    self._resp: _RetType = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 770, in _request
    resp = await handler(req)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 624, in connect
    await self._wait_for_available_connection(key, traces)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 694, in _wait_for_available_connection
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/app/openai/v2/api/chat.py", line 232, in get_completion_response_with_retry
    result = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

[AshuBik]

@RobertCraigie can you please help here ?

[karpetrosyan]

I’ve released version 0.1.9 of httpx-aiohttp. Could you please try bumping the version to 0.1.9 and see if it fixes the issue?

[AshuBik]

@karpetrosyan thanks, any steps to replicate this behaviour in local ? and what is the root cause ?

[karpetrosyan]

I think the main problem was that httpx-aiohttp used content= instead of stream= when creating the response instance. This doesn't register a stream with proper cleanup (the aclose method), so cleanup was ignored.

I think you can reproduce it by just sending requests using the same httpx client until you get stuck