Reusing AsyncOpenAI client results in openai.APIConnectionError #1059
Description
Confirm this is an issue with the Python library and not an underlying OpenAI API
- This is an issue with the Python library
Describe the bug
Reusing an instance of AsyncOpenAI client for multiple calls of asyncio.gather results in an openai.APIConnectionError. Retried requests (either via the openai library directly or backoff decorator) succeed, but the first try of the second use of the client always fails.
I suspect that this usage of AsyncOpenAI is not ideal, but the behavior nonetheless feels buggy. Even if the reuse of the client should fail, I'm confused understand why retries succeed.
Bizarrely, in my application all retries after the initial openai.APIConnectionError result in unending openai.APITimeoutError instead of success, but I am unable to repro this outside of the application. However, I strongly suspect that the issue is related as reusing the client solves both the initial error as well as the timeouts.
To Reproduce
- Create an instance of
AsyncOpenAIwith no retries enabled - Use
AsyncOpenAI().chat.completions.createto create a list ofFutureobjects (any number will do) - Use
asyncio.gatherto get the results of the API calls - Executes steps 3+4 again - this causes the error
Code snippets
import asyncio
import openai
from openai import AsyncOpenAI
import httpx
import backoff
print(f"OpenAI version: {openai.__version__}")
OPENAI_API_KEY = "redacted"
api_params = {
"temperature": 0.2,
"max_tokens": 500,
"model": "gpt-3.5-turbo-1106",
}
messages = [{"role": "user", "content": "What is the capital of Quebec?"}]
@backoff.on_exception(
backoff.expo,
(
openai.RateLimitError,
openai.APIStatusError,
openai.APIConnectionError,
openai.APIError,
openai.APITimeoutError,
openai.InternalServerError,
),
)
async def create_request_retry(client, messages, api_params):
return await client.chat.completions.create(messages=messages, **api_params)
async def create_request_no_retry(client, messages, api_params):
return await client.chat.completions.create(messages=messages, **api_params)
# No retries, new client for each set of requests - succeeds
def succeed1():
for i in range(2):
client = AsyncOpenAI(
api_key=OPENAI_API_KEY,
timeout=10.0,
http_client=httpx.AsyncClient(limits=httpx.Limits(max_keepalive_connections=500, max_connections=100)),
max_retries=0,
)
arequests = []
for _ in range(5):
arequests.append(create_request_no_retry(client, messages, api_params))
responses = asyncio.run(asyncio.gather(*arequests))
results = [response.choices[0].message.content for response in responses]
print(f"{i}: {results}")
# Retry using backoff decorator, reuse client - succeeds
def succeed2():
client = AsyncOpenAI(
api_key=OPENAI_API_KEY,
timeout=10.0,
http_client=httpx.AsyncClient(limits=httpx.Limits(max_keepalive_connections=500, max_connections=100)),
max_retries=0,
)
for i in range(2):
arequests = []
for _ in range(5):
arequests.append(create_request_retry(client, messages, api_params))
responses = asyncio.run(asynciogather(*arequests))
results = [response.choices[0].message.content for response in responses]
print(f"{i}: {results}")
# Retry using openai library, reuse client - succeeds
def succeed3():
client = AsyncOpenAI(
api_key=OPENAI_API_KEY,
timeout=10.0,
http_client=httpx.AsyncClient(limits=httpx.Limits(max_keepalive_connections=500, max_connections=100)),
max_retries=2,
)
for i in range(2):
arequests = []
for _ in range(5):
arequests.append(create_request_no_retry(client, messages, api_params))
responses = asyncio.run(asynciogather(*arequests))
results = [response.choices[0].message.content for response in responses]
print(f"{i}: {results}")
# No retries, reuse client - fails
def error():
client = AsyncOpenAI(
api_key=OPENAI_API_KEY,
timeout=10.0,
http_client=httpx.AsyncClient(limits=httpx.Limits(max_keepalive_connections=500, max_connections=100)),
max_retries=0,
)
for i in range(2):
arequests = []
for _ in range(5):
arequests.append(create_request_no_retry(client, messages, api_params))
responses = asyncio.run(gather(*arequests))
results = [response.choices[0].message.content for response in responses]
print(f"{i}: {results}")OS
macOS
Python version
Python 3.11.5
Library version
openai v1.6.1