AsyncAzureClient connection hangs frequently

[nsbradford]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

something very strange: the python OpenAI AzureClient works fine, but using the AsyncAzureClient results in hanging connection ~5% of the time

To Reproduce

Run a bunch of requests (~100) to GPT-4 using AsyncAzureClient, eventually one of them will hang for 20+ minutes before timing out

Code snippets

No response

OS

macOS

Python version

Python v3.11.6

Library version

1.6.1

[rattrayalex]

Hi @nsbradford – what gives you the sense that this is an SDK problem rather than an API problem on the Azure end?

Can you share example code that is causing the problem, and a bit of code that hits the same API without this problem? For example, hitting Azure with OpenAI(base_url="https://some.azure.deployment/openai")?

[rattrayalex]

cc @kristapratico

[nsbradford]

hi there! if I run my suite of tests I only ever experience this using AsyncAzureClient, with sync AzureClient it always works fine. I'll try and create a minimal example, but because it's so low-frequency it might be tricky...

[nsbradford]

I seem to be able to replicate it pretty reliably (maybe 10-20% of the time?) with requests to "gpt-4-32k-0613" that are over 20k tokens using the async client - the problem happens with both Azure and OpenAI endpoints. But never with the sync client (!?). Trying to build a script...

edit: ok, it's pretty bizarre. Running the bad request via the script below directly does not timeout, but it does timeout when running my application. Is there any other possible reason this could be happening so consistently?

@pytest.mark.parametrize(
    "client", [openai_async_client, azure_client, azure_async_client]
)
async def test_flaky_timeouts_large_request(
    client: Callable[[LLMCache], ChatCompletionClient]
):
    with open("ellipsis/test/flaky_integration/bad_request_content.json") as f:
        request_1 = ChatCompletionRequest.parse_raw(f.read())
    n_tests = 20
    timeout = 120
    results = await asyncio.gather(
        *(single_test_iteration(client, request_1, i, timeout) for i in range(n_tests))
    )
    n_failures = results.count(False)
    failure_rate = (n_failures / n_tests) * 100
    print(f"Total Failures: {n_failures}/{n_tests}, Failure Rate: {failure_rate:.2f}%")
    
async def single_test_iteration(client, request, iteration, timeout) -> bool:
    try:
        response = await asyncio.wait_for(client(mock_cache).chat_completion(request), timeout)
        success = True
    except Exception as e:
        success = False
    return success

[kristapratico]

@nsbradford do I have this right?

• Hang only occurs with async
• Reproduces with both Azure and OpenAI endpoints

I'll try to repro this on my end. At first I suspected rate limiting with a large retry-after value, but that wouldn't make sense if you aren't seeing this with the sync client.

I'd suggest enabling debug logging to help troubleshoot this issue and also upgrading your openai library to the latest if you haven't already.

[rattrayalex]

Running the bad request via the script below directly does not timeout, but it does timeout when running my application. Is there any other possible reason this could be happening so consistently?

Interesting. Can you share more about your application setup, in particular what kind of async setup you're using, what multithreading/similar you're doing, and how many things dealing with OpenAI in parallel you may be doing?

And just to be clear - you're seeing this with both AsyncOpenAI and AsyncAzureOpenAI, yes?

[rattrayalex]

cc @RobertCraigie for thoughts here

[nsbradford]

Thanks! It seems to happen both when i'm using pytest, and when I'm running in production on Celery workers. When I start a workflow that runs several LLM requests back to back, it's never the first one that fails, usually after only a few. There are never any other tests or requests being made at the same time. It seems to be only on larger requests; the one i'm able to see happen regularly is 20k+ tokens to gpt-4-32k

Anecdotally, the only thing that this reminds me of is that the JavaScript SDK also used to hang sometimes if you submitted a very large request to GPT-4 (but haven't tested that in 6 months) - and people used to circumvent this by using the streaming client. I might try that.

But again, we've never seen it happen with the Sync openai client, while on the Async client it happens quite often in both dev and prod :( My totally harebrained suspicion: something about the underlying http client behaves weirdly on very large/long-running requests?

[RobertCraigie]

@nsbradford I suspect the underling cause of your issue is this one which should hopefully be fixed soon.

[aryan1107]

#1134 Also is similar lot of timeouts in Azure and customer can't wait for 120+ seconds for answer.

[nsbradford]

Thanks @RobertCraigie (and good to see the Stainless team again), looking forward to the release 🙏

[RobertCraigie]

Please try installing httpcore==1.0.3! This should fix the issues you're seeing.