OpenAI Azure chat.completions.create with stream=True stalling without throwing exception

[begilbert238]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

I have a fastapi endpoint which streams a response from Azure OpenAI:

@router.post("/message")
def message(sessionID: str = Form(...), messages: str = Form(...), selectedModel: str = Form(...), file: UploadFile = Form(None)):
    message_list = json.loads(messages)
    return StreamingResponse(stream_chat(message_list, sessionID, selectedModel))

def stream_chat(messages: list[ChatCompletionMessageParam], session_id: str, selected_model: str) -> Generator[StreamingResponseChunk, None, None]:
    print('entered stream_chat')
    completion_stream = azure_openai_client.chat.completions.create(
        model=selected_model,
        messages=messages,
        stream=True,
        timeout=45,
        **DEFAULT_MODEL_CONFIG
    )
    print('completion_stream created!')
    #. . . 

On occasion, azure_openai_client.chat.completions.create will infinitely stall, and cause every subsequent request to infinitely stall as well. Once the error occurs, every request reaches 'entered stream_chat' but not 'completion_stream created!', until I reset the FastAPI instance. No exceptions are raised, even after hitting the 45 second timeout. I haven't been able to recreate the bug consistently either. What on earth is going on?
For reference, here are the debug logs when the error starts:

DEBUG:openai._base_client:Sending HTTP Request: POST https://[my-deployment]/openai/deployments/gpt-4o/chat/completions?api-version=2024-08-01-preview
DEBUG:httpcore.http11:receive_response_body.failed exception=GeneratorExit()
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete

To Reproduce

I am unsure how to reproduce consistently, it seems to happen at random, fairly rarely.

Code snippets

OS

macOS

Python version

Python v3.11.4

Library version

openai v1.75.0

[qkfang]

It might related to rate limit 429 or azure content safety 400 http status code.
a similar bug is here for azure .net sdk ([BUG] Chat streaming implementation hangs on a 429 response, no async way to access response body) Azure/azure-sdk-for-net#45618

[begilbert238]

Thank you for the response, I don't think this is the issue, I've run into both of those errors before and I can see them in the debug logs. With this issue, no logs are shown other than what I posted. And it doesn't correlate to the content / frequency of the message - I wasn't able to reproduce the bug by spamming Azure.

[begilbert238]

Upon further investigation I believe this is actually an issue with httpcore which is a sub-dependency of the library, I'm almost positive that this discussion is talking about the same issue, so I don't think this library can do anything about it.