Connection Not Closing Properly When Using GPT API with Stream

[rodrigoGA]

Describe the bug

I am experiencing a connection issue when utilizing OpenAI's GPT API with the streaming feature in my application. When I close the stream using the method I've implemented, the connection to OpenAI does not terminate correctly. After a certain duration, a timeout error is thrown by the discarded thread. The error message I receive is:

CRITICAL - Request timed out: HTTPSConnectionPool(host='api.openai.com', port=443): Read timed out. (read timeout=600)
...
openai.error.Timeout: Request timed out: HTTPSConnectionPool(host='api.openai.com', port=443): Read timed out. (read timeout=600)

To Reproduce

1. Initiate the GPT API with streaming using the code snippet provided.

def gpt(prompt: str):
    for chunk in openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": 'user', "content": prompt}],
        stream=True
    ):
        if (text_chunk := chunk['choices'][0]['delta'].get('content'))  is not None:
            yield text_chunk

2. In specific scenarios, discard the API response and close the stream.

gpt_stream = write(prompt)
for text_chunk in gpt_stream:
	if discarded:
		gpt_stream.close()
		return;
	... continue

3. Wait for a certain period.
4. Observe the aforementioned timeout error.

Code snippets

No response

OS

ubuntu 22.04

Python version

3.9

Library version

0.27.8

[rishabhRsinghvi]

@rodrigoGA please assign this stuff to me.

[RobertCraigie]

This should be fixed in the upcoming v1 beta! Please try it and let us know!

To explicitly close a streamed response you can call stream.response.close().