Interrupt streaming for chat completions using the API like ChatGPT

[JohnnyRacer]

Confirm this is a feature request for the Python library and not the underlying OpenAI API.

• This is a feature request for the Python library

Describe the feature or improvement you're requesting

Hello, I saw this thread on the forums here to allow for the ability to stop or interrupt the generation using the chat assistants API. There does not seem like there is anything in the docs to allow to work. Is it possible this can be implemented in the future to save tokens and generations on streaming responses?

Additional context

No response

[rattrayalex]

To interrupt the stream, simply break from the iterator. For example:

stream = client.chat.completions.create(stream=True, …)
for chunk in stream:
  print(chunk)
  if you_are_done:
    break

Is that what you were looking for?

Note that doing this will leak the response, so you should ideally do this:

stream = client.chat.completions.create(stream=True, …)
try:
  for chunk in stream:
    print(chunk)
    if you_are_done:
      break
finally:
  stream.response.close()

(we're working on improving the ergonomics of this soon)

[JohnnyRacer]

@rattrayalex Much thanks, does it method also prevents the tokens from being continuously generated after the stream has been interrupted? I tried to terminate the actual API requests but it seems like it will throw an error on how it was still processing the last run when I tried to start another generation and I was billed for the generations that were created after the connection was terminated.

[rattrayalex]

Hmm, that shouldn't happen – can you provide more details?

[zlxie0106]

@rattrayalex my openai version is 0.27.6.
here is my code:
task = openai.ChatCompletion.create( model="gpt-3.5-turbo", temperature=0.7, n=1, messages=messages, stream=True, ) task.response.close() <----AttributeError: 'generator' object has no attribute 'response'

As far as I know, OpenAI sends back the API response using Server-Sent Events (SSE). Even if we break out of the loop, the generator continues to generate, and charges may still apply. However, I don't know how to properly close the SSE connection. If I explicitly close it, using task.response.close(), an AttributeError occurs ('generator' object has no attribute 'response').
Is there a correct way to stop it?

[rattrayalex]

Version 0.x is no longer supported – you'll need to upgrade to 1.x.

[punkpeye]

It looks like it is just:

const { data: completion } = await client.chat.completions.create({})

completion.controller.abort();