GPT-4-0613 Finish_Reason Null Issue in Streaming Mode

[pratikchhapolika]

Describe the bug

I am facing an issue while using the latest OpenAI GPT-4-0613 models in streaming mode. Specifically, I have noticed that for some cases, the finish_reason field in the response comes as Null instead of the expected value 'stop' for the last token. I have provided the necessary context along with my question in the prompt. Similar Question is asked in Openai Forum: https://community.openai.com/t/completion-finish-reason-is-missing-when-stream-true/90526

My understanding is that the finish_reason field should indicate 'stop' for the last token when using the streaming mode. However, in certain cases, it appears to be Null. This behavior seems inconsistent with the expected behavior of the model.

Notice: However, gpt-0314 always gives finish_reason as stop for last token

To Reproduce

Take a prompt which has Context and Question. ( Make a prompt size of 5200 approx)

Code snippets

response = openai.ChatCompletion.create(
    model='gpt-4-0613',
    messages=[
        {'role': 'user', 'content': "Context along with question"}
    ],
    temperature=0,
    stream=True 
)

for chunk in response:
    print(chunk)

OS

macOS

Python version

Python3.9

Library version

openai from azure

[microsoftbuild]

Hi @pratikchhapolika,

I was unable to reproduce the bug by following the details from your issue.
Can you share the specific chat completion chunk that you received with those attributes?

[hughyyyy]

I am also facing the same issue but in typescript. Thus results in this following error:
SyntaxError: Unterminated string in JSON at position 870.

This is due to the return response is unfinished and the JSON string is not completed. What should user do when we encounter a unfinished JSON string response?

[microsoftbuild]

@hughyyyy ,
If the chat completion your receive is supposed to be a JSON string, you should:

1. Wait for the streaming response to finish.
2. Validate the received JSON string to check if it's valid JSON.
3. Proceed only if it's valid json else retry.

[hughyyyy]

How do you wait? I am using typescript btw.

Currently this is what I am getting:

{
"id": "chatcmpl-xxxxxxxxxxxxxxxxxx",
"object": "chat.completion",
"created": xxxxxxxxxxxxx,
"model": "gpt-3.5-turbo-16k-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and the json string is left open and unfinished...

[microsoftbuild]

Interesting and strange @hughyyyy

Can you share the code that's calling the API and reading from it?

It's very unlikely for this to happen.

[hughyyyy]

I just do

const res = await fetch('xxx/v1/chat/completions', ...)
const value = await res.body.getReader().read();
const decoder = new TextDecoder();
console.log(decoder.decoder(value?.value))

And the log is (json string) is left open and unfinished.

[microsoftbuild]

@hughyyyy
Currently the output you're getting is chat.completion object which is being badly handled. In order to receive streaming response, you need to set stream to true.

Then you will receive the streaming response in chat.completion.chunk objects which look like:

{
  "id": "chatcmpl-123",
  "object": "chat.completion.chunk",
  "created": 1677652288,
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "delta": {
      "content": "Hello",
    },
    "finish_reason": "stop"
  }]
}

Here's how you can stream responses using openai-nodeJS

[logankilpatrick]

I just tested this and was not able to replicate it. Please let me know if it continues to be an issue.