What happened?
When we directly call the inference / model endpoint deployed via vLLM (with stream=true) the last chunk contains finish_reason = stop and empty content string as shown below. But when we add same endpoint in litellm and call litellm endpoint, the last chunk contains finish_reason = stop but with delta as empty object as shown below.
I understand that finish_reason=stop key value is good enough to understand that its the last chunk but there are applications where it checks the content field even for last chunk.
Even the OpenAI last chunk contains empty content string.
Can we please get this issue fixed so that its sync with vLLM and also OpenAI.
Relevant log output
vLLM Output:
{
"id": "chatcmpl-679c23e0aae75f2e24744f2a06edbc32",
"object": "chat.completion.chunk",
"created": 1751952793,
"model": "meta-llama/Llama-3.3-70B-Instruct",
"choices": [
{
"index": 0,
"delta": {
"content": ""
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
]
}
Litellm Output:
{
"id": "chatcmpl-497e2040-1ab3-4a01-a155-95d40cb042e0",
"created": 1751952833,
"model": "meta-llama/Llama-3.3-70B-Instruct",
"object": "chat.completion.chunk",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"delta": {}
}
]
}Are you a ML Ops Team?
Yes
What LiteLLM version are you on ?
v1.61.20
Twitter / LinkedIn details
No response
What happened?
When we directly call the inference / model endpoint deployed via vLLM (with stream=true) the last chunk contains finish_reason = stop and empty content string as shown below. But when we add same endpoint in litellm and call litellm endpoint, the last chunk contains finish_reason = stop but with delta as empty object as shown below.
I understand that finish_reason=stop key value is good enough to understand that its the last chunk but there are applications where it checks the content field even for last chunk.
Even the OpenAI last chunk contains empty content string.
Can we please get this issue fixed so that its sync with vLLM and also OpenAI.
Relevant log output
vLLM Output: { "id": "chatcmpl-679c23e0aae75f2e24744f2a06edbc32", "object": "chat.completion.chunk", "created": 1751952793, "model": "meta-llama/Llama-3.3-70B-Instruct", "choices": [ { "index": 0, "delta": { "content": "" }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ] } Litellm Output: { "id": "chatcmpl-497e2040-1ab3-4a01-a155-95d40cb042e0", "created": 1751952833, "model": "meta-llama/Llama-3.3-70B-Instruct", "object": "chat.completion.chunk", "choices": [ { "finish_reason": "stop", "index": 0, "delta": {} } ] }Are you a ML Ops Team?
Yes
What LiteLLM version are you on ?
v1.61.20
Twitter / LinkedIn details
No response