[Bug]: Synthetic include_usage chunk violates OpenAI spec — usage event has non-empty choices instead of choices: []`

[dborysenko]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Related: #8450 (same symptom, closed stale), PR #8751 (proposed fix, not merged).

With stream: true and stream_options: {"include_usage": true}, the OpenAI streaming spec requires the final usage chunk to have choices: []. LiteLLM emits a synthetic usage chunk at stream end with a non-empty choices array (see log output below).

OpenAI's documented shape for the final usage chunk:

{
  "choices": [],
  "usage": { "prompt_tokens": 13, "completion_tokens": 20, "total_tokens": 33 }
}

Spec reference: Chat Completions streaming — include_usage

The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array.

Steps to Reproduce

curl -N 'http://127.0.0.1:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-...' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bedrock-claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Inspect the last JSON event before data: [DONE]. The chunk with non-null usage has choices with one entry ({"index": 0, "delta": {}}) instead of [].

Root cause

On StopIteration, when send_stream_usage=True, CustomStreamWrapper builds a synthetic chunk via model_response_creator() and attaches usage from stream_chunk_builder:

# litellm/litellm_core_utils/streaming_handler.py
response = self.model_response_creator()
setattr(response, "usage", getattr(complete_streaming_response, "usage"))
return response

model_response_creator() fills in a default choice when none is provided:

# litellm/litellm_core_utils/streaming_handler.py
model_response.choices = [StreamingChoices(finish_reason=None)]

The usage chunk should set choices = [] before returning.

Impact

LangChain.js (@langchain/openai, streamUsage: true) double-counts tokens when backed by LiteLLM.

In _streamResponseChunks, LangChain captures usage from any chunk that has it, but only skips delta emission when choices is empty. With OpenAI-direct, the final usage chunk has choices: [], so LangChain records usage once and emits a single trailing usage_metadata chunk. With LiteLLM, the usage chunk has choices: [{index: 0, delta: {}}], so LangChain treats it as a content event (yields an empty delta chunk with usage attached) and emits its own trailing usage chunk. Aggregating streamed chunks (e.g. concat()) then sums token counts twice (~2× prompt/completion/total).

Also affects Dify and other clients that detect the usage-only chunk via len(choices) == 0 (reference).

Relevant log output

Final SSE event before `data: [DONE]` (model `bedrock-claude-sonnet-4-6`, LiteLLM proxy):


data: {"id":"chatcmpl-31aaa6b2-e51f-4b42-88e9-e1be7c084a50","created":1779589160,"model":"bedrock-claude-sonnet-4-6","object":"chat.completion.chunk","choices":[{"index":0,"delta":{}}],"usage":{"completion_tokens":20,"prompt_tokens":13,"total_tokens":33,"completion_tokens_details":{"reasoning_tokens":0,"text_tokens":20},"prompt_tokens_details":{"cached_tokens":0,"text_tokens":13,"cache_creation_tokens":0},"cache_creation_input_tokens":0,"cache_read_input_tokens":0}}

choices should be [], not [{"index":0,"delta":{}}].

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.85.0

Twitter / LinkedIn details

No response

[FuturMix]

The root cause is clear from your analysis — model_response_creator() fills a default StreamingChoices entry, and the usage chunk path doesn't override it to [].

Why this matters beyond cosmetics

The choices: [] in the usage chunk isn't just a stylistic convention — it's a sentinel that downstream parsers use to distinguish "this is a metadata-only event" from "this is a content event that also carries usage data."

The spec-defined contract:

Regular chunk:     choices = [{ delta: { content: "..." }, ... }]    usage = null
Final content:     choices = [{ delta: {}, finish_reason: "stop" }]  usage = null
Usage-only chunk:  choices = []                                      usage = { ... }
[DONE]

When LiteLLM emits choices: [{index: 0, delta: {}}] on the usage chunk, parsers that check len(choices) == 0 see it as a content event. The specific impacts:

LangChain.js (@langchain/openai): _streamResponseChunks only skips delta emission when choices.length === 0. With the non-empty choices, it yields an empty delta chunk with usage attached, then its own trailing usage metadata chunk — resulting in 2× token counts when aggregated via concat().

Dify: The provider adapter checks len(chunk.choices) == 0 to detect the usage-only event. With non-empty choices, it falls through to the content path, and usage may be recorded from a different code path or missed entirely.

Any client that concat()s streaming chunks: The empty delta: {} in the usage chunk gets concatenated, which is harmless for content but means usage data gets attached to a "content" chunk instead of being isolated in a metadata-only chunk.

Fix

In streaming_handler.py, where the synthetic usage chunk is built on StopIteration:

response = self.model_response_creator()
response.choices = []  # ← add this line
setattr(response, "usage", getattr(complete_streaming_response, "usage"))
return response

One line. This matches the OpenAI reference implementation behavior and fixes all three downstream parsers mentioned above.

Edge case to consider

Some clients may be written to always expect at least one choice entry and would break on choices: []. If there's concern about backward compatibility, a feature flag (spec_compliant_usage_chunk: bool, default True) lets users opt out. But given that this is the documented OpenAI behavior, I'd lean toward just fixing it — clients that break on choices: [] are already broken against OpenAI-direct.

[dborysenko]

Another impacted tool is Librechat, its token count system is confused by LiteLLM proxy.
Feels like major functionality is broken (OpenAI compatibility) and the fix seems to be relatively easy, can maintainers take a look and approve one of the proposed PRs, please?

[dborysenko]

Built a custom image with PR #28736 on top of release/v1.86.1, deployed to our lab environment, and verified streaming with stream_options: {"include_usage": true}.

The final usage chunk now has choices: [] as expected. LangChain.js and LibreChat both report correct token usage (no more double-counting). Looking forward to seeing this merged — in the meantime we'll stay on our custom LiteLLM build.