fix(llm): tolerate list-shaped streaming delta content in strip_thinking_tokens - #6625
Merged
chenghao-mou merged 2 commits intoJul 31, 2026
Merged
Conversation
…ing_tokens The OpenAI-compatible ecosystem allows delta.content to be a list of typed content parts instead of a string - Mistral emits this shape on some streamed chunks - and the openai SDK constructs stream models without validation, so the list reaches strip_thinking_tokens as-is when the plugin points at such a provider. The filter assumed str and crashed the session; wrapped as a retryable APIConnectionError mid-stream, a FallbackAdapter then silently moved the whole session to its fallback leg. Flatten list-shaped content to its concatenated text parts (dict parts, plain strings, and typed objects with a text attribute) before the tag scan. Unknown content types are dropped with a warning instead of crashing. Supersedes the stalled livekit#6324; credit to KSerProject for the original report and fix direction. Fixes livekit#6323
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
chenghao-mou
approved these changes
Jul 31, 2026
chenghao-mou
left a comment
chenghao-mou
left a comment
Member
There was a problem hiding this comment.
lgtm, thanks for the PR. One small nit.
a list of content parts with nothing textual in it (an image part, or an empty list) was flattened to an empty string, claiming the provider sent empty text. It now reports no content, like the None and unexpected-type paths already do. A part that carries an empty string still yields "", the same as a plain "" delta.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6323
Problem
strip_thinking_tokensassumesdelta.contentisstr | None. The OpenAI-compatible ecosystem allows it to be a list of typed content parts ([{"type": "text", "text": ...}]) — Mistral emits this shape on some streamed chunks — and the openai SDK constructs stream models without validation (construct()), so whenopenai.LLMpoints at such a provider viabase_url, the raw list reaches the filter and crashes the session. Wrapped as a retryableAPIConnectionErrormid-stream, aFallbackAdapterthen silently fails the whole session over to its fallback leg — a hard-to-diagnose degradation.Re: the open question on the issue about the source of the claim ("the OpenAI streaming spec only documents string"): correct for OpenAI's own service — the authority here is Mistral's chat-completions API, whose assistant/delta message
contentis typedstring | ContentChunk[]in their reference and SDK. Since the plugin explicitly supports OpenAI-compatible providers throughbase_url, and the SDK's unvalidated stream models pass whatever the provider sends, the list shape is reachable in practice (that's how the original reporter hit it in production). Answered in more detail on the issue thread.Change
Flatten list-shaped content to its concatenated text parts before the tag scan — dict parts (
{"text": ...}), plain strings, and typed objects with a.textattribute. Unknown content types are dropped with a warning instead of crashing. String/None behavior is byte-identical.Tests
Five new cases in
tests/test_llm_utils.py: the issue's exact repro ([{"type": "text", "text": "Hallo"}]), mixed part lists, thinking-tag stripping across list-shaped chunks, non-text parts ignored, and typed objects with.text.Relationship to #6324
This supersedes the stalled #6324 (same diagnosis, by @KSerProject — credit where due): that PR has had merge conflicts and an unsigned CLA for four weeks. Happy to close this one if the original author returns.