fix(llm): tolerate list-shaped streaming delta content - #2188
Merged
Conversation
🦋 Changeset detectedLatest commit: 35cacea The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
chenghao-mou
approved these changes
Jul 31, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports livekit/agents#6625.
Summary
Source diff coverage
Source diff coverage
livekit-agents/livekit/agents/llm/utils.py: Adapted toagents/src/llm/utils.ts. The Python flattening behavior is translated to TypeScript, including string parts, record/object.textparts, empty-list semantics, and warning/drop behavior through the target logger.tests/test_llm_utils.py: Adapted toagents/src/llm/utils.test.ts. All seven source test cases are ported to the existing Vitest suite with only language- and framework-specific assertion changes.Testing
pnpm exec vitest run agents(111 files, 1,523 passed, 5 skipped)pnpm --filter @livekit/agents buildpnpm --filter @livekit/agents lintPorted from livekit/agents#6625
Original PR description
Fixes livekit/agents#6323
Problem
strip_thinking_tokensassumesdelta.contentisstr | None. The OpenAI-compatible ecosystem allows it to be a list of typed content parts ([{"type": "text", "text": ...}]) — Mistral emits this shape on some streamed chunks — and the openai SDK constructs stream models without validation (construct()), so whenopenai.LLMpoints at such a provider viabase_url, the raw list reaches the filter and crashes the session. Wrapped as a retryableAPIConnectionErrormid-stream, aFallbackAdapterthen silently fails the whole session over to its fallback leg — a hard-to-diagnose degradation.Re: the open question on the issue about the source of the claim ("the OpenAI streaming spec only documents string"): correct for OpenAI's own service — the authority here is Mistral's chat-completions API, whose assistant/delta message
contentis typedstring | ContentChunk[]in their reference and SDK. Since the plugin explicitly supports OpenAI-compatible providers throughbase_url, and the SDK's unvalidated stream models pass whatever the provider sends, the list shape is reachable in practice (that's how the original reporter hit it in production). Answered in more detail on the issue thread.Change
Flatten list-shaped content to its concatenated text parts before the tag scan — dict parts (
{"text": ...}), plain strings, and typed objects with a.textattribute. Unknown content types are dropped with a warning instead of crashing. String/None behavior is byte-identical.Tests
Five new cases in
tests/test_llm_utils.py: the issue's exact repro ([{"type": "text", "text": "Hallo"}]), mixed part lists, thinking-tag stripping across list-shaped chunks, non-text parts ignored, and typed objects with.text.Relationship to #6324
This supersedes the stalled #6324 (same diagnosis, by @KSerProject — credit where due): that PR has had merge conflicts and an unsigned CLA for four weeks. Happy to close this one if the original author returns.