fix(models): accumulate streamed logprobs in place to avoid O(n^2) copies#3762
Merged
Conversation
…pies The Chat Completions stream handler rebuilt the entire accumulated output-text logprobs list on every content delta via `existing_logprobs + output_logprobs`, allocating and copying O(n^2) elements over an n-token stream when logprobs are enabled. Extend the existing list in place instead, keeping the `None`-provider guard. Behavior is unchanged: the converted logprobs are a fresh, unaliased list and the part's `logprobs` is process-local, so the accumulated result is identical while the work is now linear. Add a regression test that streams several logprob-bearing deltas and asserts the final output-text part accumulates every token in order.
seratch
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When Chat Completions streaming is used with logprobs enabled, the stream handler
rebuilt the entire accumulated output-text logprobs list on every content delta:
list + listallocates and copies a brand-new list of the full accumulated lengthfor each delta, so over an n-token stream this is O(n^2) allocation and copying
(the adjacent
... .text += delta.contenton the line above already grows inplace). The accumulator was introduced in #2134 using
+where an in-place extendwas intended.
This extends the existing list in place instead, keeping the
None-provider guardso non-OpenAI providers (for example LiteLLM), which may leave
logprobsunset,still behave exactly as before. The accumulated contents and their order are
unchanged; only the per-delta work drops from O(n) to O(1), making the whole stream
O(n).
The converted logprobs are a fresh, unaliased
list[Logprob]fromChatCmplHelpers.convert_logprobs_for_output_text, and the response part'slogprobsis process-local (initialized to[]at part creation), so extending itin place is safe and observably identical to the previous concatenation.
Test plan
test_stream_response_accumulates_logprobs_across_many_deltasintests/models/test_openai_chatcompletions_stream.py: streams severallogprob-bearing content deltas and asserts the final output-text part accumulates
every token's logprob in order (guarding that the in-place refactor preserves
ordered accumulation). This complements the existing
test_stream_response_includes_logprobs.make format(no changes),make lint, andmake typecheck(pyright clean) allpass on the change.
uv run pytest tests/models/passes (all model tests, including both streaminglogprob tests).
Issue number
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PR