Fix OpenAiChatModel.stream() to only buffer tool calls.#6288
Closed
ericbottard wants to merge 1 commit into
Closed
Fix OpenAiChatModel.stream() to only buffer tool calls.#6288ericbottard wants to merge 1 commit into
ericbottard wants to merge 1 commit into
Conversation
Previously, the new implementation of OpenAiChatModel.stream() buffered the entire response, which meant that the caller would not receive any tokens until the entire response was received. This change modifies the implementation to only buffer tool calls, allowing the caller to receive tokens as they are generated. Fix spring-projects#5987 Signed-off-by: Eric Bottard <eric.bottard@broadcom.com>
ericbottard
commented
Jun 4, 2026
|
|
||
| Flux<ChatResponse> flux = chatResponses | ||
| Flux<ChatResponse> observedResponses = chatResponses.doOnError(observation::error) | ||
| .doFinally(s -> observation.stop()) |
Member
Author
There was a problem hiding this comment.
@tzolov pay attention to this comment as well, maybe this needs changing.
tzolov
added a commit
that referenced
this pull request
Jun 4, 2026
Previously the entire response was buffered before emitting any tokens. The new implementation uses bufferUntil(finish_reason=TOOL_CALLS) so only tool-call chunks are held back for aggregation; all other tokens stream immediately. Also removes dead code and broken tests left over after the internal tool execution loop was deleted from OpenAiChatModel. Closes #5987 Signed-off-by: Eric Bottard <eric.bottard@broadcom.com> Signed-off-by: Christian Tzolov <christian.tzolov@broadcom.com> Co-authored-by: Christian Tzolov <christian.tzolov@broadcom.com>
Contributor
|
Rebased, fix tests, squashed and merged at 12f5cbe |
tzolov
added a commit
that referenced
this pull request
Jun 4, 2026
Follows-up #6288 Signed-off-by: Christian Tzolov <christian.tzolov@broadcom.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, the new implementation of OpenAiChatModel.stream() buffered
the entire response, which meant that the caller would not receive
any tokens until the entire response was received. This change modifies
the implementation to only buffer tool calls, allowing the caller to
receive tokens as they are generated.
Fix #5987