Streaming output is not token-by-token when using tools in ChatClient prompt (Ollama Qwen3:8b) #3373
Description
Please do a quick search on GitHub issues first, there might be already a duplicate issue for the one you are about to create.
If the bug is trivial, just go ahead and create the issue. Otherwise, please take a few moments and fill in the following sections:
Bug description
When using the ChatClient with .tools(new DateTimeTools()) in the nativeChat method, the streaming output is not delivered token-by-token as expected. Instead, the entire response is returned at once after all tool calls are completed. This behavior is different from previous versions, where streaming output was delivered incrementally even when tools were used.
Environment
Spring AI version: 1.0.0
LLM backend: Ollama qwen3:8b
Java version: JDK 17.0.15
OS: Windows
Steps to reproduce
- native stream chat without toolcalling:
public Flux<CustomChatResponse> streamChat(ChatRequest request) {
log.info("Start streaming chat: {}", request.getMessage());
Flux<ChatResponse> output = chatClient.prompt()
.user(request.getMessage())
.stream()
.chatResponse()
.doOnNext(response -> {
log.info("Streaming chat response: {}", response.getResult().getOutput().getText());
});
return responseStreamUtil.handleStreamingResponse(output);
}response:
025-05-29 17:53:49.916 [http-nio-8088-exec-6] INFO o.a.a.service.impl.ChatServiceImpl - Start streaming chat: 今天的日期是什么？
2025-05-29 17:53:50.246 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: <think>
2025-05-29 17:53:50.325 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response:
2025-05-29 17:53:50.336 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 好的
2025-05-29 17:53:50.342 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: ，
2025-05-29 17:53:50.350 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 用户
2025-05-29 17:53:50.359 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 问
2025-05-29 17:53:50.367 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 今天的
2025-05-29 17:53:50.378 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 日期
2025-05-29 17:53:50.387 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 是什么
2025-05-29 17:53:50.396 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Streaming chat response: 。
- native stream chat with toolcalling:
public Flux<ChatResponse> nativeChat(ChatRequest request) {
log.info("Start native chat: {}", request.getMessage());
return chatClient.prompt()
.user(request.getMessage())
.tools(new DateTimeTools())
.stream()
.chatResponse()
.doOnNext(response -> {
log.info("Native chat response: {}", response.getResult().getOutput().getText());
});
}response:
2025-05-29 17:52:39.598 [http-nio-8088-exec-2] INFO o.a.a.service.impl.ChatServiceImpl - Start native chat: What day is it today?
2025-05-29 17:52:43.807 [boundedElastic-1] INFO o.a.a.service.impl.ChatServiceImpl - Native chat response: <think>
好的，用户问今天是什么日子，我需要先确定当前日期。已经调用了getCurrentDateTime函数，返回的日期是2025年5月29日。现在需要把这个日期 以自然的方式告诉用户。
首先，确认日期格式是否正确，确保没有时区错误。返回的日期是北京时间，用户可能位于中国，所以直接提供日期即可。然后，用友好的语气回复，比如“今天是2025年5月29日，星期四。”同时，可以询问用户是否需要关于儿童肥胖的建议，这样能主动提供帮助，符合医生的角色定位。检查是 否有其他需要补充的信息，但用户的问题只问了日期，所以保持回答简洁明了。
</think>
Today is May 29, 2025,星期四 (Thursday). How can I assist you regarding children's obesity management today?
Expected behavior
When calling .stream().chatResponse() with tools enabled, the response should be streamed token-by-token (or chunk-by-chunk) as soon as they are generated, not buffered until the entire response is ready.
The streaming experience should be consistent with the behavior when no tools are used.
Minimal Complete Reproducible example
Please provide a failing test or a minimal complete verifiable example that reproduces the issue.
Bug reports that are reproducible will take priority in resolution over reports that are not reproducible.