-
Notifications
You must be signed in to change notification settings - Fork 155
Closed
Labels
Description
Hi folks!
After I added the tool to the ChatCompletionCreateParameters
, it seems that it only returns a complete function call parameter once.
I use the following code to make the backend server forward the streaming results.
StreamResponse<ChatCompletionChunk> streamResponse = client.chat().completions().createStreaming(params);
Observable<ChatCompletionChunk> observable = Observable.fromIterable(streamResponse.stream()::Iterator);
Flowable<ChatCompletionChunk> flowable = observable.toFlowable(BackpressureStrategy.BUFFER);
Flowable<ServerSentEvent> sse = flowable.map(chunk->{
handleGenerationResult(chunk);
return ServerSentEvent.builder().id(String.valueOf(eventCount).event("ai-response").data(nowContent.toString()).build();
});
Publisher<ServerSentEvent> s = transformable(sse);
ctx.render(ServerSentEvents.builder().build(s));
I want it to output some reasons for the call before the tool is invoked, but it seems to only output the tokens from the first response.
If the params does not contain the tool, the streaming response will proceed as normal.
If I make a second repeated request, it will be returned as a stream.
Is it possible to transmit the tool_call in a streaming manner? I can concatenate the parameters myself.
Thanks a lot! 😀