How to customize response handlers？

[niltor]

Problems I encountered

When I use Kernel.InvokePromptStreamingAsync(prompt);, I have errors:

System.ArgumentNullException: Value cannot be null. (Parameter 'value')
         at Azure.AI.OpenAI.ChatRole.op_Implicit(String value)
         at Azure.AI.OpenAI.StreamingChatCompletionsUpdate.DeserializeStreamingChatCompletionsUpdates(JsonElement element)
         at Azure.Core.Sse.SseAsyncEnumerator`1.EnumerateFromSseStream(Stream stream, Func`2 multiElementDeserializer, CancellationToken cancellationToken)+MoveNext()
         at Azure.Core.Sse.SseAsyncEnumerator`1.EnumerateFromSseStream(Stream stream, Func`2 multiElementDeserializer, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext()
         at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

Because the LLM what I used return data like:

data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":" today","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null}

data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"?","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null}

data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"","function_call":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":6030235,"model":"DeepSeek-LLM-67B-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":{"prompt_tokens":16,"completion_tokens":10,"total_tokens":26}}

data: [DONE]

All the previous data can be obtained. Please pay attention to the end mark (data: [DONE]) of the last line.

This is not a json format. The parsing may return null and an error may be reported when using it later.

Solution

Is it possible to customize the parsing process so that I can perform special processing on the data returned by different language models and finally process it into a unified format?

[dmytrostruk]

@niltor When you configure AI connector, it's possible to pass custom HttpClient. In your custom HttpClient, you can intercept response with raw data and then modify it to your needs, before it is parsed. Is it something that can cover your scenario?

[niltor]

It sounds feasible, I will look into the relevant documentation to see how to use a custom connector.
The problem is that I still want to use InvokePromptStreamingAsync. It would be great if there are relevant documents or examples. The main problem is how to associate it with SK after the custom handler is processed. Is it necessary to return a new HttpResponseMessage object?

[dmytrostruk]

@niltor Here is an example how to inject custom HttpClient in connector:
https://github.com/microsoft/semantic-kernel/blob/2530367c17c71c6210afc4b890f97847ac557928/dotnet/samples/Concepts/ChatCompletion/Connectors_CustomHttpClient.cs

It's still will be possible to use InvokePromptStreamingAsync method. We don't have examples for such scenario yet, but we do have unit tests that you can use as a reference:

semantic-kernel/dotnet/src/Connectors/Connectors.UnitTests/OpenAI/ChatCompletion/AzureOpenAIChatCompletionServiceTests.cs

Lines 432 to 451 in 2530367

	public async Task GetStreamingChatMessageContentsWorksCorrectlyAsync()
	{
	// Arrange
	var service = new AzureOpenAIChatCompletionService("deployment", "https://endpoint", "api-key", "model-id", this._httpClient);
	using var stream = new MemoryStream(Encoding.UTF8.GetBytes(OpenAITestHelper.GetTestResponse("chat_completion_streaming_test_response.txt")));
	this._messageHandlerStub.ResponsesToReturn.Add(new HttpResponseMessage(HttpStatusCode.OK)
	{
	Content = new StreamContent(stream)
	});
	// Act & Assert
	var enumerator = service.GetStreamingChatMessageContentsAsync([]).GetAsyncEnumerator();
	await enumerator.MoveNextAsync();
	Assert.Equal("Test chat streaming response", enumerator.Current.Content);
	await enumerator.MoveNextAsync();
	Assert.Equal("stop", enumerator.Current.Metadata?["FinishReason"]);
	}

In this test we use .txt file with mocked streaming response with following format:

semantic-kernel/dotnet/src/Connectors/Connectors.UnitTests/OpenAI/TestData/chat_completion_streaming_test_response.txt

Lines 1 to 5 in 2530367

	data: {"id":"chatcmpl-96fqQVHGjG9Yzs4ZMB1K6nfy2oEoo","object":"chat.completion.chunk","created":1711377846,"model":"gpt-4-0125-preview","system_fingerprint":"fp_a7daf7c51e","choices":[{"index":0,"delta":{"content":"Test chat streaming response"},"logprobs":null,"finish_reason":null}]}
	data: {"id":"chatcmpl-96fqQVHGjG9Yzs4ZMB1K6nfy2oEoo","object":"chat.completion.chunk","created":1711377846,"model":"gpt-4-0125-preview","system_fingerprint":"fp_a7daf7c51e","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
	data: [DONE]

You can clone the repo, try to modify that response based on your needs, and run the test to see how it works. You can also update DelegatingHandler _messageHandlerStub to process your custom response, which should help you to achieve your scenario. In DelegatingHandler, you will need to return new HttpResponseMessage object with your modified data, here is an example:

semantic-kernel/dotnet/src/SemanticKernel.UnitTests/HttpMessageHandlerStub.cs

Line 47 in 2530367

return await Task.FromResult(this.ResponseToReturn);