.Net: Bug: Chaining PlugIn functions does not work.

[ddobric]

Assume there are two plugin functions: Load and Save.

 [KernelFunction]
 [Description("Loads the content from the given file.")]
 public async Task<string> LoadAsync([Description("The file to be loaded.")] string file);
     
[KernelFunction]
[Description("Saves the given text to a file.")]
public async Task<string> SaveAsync(
[Description("The text to be saved.")] string text,
[Description("Where to save the text.")] string? file=null)

Both functions are correctly invoked on appropriate prompts.
Imagine now prompt like following one:

"Load the text from file 'abc.txt' and then save it to file cde.txt".

This should orchestrate the first LoadAsync function and then SaveAsync function. This does not really happen. LoadAsync is always invoked, while SaveAsync is sometimes invoked only after several minutes of waiting, if it is invoked at all.

To investigate the issue I have implemented the streaming like shown below:

 var stream = chatCompletionService.GetStreamingChatMessageContentsAsync(
     chatHistory,
     executionSettings: settings,
     kernel: kernel);

 await foreach (var message in stream)
 {
     if (message.Content?.Length > 0)
     {
         result.AppendLine(message.Content);
         Console.Write(message.Content);
     }
     else if (message.Items.Count > 0)
     {
         foreach (var item in message.Items)
         {
             **Console.Write(".");  <------------**
         }
     }
 }

The condition message.Items.Count > 0 is continuously invoked, passing very short chunks of text as arguments.

.. and so on..

Platform

• Language: C#
• Source:

    <PackageReference Include="Microsoft.SemanticKernel.Connectors.AzureOpenAI" Version="1.47.0" />
    <PackageReference Include="Microsoft.SemanticKernel.Connectors.OpenAI" Version="1.47.0" />
    <PackageReference Include="Microsoft.SemanticKernel.Process.Core" Version="1.47.0-alpha" />
    <PackageReference Include="Microsoft.SemanticKernel.Process.LocalRuntime" Version="1.47.0-alpha" />

• AI model: OpenAI:GPT-4 Azure GPT-4o]

[SergeyMenshykh]

Hi @ddobric, I wonder if the file size is causing the issue? In the case of streaming, LLMs stream function call details chunk by chunk, usually starting with the function name in the first chunk followed by chunks with arguments. For example, here's the list of chunks an LLM may stream to call the GetWeatherForCity(string cityName, string currentDateTimeInUtc) function:

{"name":"HelperFunctions-GetWeatherForCity", "arguments": ""}  
{"name": null, arguments:"{\""}  
{"name": null, arguments:"city"}  
{"name": null, arguments:"Name"}  
{"name": null, arguments:"\":\""}  
{"name": null, arguments:"Boston"}  
{"name": null, arguments:"\",\""}  
{"name": null, arguments:"current"}  
{"name": null, arguments:"Date"}  
{"name": null, arguments:"Time"}  
{"name": null, arguments:"In"}  
{"name": null, arguments:"Utc"}  
{"name": null, arguments:"\":\""}  
{"name": null, arguments:"Tue"}  
{"name": null, arguments:","}  
{"name": null, arguments:" "}  
{"name": null, arguments:"29"}  
{"name": null, arguments:" Apr"}  
{"name": null, arguments:" "}  
{"name": null, arguments:"202"}  
{"name": null, arguments:"5"}  
{"name": null, arguments:" "}  
{"name": null, arguments:"09"}  
{"name": null, arguments:":"}  
{"name": null, arguments:"07"}  
{"name": null, arguments:":"}  
{"name": null, arguments:"45"}  
{"name": null, arguments:" GMT"}  
{"name": null, arguments:"\""}

For a 1K file, the number of chunks will be considerably higher, which may impact the time.

[SergeyMenshykh]

@ddobric, consider experimenting with temperature as well. We recently experienced a function-calling integration test that, apart from testing function-calling capabilities, set the temperature to 0.1, sporadically timing out after 30 seconds after calling the first of two functions.

[ddobric]

Thanks for the feedback @SergeyMenshykh .
I almost always use t=0.1 when invoking plug-ins.
Regarding chunk by chunk invocation, I got it. However, I still have a question: Who is trying to invoke the function chunk by chunk? The model or SK?

Whatever it is, it is not the smartest way to do it, and I cannot imagine that this will hold in the future.
Imagine Hello world use-case: "Summarise the paper and translate it to XY". This will always result in a very long stream of chunks.
Even if the model is not invoked via streaming call, the invocation takes forever, mostly resulting in timeouts.

In my opinion, this behaviour = restriction kills the true orchestration capability of models(SK).
Sure, we can work around it by implementing triggering instead of passing results from function to function. In that case, triggered (orchestrated) functions would load the data from somewhere else and always return very short information like a return code.

[SergeyMenshykh]

@ddobric

However, I still have a question: Who is trying to invoke the function chunk by chunk? The model or SK?

The way it works is: the AI model calls functions, and SK invokes them. When the model calls a function, it responds with the function name it wants to use/call and its arguments to the original prompt HTTP request that a client code (SK) had made. When SK receives the response with the function call, it invokes it by looking it up in SK plugins and calling its InvokeAsync method. The function invocation process is almost the same for both streaming and non-streaming scenarios with a small difference.

In a non-streaming scenario, the function call made by the AI model comes as one chunk (all function information: id, name, and arguments are supplied in one response), and SK just invokes it. In a streaming scenario, a function call comes chunk by chunk, and SK buffers/collects them until the last chunk is received before invoking it.

So, the answer to your question "Who is trying to invoke the function chunk by chunk? The model or SK?" is AI models call functions chunk by chunk.

Whatever it is, it is not the smartest way to do it, and I cannot imagine that this will hold in the future.
Imagine Hello world use-case: "Summarise the paper and translate it to XY". This will always result in a very long stream of chunks.

That's how AI models handle function calling today.

Even if the model is not invoked via streaming call, the invocation takes forever, mostly resulting in timeouts.

That would be very interesting to understand what causes it. AI model? HTTP latency? SK? Temperature?

In my opinion, this behaviour = restriction kills the true orchestration capability of models(SK).
Sure, we can work around it by implementing triggering instead of passing results from function to function. In that case, triggered > (orchestrated) functions would load the data from somewhere else and always return very short information like a return code.

Hopefully this problem will be recognized by AI vendors and a corresponding design put in place to address it.