Auto function invocation filter does not show information about all functions which will be invoked(list is empty)

[EVENFLOW212]

Hi

We are using SK - 1.13. We follow the following process.

register 4 native plugins using kernel.ImportPluginFromObject

RegisterNativePlugins();

FunctionInvocationFilters.Add(new FunctionFilter());
AutoFunctionInvocationFilters.Add(new AutoFunctionInvocationFilter());

ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions

chat history has system prompt and user prompt

GetStreamingChatMessageContentsAsync is being used

in FunctionFilter

OnFunctionInvoking(context);
await next(context);
await OnFunctionInvoked(context);

in OnAutoFunctionInvocationAsync in AutoFunctionInvocationFilter

the chat history has system prompt and user prompt
The user prompt should defintely invoke 2 native plugin functions

var functionCalls = FunctionCallContent.GetFunctionCalls(context.ChatHistory.Last());

the function calls list from the above call is empty and no way to determine the information on all the function calls which will be invoked and hence cannot use context.Terminate = true since the is list of function calls is empty

If I set the context.Terminate = true the last function(of the 2 plugins) will "not" be invoked. If I set context.Terminate = false both functions are invoked but dont know when to set context.Terminate = true since we need to set the context.Terminate = true based on certain condition. Hence we need the information on all the function calls which will be invoked.

Please help

Thanks

[sophialagerkranspandey]

We've created an issue from this discussion question.

[dmytrostruk]

Hi @EVENFLOW212 !
I added an example with auto function invocation filter that gets an information about all function calls in this PR:
#6800

In my example, I used gpt-3.5-turbo-1106 model, which supports parallel function calling, so I can see all functions to be called before actual execution, because the model supports that, and my functions don't depend on each other. Maybe you could try the example in the PR above and try to apply your use-case on top of it.

Here is a list of supported models for function calling:
https://platform.openai.com/docs/guides/function-calling/supported-models

[EVENFLOW212]

Hi @dmytrostruk

I have used the example that you have shown above and it does not seem work as expected, where the dependent function calls are showing empty or null arguments.

_kernel.ImportPluginFromFunctions("CityNamePlugin",
[
_kernel.CreateFunctionFromMethod(() => "Boston", "GetCityName", "Retrieves the city name."),
]);
_kernel.ImportPluginFromFunctions("CityWeatherPlugin",
[
_kernel.CreateFunctionFromMethod((string cityName) =>
cityName switch
{
"Boston" => "61 and rainy",
"London" => "55 and cloudy",
"Miami" => "80 and sunny",
"Paris" => "60 and rainy",
"Tokyo" => "50 and sunny",
"Sydney" => "75 and sunny",
"Tel Aviv" => "80 and sunny",
_ => "31 and snowing",
}, "Get_Weather_For_City", "Gets the current weather for the specified city"),
]);

The kernel plugins info is as below

We are using ToolCallBehavior.EnableKernelFunctions, hence manual function calling as explained before and our setting is as below

OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.EnableKernelFunctions };

kernel.GetRequiredService();

ChatMessageContent result = await chat.GetChatMessageContentAsync(chatHistory, settings, kernel);

Function calls details below - here the arguments to functions don't have any value

the chat history is as below

Yes the dependent calls are showing in IEnumerable function calls but without have proper arguments, this example above is not helpful.

If I change the example above to provide the plugins and the functions below, we only get one function call at a time

prompt - For France get the city name and the weather for it

_kernel.ImportPluginFromFunctions("CityNamePlugin",
[
_kernel.CreateFunctionFromMethod((string country) =>
country switch
{
"USA" => "Boston",
"France" => "Paris",
"Japan" => "Tokyo",
_ => "Boston",
}, "GetCityName", "Retrieves the city name for the specified country"),
]);

_kernel.ImportPluginFromFunctions("CityWeatherPlugin",
[
_kernel.CreateFunctionFromMethod((string cityName) =>
cityName switch
{
"Boston" => "61 and rainy",
"London" => "55 and cloudy",
"Miami" => "80 and sunny",
"Paris" => "60 and rainy",
"Tokyo" => "50 and sunny",
"Sydney" => "75 and sunny",
"Tel Aviv" => "80 and sunny",
_ => "31 and snowing",
}, "Get_Weather_For_City", "Gets the current weather for the specified city"),
]);

I have actually now tried with simple example and it does not show multiple plugins/functions on the the first call to FunctionCallContent.GetFunctionCalls(result); for dependent plugins.

Is it possible to provide an example of dependent functions calls, with native plugin, open ai function calling mechanism that actually works?

Thanks

[dmytrostruk]

We are using ToolCallBehavior.EnableKernelFunctions, hence manual function calling as explained before

@EVENFLOW212 Your initial issue description under this discussion thread mentions ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions and problem with auto function invocation filter. Small note that auto function invocation filter is not working with ToolCallBehavior.EnableKernelFunctions because EnableKernelFunctions is manual function calling scenario.

Yes the dependent calls are showing in IEnumerable function calls but without have proper arguments, this example above is not helpful.

From your screenshots it looks like you received 2 function calls on the first request. I assume the reason why second function does not contain value for cityName argument is because you didn't invoke first function yet to get that city name (in your case Boston). AI model didn't know which value to provide for this argument, so it provided just empty string. To get valid argument value, in manual scenario, you need to call that function first, like in this example:

semantic-kernel/dotnet/samples/Concepts/FunctionCalling/OpenAI_FunctionCalling.cs

Lines 103 to 108 in d22ee5a

	foreach (FunctionCallContent functionCall in functionCalls)
	{
	try
	{
	// Invoking the function
	FunctionResultContent resultContent = await functionCall.InvokeAsync(kernel);

I have actually now tried with simple example and it does not show multiple plugins/functions on the the first call to FunctionCallContent.GetFunctionCalls(result); for dependent plugins.

Sometimes the model responds with all functions to call on the first request and sometimes multiple requests are needed. It depends on actual model, which is used, nature of functions, their arguments, system and user prompt.

However, I don't think it's possible to have all dependent functions to call on the first request with pre-populated function arguments. Your second function depends on result from first one. The model cannot know what the result of your first function would be, until you call that function, get a result and provide that result back to the model. In automatic function calling scenario, Semantic Kernel calls that function for you and provides that result to the model, so it can properly call second function with valid arguments. In manual scenario, you have more control, but you need to call that function on your side, otherwise there is no way how to get an input for second function, if it depends on output from the first one.

Thanks.

[EVENFLOW212]

Hi @dmytrostruk

However, I don't think it's possible to have all dependent functions to call on the first request with pre-populated function arguments. Your second function depends on result from first one. The model cannot know what the result of your first function would be, until you call that function, get a result and provide that result back to the model. In automatic function calling scenario, Semantic Kernel calls that function for you and provides that result to the model, so it can properly call second function with valid arguments. In manual scenario, you have more control, but you need to call that function on your side, otherwise there is no way how to get an input for second function, if it depends on output from the first one.

Please forgive the mixing up of manual vs auto. I thought the other issue I had raised was being discussed here as well. Also forgive the "dependent functions" point you are making, got that part.

Sometimes the model responds with all functions to call on the first request and sometimes multiple re

This is our main concern, if possible to have the function calls deterministic? I have seen indeterministic behavior even with 2 functions dependent on each other. Sometimes it available with all the function calls on the first request and sometimes its multiple request. So I am assuming this will always be indeterministic, and it will not matter whether its manual or auto function calls, Am I right?

Thanks

[dmytrostruk]

Sometimes it available with all the function calls on the first request and sometimes its multiple request. So, I am assuming this will always be indeterministic, and it will not matter whether its manual or auto function calls, Am I right?

@EVENFLOW212 That's correct, it doesn't depend on whether it's manual or auto function calls. It really depends on your function's structure, system and user prompt you are sending to the model etc. I would recommend adding as many instructions in system prompt as possible to make it more reliable, stick with the same model (because if you change the model, there may be some differences in responses) and of course add some testing. You can also experiment with temperature, add more description to functions and their arguments. As soon as you find a good setup which makes responses deterministic for your case, you should use it in your application. But if you will change any of those parameters later, you need to be prepared that the model could start to respond differently.

But once you find a good setup of your prompts and functions that cover your case, it should work pretty reliable. In SK infrastructure, we have integration tests which we run on regular basis, and it works well. Sometimes we need to slightly modify our inputs by specifying even more details in instructions, but in most cases, it responds in the same way.

[EVENFLOW212]

thanks for all your help @dmytrostruk