.Net: Find Associated Tool(s) Called

[aherrick]

I'm trying to figure out the cleanest way to figure out what tools were invoked in a call. Here is what I've come up with but there has to be a better way?

using System.ComponentModel;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;

var kernel = Kernel
    .CreateBuilder()
    .AddAzureOpenAIChatCompletion(
        deploymentName: "gpt-4o",
        endpoint: "",
        apiKey: ""
    )
    .Build();

kernel.Plugins.AddFromType<WeatherPlugin>();

var chatHistory = new ChatHistory();

chatHistory.AddUserMessage("whats the weather");

var promptExecutionSettings = new PromptExecutionSettings
{
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
};

var chatService = kernel.GetRequiredService<IChatCompletionService>();
var result = await chatService.GetChatMessageContentAsync(
    chatHistory,
    promptExecutionSettings,
    kernel
);

Console.WriteLine(result.Content);

var plugins = GetPluginsUsed(chatHistory);
foreach (var plugin in plugins)
{
    Console.WriteLine(plugin);
}

Console.ReadLine();

static List<string> GetPluginsUsed(ChatHistory chatHistory)
{
    // Find the index of the most recent user message
    int lastUserMessageIndex = -1;
    for (int i = chatHistory.Count - 1; i >= 0; i--)
    {
        if (chatHistory[i].Role == AuthorRole.User)
        {
            lastUserMessageIndex = i;
            break;
        }
    }

    // Extract only tool messages that appear AFTER the last user message
    var toolMessages = chatHistory
        .Skip(lastUserMessageIndex + 1) // Start from the message after the user's last input
        .Where(msg => msg.Role == AuthorRole.Tool)
        .SelectMany(toolMsg =>
            toolMsg
                .Items.OfType<FunctionResultContent>()
                .Select(x => $"({x.PluginName}: {x.FunctionName})")
        )
        .ToList();

    return toolMessages;
}

public class WeatherPlugin
{
    // Hard coded weather data
    private readonly string[] weatherData =
    [
        "Sunny, 15°C",
        "Cloudy, 12°C",
        "Rainy, 10°C",
        "Snowy, -2°C",
    ];

    private readonly Random random = new();

    [KernelFunction, Description("Get weather information")]
    public string GetWeather()
    {
        // Get random weather data
        int index = random.Next(weatherData.Length);
        return weatherData[index];
    }
}

[moonbox3]

Adding @SergeyMenshykh to help.

[aherrick]

@moonbox3 @SergeyMenshykh additionally, can data be pulled as to why the LLM picked which plugin(s)?

[romainsemur]

Hi, maybe you can use https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/filters?pivots=programming-language-csharp#function-invocation-filter

[SergeyMenshykh]

Hi @aherrick, I can think of a few ways to get called functions:

1. Traverse chat history up to the first user message and collect all items of FunctionCallContent type along the way. That's what your code snipper does.
2. Register either a function invocation filter or an auto function invocation filter, as pointed out by @romainsemur. Both filters will be called per function call.
3. Enable manual function invocation as described here and here, where the GetChatMessageContentAsync method gives you full access to the content returned by the AI model, including function calls. However, in this case, you will have to invoke the functions yourself.

If you just need to identify called functions, go with option 1 or option 2 because it may be easier from an implementation point of view. If you need a little bit more control than just identifying which function was called, such as terminating function invocation or changing the invocation results, go with option 2. If you need full control over interpreting function calls, such as getting a list of functions called, deciding whether to invoke them or not, changing arguments and results, etc., go with option 3.

[aherrick]

@SergeyMenshykh thanks for the detailed response! my thought was that the "result" object that is returned from the GetChatMessageContentAsync would simply have a list of functions invoked. But I'm ok traversing for now as I need the plugins used inline with the call, so a Function Filter doesn't really help me.

[SergeyMenshykh]

The issue has been resolved. If you have any other questions or need more help, please reach out.