.Net Bug: SequentialOrchestration does not invoke IPromptRenderFilter or IFunctionInvocationFilter

[Daniellled]

Describe the bug
When using SequentialOrchestration the IPromptRenderFilter or IFunctionInvocationFilter objects associated with the ChatCompletionAgent's kernel object are not getting invoked.

To Reproduce
Steps to reproduce the behavior:

1. Create a SequentialOrchestration (or just follow the example here
2. Implement and add an IFunctionInvocationFilter to the kernel
3. Implement and add an IPromptRenderFilter to the kernel
4. Invoke the SequentialOrchestration

Expected behavior
Since the SequentialOrchestration is simply a collection of kernel objects. The IPromptRenderFilter and IFunctionInvocationFilter objects associated with those kernel objects should get invoked when the runtime switches to those agents.

Screenshots
If applicable, add screenshots to help explain your problem.

Platform

• Language: C#
• Source: Microsoft.SemanticKernel Version="1.58.0
• AI model: llama3.2
• IDE: Visual Studio
• OS: Windows

Additional context
Sample code.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents;
using Microsoft.SemanticKernel.Agents.Orchestration;
using Microsoft.SemanticKernel.Agents.Orchestration.Sequential;
using Microsoft.SemanticKernel.Agents.Runtime.InProcess;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
#pragma warning disable SKEXP0070 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
#pragma warning disable SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
namespace SequentialTest
{
    internal class SQ
    {
        public static async Task<string> RunSequentialOrchestrationAsync(string input, string model = "llama3.2")
        {
            ChatCompletionAgent analystAgent =
           CreateAgent(
               name: "Analyst",
               instructions:
               """
                You are a marketing analyst. Given a product description, identify:
                - Key features
                - Target audience
                - Unique selling points
                - Keep your response short (around 50 words)
                """,
               description: "A agent that extracts key concepts from a product description.",
                    model);
            ChatCompletionAgent writerAgent =
                CreateAgent(
                    name: "copywriter",
                    instructions:
                    """
                You are a marketing copywriter. Given a block of text describing features, audience, and USPs,
                compose a compelling marketing copy (like a newsletter section) that highlights these points.
                Output should be short (around 150 words), output just the copy as a single text block.
                """,
                    description: "An agent that writes a marketing copy based on the extracted concepts.",
                    model);
            ChatCompletionAgent editorAgent =
                CreateAgent(
                    name: "editor",
                    instructions:
                    """
                You are an editor. Given the draft copy, correct grammar, improve clarity, ensure consistent tone,
                give format and make it polished. Output the final improved copy as a single text block. Output should be short (around 150 words).
                """,
                    description: "An agent that formats and proofreads the marketing copy.",
                    model);


            SequentialOrchestration orchestration =
           new(analystAgent, writerAgent, editorAgent)
           {
               ResponseCallback = ResponseCallback,
           };


            // Start the runtime
            InProcessRuntime runtime = new();
            await runtime.StartAsync();

            // Run the orchestration
            Console.ForegroundColor = ConsoleColor.Blue;
            Console.WriteLine($"\n# INPUT: {input}\n");
            Console.WriteLine("**************************************************************************************");
            Console.ResetColor();
            OrchestrationResult<string> result = await orchestration.InvokeAsync(input, runtime);
            string text = await result.GetValueAsync(TimeSpan.FromSeconds(25000));
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine("**************************************************************************************");
            Console.WriteLine($"\n# RESULT: {text}");
            Console.ResetColor();

            await runtime.RunUntilIdleAsync();
            return text;
        }

        public static async Task<string> RunKernelDirectlyAsync(string prompt, string model = "llama3.2")
        {
            Kernel kernel = CreateKernel(model);
            var results = await kernel.InvokePromptAsync(
                        promptTemplate: prompt);
            Console.WriteLine($"\n# RESULT: {results}");
            return results.ToString();
        }

        private static ValueTask ResponseCallback(ChatMessageContent response)
        {
            Console.ForegroundColor = ConsoleColor.Green;
            Console.WriteLine($"\n# RESPONSE CALLBACK:{response?.AuthorName}: {response?.Role}, {response?.Content}");
            Console.ResetColor();
            return default;
        }

        private static ChatCompletionAgent CreateAgent(string name, string instructions, string description, string model)
        {
            return new ChatCompletionAgent()
            {
                Name = name,
                Description = description,
                Instructions = instructions,
                Kernel = CreateKernel(model)
            };
        }

        private static Kernel CreateKernel(string model)
        {
            IKernelBuilder kernelBuilder = Kernel.CreateBuilder();

            kernelBuilder.AddOllamaChatCompletion(
                modelId: model,
                httpClient: CreateHttpClient(new Uri("http://localhost:7869/v1"), 5000),
                serviceId: "OllamaChatCompletion"
            );

            kernelBuilder.Services.AddSingleton<IPromptRenderFilter, PromptFilter>();
            kernelBuilder.Services.AddSingleton<IFunctionInvocationFilter, FunctionFilter>();
            kernelBuilder.Services.AddSingleton<IAutoFunctionInvocationFilter, AutoFunctionFilter>();

            return kernelBuilder.Build();
        }

        private static HttpClient CreateHttpClient(Uri baseAddress, long minutes = 10)
        {
            return new HttpClient
            {
                Timeout = TimeSpan.FromMinutes(minutes),
                BaseAddress = baseAddress,
            };
        }

        
    }
}
#pragma warning restore SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
#pragma warning restore SKEXP0070 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

The main method to execute the two test

// The filters are not executed
await SequentialTest.SQ.RunSequentialOrchestrationAsync("An eco-friendly stainless steel water bottle that keeps drinks cold for 24 hours");

Console.WriteLine("**************************** Press any key ****************************");
Console.Read();

// The filters are executed
await SequentialTest.SQ.RunKernelDirectlyAsync("Why is the sky blue?");

Console.WriteLine("**************************** DONE ****************************");
Console.Read();

Filters

using Microsoft.SemanticKernel;

namespace SequentialTest
{
    public class FunctionFilter : IFunctionInvocationFilter
    {
        public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
        {
            Console.WriteLine($"FunctionInvoking - {context.Function.PluginName}.{context.Function.Name}");

            await next(context);

            Console.WriteLine($"FunctionInvoked - {context.Function.PluginName}.{context.Function.Name}");
        }
    }
}

using Microsoft.SemanticKernel;

namespace SequentialTest
{
    public class AutoFunctionFilter : IAutoFunctionInvocationFilter
    {
        public async Task OnAutoFunctionInvocationAsync(AutoFunctionInvocationContext context, Func<AutoFunctionInvocationContext, Task> next)
        {
            Console.WriteLine($"FunctionInvoking - {context.Function.PluginName}.{context.Function.Name}");

            await next(context);

            Console.WriteLine($"FunctionInvoked - {context.Function.PluginName}.{context.Function.Name}");
        }
    }
}

using Microsoft.SemanticKernel;

namespace SequentialTest
{
    internal class PromptFilter : IPromptRenderFilter
    {
        public async Task OnPromptRenderAsync(PromptRenderContext context, Func<PromptRenderContext, Task> next)
        {
            Console.WriteLine($"PromptRendering - {context.Function.PluginName}.{context.Function.Name} {context.RenderedPrompt}");

            await next(context);

            Console.WriteLine($"PromptRendered - {context.Function.PluginName}.{context.Function.Name} {context.RenderedPrompt}");
        }
    }
}

[crickman]

@Daniellled - Thanks for asking this question.

Since your orchestrated agents aren't using any type of kernel function or prompt-template, I don't believe there's anything that would trigger either of your filters. If you add a plugin/tools to one of the agent kernels I'd expect the filter function to be invoked.

The code where you are explicitly calling Kernel.InvokePromptAsync doesn't correspond with how an agent executes.

Whether or not the agent is used within an orchestration has no impact on the reported behavior.

Here's a sample that shows using a filter with agents:

https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/ChatCompletion_FunctionTermination.cs

[Daniellled]

@crickman - Thank you for the quick response.

When I explicitly call the Kernel.InvokePromptAsync function I see the following happen.

1. IFunctionInvocationFilter - InvokePromptAsync_7faa1edd841541a9a90b6bebfb85d2b1 (Before firing the next method)
2. IPromptRenderFilter - InvokePromptAsync_7faa1edd841541a9a90b6bebfb85d2b1 (Before firing the next method)
3. IPromptRenderFilter - InvokePromptAsync_7faa1edd841541a9a90b6bebfb85d2b1 RenderPrompt is set (After firing the next method)
4. IFunctionInvocationFilter - InvokePromptAsync_7faa1edd841541a9a90b6bebfb85d2b1 RenderPrompt is set (After firing the next method)

I would have expected similar logic and flow when using the orchestrated agents. For example, when I transition from agent to agent the IFunctionInvocationFilter would be invoked followed by the IPromptRenderFilter.

When using an orchestrated agent is it possible to have something like a IPromptRenderFilter? Or is the only way possible is to modify the response via the ResponseCallback? Do all Orchestrators have a ResponseCallback?

My end goal was to have a collection of IFunctionInvocationFilter and IPromptRenderFilter objects that I could share between my different kernels and agents. I could also test them using the Kernel.InvokePromptAsync function and know that they would work in the orchestration scenario (the orchestrations take longer to run :( ) - Hence why I was hoping things worked the same between Kernel.InvokePromptAsync and SequentialOrchestration.InvokeAsync.

[crickman]

Understood. Invoking an agent result in an entirely different execution path and is not equiavlent to Kernel.InvokePromptAsync. An orchestration isn't directly responsible for any tooling or prompts and has no intersection with filtering. Filters only apply at the agent level.

[Daniellled]

Understood that the orchestration doesn't do anything related to tooling or prompts. If my understanding is correct, it is simply a scheduling (or orchestrator) for the agents assigned to it. Passing messages from one agent to another. Now, if filters only apply at the agent level, and the agent is made up of a kernel why do the prompt filters not fire on the individual agents? What would be the equivalent?

[crickman]

Because the agent isn't a prompt-function and you aren't calling any prompt-functions. If your agent calls a plugin function, you'll see the function filter fire.