Memory Management and Plugin: MagenticOrchestration

[mjnong]

Problem

I’m experimenting with the new orchestration APIs in Semantic Kernel and I’ve run into a few limitations:

1. No history insertion/retrieval
   There doesn’t seem to be any way to insert or fetch conversation history (e.g., by thread or session ID). Is this functionality available, or am I missing something?

2. No plugin support at the Orchestrator level
   While agents can register plugins (e.g., TimePlugin()), I’d love to be able to attach plugins directly to the Orchestrator so they could be shared across all orchestrations.

3. No streaming output
   I haven’t found a way to stream generated content back to the caller. Is streaming support on the roadmap?

---

What I’m looking for

• Memory / Context management

  • Ability to persist and retrieve arbitrarily-keyed context or history for an orchestration.

• Plugins inside the Orchestrator

  • A mechanism to register and invoke plugins (e.g., logging, timers, data enrichment) directly on the orchestrator instance.

• Streaming responses

  • Support for chunked or real-time streaming of content generated by the orchestration.

---

If any of these are already supported or there are recommended workarounds, I’d appreciate pointers. Otherwise, are there plans to add these capabilities?

Thanks!
— Marcus

[moonbox3]

Thank you for the feedback, @mjnong. Adding @TaoChenOSU.

[mjnong]

I'm also wondering whether or not this is working with AzureAIAgents because I didn't get it to work.

[seizeuniverse]

I am also interested how to make it work with AzureAIAgents and + history context into this orch.

[TaoChenOSU]

Hi @mjnong,

Thank you for your questions!

Here are the answers:

1. The conversation history is not available via an API. However, you can observe the conversation as it is happening via the agent_response_callback and now the streaming_agent_response_callback. You can display the conversation as it goes or save the message for later processing. Please note that the callbacks only provide the agents' final responses. The agents do not share one history as the agent's own history can contain tool calls which are shared to other agents within an orchestration. Please see example here and here. We are working on ways for developers to retrieve the individual thread/chat history.
2. Inserting history to individual agents within an orchestration isn't supported but you can invoke the orchestration with a list of messages.
3. At the orchestrator level, you cannot insert plugins as the orchestrator isn't an agent. I am not sure what you meant by sharing plugins across orchestrations. Could you provide more context?
4. Streaming support has been added and is available since 1.32.2. Sample is here.

Please don't hesitate to reach out if you have more questions.

[TaoChenOSU]

I'm also wondering whether or not this is working with AzureAIAgents because I didn't get it to work.

Yes, it should work with AzureAIAgent. Could you provide more context so that we can assist you further?

[TaoChenOSU]

I am also interested how to make it work with AzureAIAgents and + history context into this orch.

Hi!

Yes, the orchestrations are built on top of the Agent abstraction so it should work with all agent types in SK.

As I mention in a previous response, it's currently not supported to inject chat history into individual agents in an orchestration now but you can invoke an orchestration with a list of chat messages.

[mjnong]

I'm also wondering whether or not this is working with AzureAIAgents because I didn't get it to work.

Yes, it should work with AzureAIAgent. Could you provide more context so that we can assist you further?

I believe the issue stem from me having incorrect versioning. I figured out that it was not just enough to just update semantic-kernel pkg to latest version. There are a lot more dependencies.

[moonbox3]

@mjnong if you're using SK Python >= 1.32.1, the base installation pip install semantic-kernel will now handle adding required packages to use the AzureAIAgent without any extras installs (previously it was pip install semantic-kernel[azure]).

You can see this

• on our release notes: https://github.com/microsoft/semantic-kernel/releases/tag/python-1.32.1
• in our docs: https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-types/azure-ai-agent?pivots=programming-language-python#preparing-your-development-environment
• in our Foundry GA Migration guide: https://learn.microsoft.com/en-us/semantic-kernel/support/migration/azureagent-foundry-ga-migration-guide?pivots=programming-language-python#ga-foundry-project-1

[mjnong]

Hi @mjnong,

Thank you for your questions!

Here are the answers:

1. The conversation history is not available via an API. However, you can observe the conversation as it is happening via the agent_response_callback and now the streaming_agent_response_callback. You can display the conversation as it goes or save the message for later processing. Please note that the callbacks only provide the agents' final responses. The agents do not share one history as the agent's own history can contain tool calls which are shared to other agents within an orchestration. Please see example here and here. We are working on ways for developers to retrieve the individual thread/chat history.

I see, that make sense. History management would have to be done elsewhere in that case I guess. I do believe that if the orchestration is making use of AzureAIAgent then when an agent is interacting it will create a thread which will be possible to view in AI Foundry, correct?

2. Inserting history to individual agents within an orchestration isn't supported but you can invoke the orchestration with a list of messages.
3. At the orchestrator level, you cannot insert plugins as the orchestrator isn't an agent. I am not sure what you meant by sharing plugins across orchestrations. Could you provide more context?
   Got it, my current setup uses an agent as orchestrator similar to yours example of triage agent.

5. Streaming support has been added and is available since 1.32.2. Sample is here.>

Great news!

Please don't hesitate to reach out if you have more questions.

Do you guys have any data on how the orchestration behaves depending on how many messages one would include in the request? For example if I included all the messages from a long conversation will that give a worse result compared to just including the last 5-10 messages as context?

Another question is how the design regarding the InProcessRuntime is intended to be used. E.g. if I provide an API to access this orchestration and I want to be able to handle many requests is the idea that I should run a process per API call. It is the same with the Kernel?

[anktsrkr]

Hi @TaoChenOSU, was thinking something like below -

 public async ValueTask ResponseCallback(ChatMessageContent response)
 {
     var chatHistory = kernel.GetRequiredService<IChatHistoryProvider>();
     await chatHistory.CommitAsync(response);
 }

var orchestration = kernel.GetRequiredService<HandoffOrchestration<ChatHistory, string>>();
   
ChatHistory chatMessages = await kernel.Services.GetRequiredService<IChatHistoryProvider>().GetHistoryAsync();
chatMessages.Add(new ChatMessageContent(AuthorRole.User, request.content));
   
InProcessRuntime runtime = new();
await runtime.StartAsync();

var result = await orchestration.InvokeAsync(chatMessages, runtime);

var response = await result.GetValueAsync();

That seems works, but these more like Orchestration level memory, how do we access specific agent's memory.