[.Net] Bring Semantic-Kernel to AutoGen && Propose a new message abstraction to support multi-modal/parallel function call && potential breaking change from openai api

[LittleLittleCloud]

Update on 2024/03/03

This PR is ready to be reviewed and merged, apology for creating such a large PR and I wish to be able to separate it into small pieces. But considering that this PR introduces several API break changes and it's probably better to break them all together in one PR.

So, here's a few scenarios with the new IMessage abstraction and built-in message types.

Scenario 1: type-matching in processing messages

Below scenario shows how to use type-matching to convert an image message to text message so it can be passed to an agent that only understand text.

var chatHistory = [
   new TextMessage("Hey explain the image below"),
   new ImageMessage(image: xxx),
]

var imageAgent // a client or agent that can understand image
var textAgent // an agent that can only process text information
var agent = textAgent
    .RegisterMiddleware(async (messages, option, innerAgent, ct) => {
    var msgs = messages.Select(msg => 
        {
            if (msg is ImageMessage) // use imageAgent to explain what's inside image and return a text message)
            else return msg;
       }

   return await innerAgent.GenerateReplyAsync(msgs);
   });

var reply = agent.SendAsync(chatHistory);

Original post

Why are these changes needed?

This PR proposes a new abstraction for the message type in AutoGen.Net, plus some built-in Message classes. The general idea behind this is to avoid crafting a general, all-in-one message for all agents and replace it with multiple simple, specific messages. This PR is also required by #1647

The original Message type is kept for backward compatibility, the new added built-in Messages are

• TextMessage: plain text message
• ImageMessage: image message
• MultiModaMessage: A combination of [text message, image message, ...]
• ToolCallMessage: a message that contains one or several tool calls
• TooCallResultMessage: a tool/function message that contains one or several tool calls result
• AggregateMessage<M1, M2>: an aggregate message of M1 + M2
• MessageEnvelope: a message wrapper for an arbitrary object

Agents can decide which messages they can accept and consume. For example, for dalle agent, would probably support TextMessage and generate ImageMessage. For GPT agents, it would support TextMessage, ImageMessage, MultiModalMessage, and so on....

If an agent workflow, like two-agent chat, the messages supported by both agents are different, there will be no magic happening and the conversation will default to an error. However, that behavior can be overwritten with middleware. For example, if a multi-modal message flows into gpt-3.5-turbo agent, the default behavior would be a 400 error from openai side, but user can use middleware to perform a check over the messages and return a default message instead of a multi-modal message detected.

So, anyway, I feel like myself typing too much and thanks for your patience if you read here. Please review the PR and leaves feedback, greatly appreciated!

Related issue number

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[LittleLittleCloud]

Maybe we need to break it into small pieces

[ekzhu]

Code-wise we can break it up into a chain of message handlers. Though functionality-wise we don't need to do it now. Unless you have some other considerations?

[LittleLittleCloud]

This might be a better implementation but yes, functionality-wise no need to change that code. Other than we can probably make it internal (no need to expose this as a public method)
https://github.com/microsoft/autogen/pull/1676/files#diff-99bde4c9640d0c773927d1462185e5bca0dae555ad22df353645469b755c921aR164

[LittleLittleCloud]

The use example for TextMessage, MultiModalMessage and so on.

[LittleLittleCloud]

GPTAgent allows you to use messages from the original aoai sdk as well

[LittleLittleCloud]

@matthewbolanos Any suggestion on making a better example with AutoGen + SemanticKernel?

[LittleLittleCloud]

@matthewbolanos and @SergeyMenshykh Could you review the convertion from AutoGen message to sk message content here

[SergeyMenshykh]

Actually, the chat completion service can return many massages. By default, it returns one, but can be configured to return many choices/completions each of which will be represented by a separate chat message content. This behavior is controlled by the ResultsPerPrompt property of the OpenAIPromptExecutionSettings settings class.

This example demonstrates how to configure it to return 2 completions - Example36_MultiCompletion

[SergeyMenshykh]

Handling only the OpenAIChatMessageContent message will not allow this agent to work with any other non-OpenAI existing or future implementations of the IChatCompletionService interface. For example -AzureOpenAIWithDataChatMessageContent.cs.

Additionally, referencing the OpenAIChatMessageContent type in the agent unnecessarily couples it with OpenAI message. This is acceptable in the short-term perspective for referencing it for tool-calling purposes; however, in the long-term perspective, the dependency of the agent on LLM-specific messages should be dropped. Instead, an LLM-agnostic model (which we plan to start working on soon) for tool-calling should be used.

So, I suggest to slightly change the message handling logic:

if (msg is OpenAIChatMessageContent oaiContent && oaiContent.ToolCalls is { Count: > 0 } toolCalls)
{
   //Get tools call info only - id, name, arguments
   return new function message
}

foreach(var content in msg.Items)
{
   ... switch ...
   {
      TextContent => text message
      ImageContent =>  image message
   }
}

[SergeyMenshykh]

1. LLM can easily request more than one tool call. Is there any reason to handle only the first one?
2. Just FYI: Tool calling is controlled by the ToolCallBehavior property. If the auto behavior is configured, the tool(s) will be called automatically behind the scenes, with no tool calling messages returned by the GetChatMessageContentsAsync method. This example demonstrates this in action: AgentAutoFunctionInvocationAsync. To get the GetChatMessageContentsAsync method to return an OpenAIChatMessageContent with a filled ToolCalls collection, the behavior should be set to manual, as demonstrated in this example: AgentManualFunctionInvocationAsync.

[LittleLittleCloud]

@SergeyMenshykh I just didn't find out an example that processes more than one tool call, so I don't know how to implement the conversation from semantic kernel to autogen. Could you share an example of multi-tool-usage in semantic kernel

[SergeyMenshykh]

BTW: We are already working on ChatCompletionAgent in the feature-agents-abstraction branch. Streaming is in progress as well - .Net: [Agents] Abstraction for streaming

[BeibinLi]

Is this the URL to the image for this message? Would a message contain more than one images?

[LittleLittleCloud]

Yes, and for multi-image situations, MultiModalMessage is the message to use.

[ekzhu]

Great PR. Perhaps it is a time to think about the next steps, in addition to documentation.