[Bug]: Wrong role set in group chat

[hardchor]

Describe the bug

When running a group chat and retrieving the chat_result, all user messages are flagged as role="assistant".
This causes an issue, e.g. when using the chat_result to later on resume the conversation (or when relying on the role value to display whether a message came from the user in a UI).

Steps to reproduce

1. Initiate a group chat

chat_result = user_proxy.initiate_chat(group_chat_manager, message="Hello world!")

2. Send a few messages, exit & inspect the chat results

Model Used

gpt-4o

Expected Behavior

Messages coming from users (i.e. via UserProxyAgent) should be marked as role="user"

Screenshots and logs

No response

Additional Information

I noticed that https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py#L653 hardcodes "assistent", which is the value that ends up in self._oai_messages. Since GroupChatManager doesn't override the send function, I believe (!) this might be the culprit.

P.S.: I noticed https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py#L774 hardcoding "user" on the receiver side, which might cause a similar issue.

[marklysze]

Hi @hardchor, would you be able to provide some code that shows the resume failing?

As a note, the use of the user roles on the internal messages may not reflect a human input, I believe they are more aligned with the alternating interaction between agents.

[hardchor]

@marklysze sorry for the delay, I was travelling.

Hi @hardchor, would you be able to provide some code that shows the resume failing?

The issue actually shows up in another bug report I came across: #2968. You can see that the initial message (create a resource in x) has 'role': 'assistant', when it arguably should be user.

Otherwise, just following the example in https://microsoft.github.io/autogen/docs/topics/groupchat/resuming_groupchat/ gives the same result. I worked around this issue by expanding UserProxy and AssistantAgent with an additional role attribute (that I then read in my custom GroupChatManager), but it's a lot of copy & pasting of code for what is essentially a couple of lines changed.

As a note, the use of the user roles on the internal messages may not reflect a human input, I believe they are more aligned with the alternating interaction between agents.

I'm not sure I understand fully - in the OpenAI protocol, messages marked with "user" (as opposed to "system" or "assistant") signifies a message coming from the user, doesn't it?

[marklysze]

As a note, the use of the user roles on the internal messages may not reflect a human input, I believe they are more aligned with the alternating interaction between agents.

I'm not sure I understand fully - in the OpenAI protocol, messages marked with "user" (as opposed to "system" or "assistant") signifies a message coming from the user, doesn't it?

I don't believe the messages structure aligns completely with a normal back-and-forth chat as one or more agents are involved. The "assistant" role is used for agents when they send messages, see here.

The use of the name field is what helps OpenAI's models distinguish between the different agents. OpenAI can use this 'name' field (and does), however non-OpenAI inference does not (hence I created this).

OpenAI doesn't seem that strict on the user/assistant roles and alternating messages. This isn't so much the case with other LLM providers, who are generally strict on it.

[hardchor]

@marklysze I understand how autogen works under the hood (using the name field), and that OpenAI doesn't actually care all that much about the role parameter (maybe). But don't you agree that (from a purely logical point of view) the roles that autogen is setting on the messages are just wrong? From the OpenAI docs:

role
string

Required
The role of the entity that is creating the message. Allowed values include:

user: Indicates the message is sent by an actual user and should be used in most cases to represent user-generated messages.
assistant: Indicates the message is generated by the assistant. Use this value to insert messages from the assistant into the conversation.

Admittedly, documentation around this is very sparse, and they are even omitting the system role from their documentation.

[marklysze]

Yes, can totally see where you're coming from in terms of the use of the role user. For back and forth conversations between an LLM that definitely should be the case.

With the overlay of agents and non-human input (there may be no human input in an AutoGen group chat / conversation), the term user in AutoGen is not exactly aligned with humans/UserAgentProxy. This is the reason that many of the non-OpenAI client classes need to adjust the messages to adhere to user/assistant/user/assistant sequences.

I think it's a worthwhile point you bring up and is something I've heard the AutoGen team discussing from time-to-time. Perhaps the role in the messages in AutoGen isn't strictly necessary and can be derived from the associated agent, instead.

If you had thoughts on what a future set of messages may look like it would be good to discuss in Discord's ideas-and-feedback channel.

For the time-being, to check if a message is from a UserProxyAgent, the name field could be used to then get the agent and check the type.

[hardchor]

I'm instead extending UserProxyAgent and AssistantAgent with a role parameter. I don't like it, but the alternative (looking up roles by name in a slightly more complex system) felt even less clean.

With the overlay of agents and non-human input (there may be no human input in an AutoGen group chat / conversation), the term user in AutoGen is not exactly aligned with humans/UserAgentProxy.

I'd love to understand in what case you'd use user in autogen (but only if you have time, otherwise feel free to close).

[marklysze]

I'm instead extending UserProxyAgent and AssistantAgent with a role parameter. I don't like it, but the alternative (looking up roles by name in a slightly more complex system) felt even less clean.

With the overlay of agents and non-human input (there may be no human input in an AutoGen group chat / conversation), the term user in AutoGen is not exactly aligned with humans/UserAgentProxy.

I'd love to understand in what case you'd use user in autogen (but only if you have time, otherwise feel free to close).

For my use to date, I haven't had a need for the user value. I've been focused on working with non-OpenAI models so have been having to cater for the user/assistant alternating role requirements of these other models, so it hasn't been overly helpful as is. I haven't thought too much about what I'd prefer to see, but if I was to put something out there I'd say replace name with a sender agent/object and a recipients containing agents/objects, would be useful so I can look at the full flow and understand it.

I'll close this soon, but please feel free to open another issue or re-open if you want to propose something that would be help you.