[Bug]: Conversation roles must alternate user/assistant/user/assistant/... in vllm for non-openai models in GroupChat

[xymou]

Describe the bug

Hi!
Thanks for your excellent project! But I have encountered some bugs, when using vllm to serve local open-sourced llms for GroupChat, such as Llama2-7b-chat-hf and Mistral, there are messages like:

1. role sequence like ['system','assistant','user'...] for the first speaker in the group
2. role sequence like ['system','user','user'...] for other speakers
   and this would cause error " Conversation roles must alternate user/assistant/user/assistant/..."

I guess it is the result of chat template and I have read the case in https://microsoft.github.io/autogen/docs/topics/non-openai-models/local-vllm#chat-template. I tried the provided template for Mistral but the errors still exist.
I want to know how to solve this problem. Thank you!

Steps to reproduce

No response

Model Used

No response

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

[rysweet]

Eric Zhu (@ekzhu) - can you please comment on this one?

[ekzhu]

Mark Sze (@marklysze) is this something you have already worked out a solution for?

[marklysze]

xymou, are you able to use Ollama? There isn't a native vLLM client and so there isn't logic to enforce a role alternation.

Here's the Ollama client documentation:
Ollama client

With Ollama you can use Llama 3.2 3B, Llama 3.1 8B, and/or Mistral 7B 0.3.

You will need to pip install ollama fix-busted-json to install the required packages for AutoGen to use.

Let us know how you go.

[ekzhu]

xymou let us know how did it go

[xymou]

Ryan Sweet (@rysweet) Mark Sze (@marklysze) Eric Zhu (@ekzhu) Thanks for your reply! I didn't try Ollama, but I appreciate your suggestions! I have solved this issue by rewriting a chat template of tokenizer for these models to allow the role alternation and assign the chat templates when serving vllm.