[Bug]: for mistral model, After the optional system message, conversation roles must alternate user/assistant/user/assistant/

[yanan1116]

Your current environment

vllm version: v0.5.3.post1

🐛 Describe the bug

launch the vllm: CUDA_VISIBLE_DEVICES=7 python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.3 --api-key "yyy" --port 1703

call the mistral model via python :

import openai
client = openai.OpenAI(
    api_key="yyy",
    base_url="http://localhost:1704/v1/",
)


response = client.chat.completions.create(model="mistralai/Mistral-7B-Instruct-v0.3", 
                                         messages=[{'role':'system', 'content':'you are wikipedia agent.'}, 
                                                    {'role':'user', 'content':'whta is the capital city of ontario ?'},
                                                    {'role':'user', 'content':'and tell me about its history.'}], 
                                         max_tokens=128, 
                                         temperature=0)

error message:

BadRequestError: Error code: 400 - {'object': 'error', 'message': 'After the optional system message, conversation roles must 
alternate user/assistant/user/assistant/...', 'type': 'BadRequestError', 'param': None, 'code': 400}

This error does not appear in other model series other than mistral.
Any hint ?

[DarkLight1337]

As mentioned in the error message, you can't have two user messages in a row. It must be user, then assistant, then back to user, and so on.

[yanan1116]

@DarkLight1337 i see.
but consecutive user roles work in other models like llama.

seems that the only way to solve is to add {'role':'assistant', 'content': 'ok'} to the user role.

[DarkLight1337]

The model might not have been trained on non-alternating roles., so the model performance may degrade significantly for such inputs. Nevertheless, you can forcibly allow this by supplying your own chat template (--chat-template CLI option) instead of defaulting to the one from HuggingFace repo.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!