Mixtral-8x7B-Instruct-v0.1 will be officially integrated inside the fastchat codebase

[jalajthanaki]

will Mixtral-8x7B-Instruct-v0.1 be officially integrated and supported by fastchat codebase specially via vllm?

[jalajthanaki]

Basically, Mixtral-8x7B-Instruct-v0.1 can be used via vllm directly but when I was trying to use it via fastchat it won't be generating the responses which are useful at all hence this is the question.

In model registry, I could not find the Mixtral-8x7B-Instruct-v0.1 integrated in fastchat makes me believe that even on chat arena, it is directly via vllm or Mixtral-Ai's endpoint.

[surak]

Are you using the pip version? I would recommend you do a clone of the repo instead. I have Mixtral running in production since one day after it came out, and it works just fine. People seem to be happy with it.

I would close it, because it really works. You are welcome to reopen if you disagree, though!

[jalajthanaki]

Thank you @surak for your reply and hint. Reopening this issue because I have still few questions :)

Here how anyone can make Mixtral-8x7B-Instruct-v0.1 work using FastChat

• Python3.10 given that megablocks only works with Python3.10
• install vllm 0.2.4 version (newer versions of vLLM are having few issues like #2219 , #2229)
• checkpoints in .pt format
• cuda version >=12.1
• pip install megablocks

Still I find the answers from the chat arena's Mixtral-8x7B-Instruct-v0.1 much better.

curl --location 'http://$IP:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "model": "Mixtral",
    "messages": [
        {
            "role": "user",
            
            "content":"who are you?"
            
        }
    ],
    "temprature": 0.7,
    "max_tokens": 1024,
    "top_p":1
}'

Answer from local inference.

{
    "id": "chatcmpl-GohBJuu5P6kXkDzU3BsCib",
    "object": "chat.completion",
    "created": 1703237717,
    "model": "Mixtral",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": " I am an artificial intelligence assistant, designed to help answer questions, provide information, and assist with various tasks to make your life easier and more convenient."
            },
            "finish_reason": "stop"


        }
    ],
    "usage": {
        "prompt_tokens": 540,
        "total_tokens": 571,
        "completion_tokens": 31
    }
}

---

Answer from the Chat Arena.

---

Still have few questions

Is there anything that I'm still missing?

• I knew LLM are not going to produce the same result but still the answer I'm getting are still having lesser quality compared to Arena.
• Is there any specific system prompt I need to use to get the right quality output?
• What is the average throughput of Mixtral-8x7B-Instruct-v0.1 model? Any information around same?