Support the Mistral AI official huggingface weights for Mixtral-8x7B-v0.1

[lostmygithubaccount]

per the current example:

Download the models from HuggingFace:
git clone https://huggingface.co/someone13574/mixtral-8x7b-32kseqlen

I'd really rather not have to re-download many GBs of weight files, and less-so from someone13574 (no offense) when there weights are posted by Mistral AI itself: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

would this work already? could the example be updated to use these weights?

[awni]

Yea it's a good point. Will look into updating the example to use the official weights instead of the ones from MixtralKit

[thegodone]

One major issue how to run it with 64 GB memory ? This is a huge model, we need to think of reducing the precision. Is it also possible to think of delta in the weights between the chat (instruction version Mixtral-8x7B-Instruct-v0.1) and the native generator (mixtral-8x7b-32kseqlen). But I don't see an easy way so far to run on M1 64 GB. running the convert give me a terminal killed message

(tf) tgg@gvalmu00008 mixtral % python convert.py --model_path mixtral-8x7b-32kseqlen/
zsh: killed     python convert.py --model_path mixtral-8x7b-32kseqlen/

[lostmygithubaccount]

I was able to use the llama.cpp convert script to get it to q4_0 and run in (M2, 96GB of RAM) -- I cannot with f16

seeing very weird output from the model though. will try to look into this more later, looking forward to learning/playing around w/ mlx on some of these models

[caseybasichis]

I'm able to get the someone13574 running.

I attempted the cat/convert process on the official instruct model, but no dice.

Is there a way to modify the convert script to get the instruct version going?

[awni]

This one right? https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

I'll test it after getting the official Mistral one working

[caseybasichis]

That's the one.

[awni]

#107