Add kvcache config for Mistral

[mgoin]

git clone https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha
sparseml.transformers.export_onnx --model_path ./zephyr-7b-alpha --task text-generation --sequence_length 512 --trust_remote_code
cp deployment/model.onnx deployment/model-orig.onnx
python ~/onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx

>>> from deepsparse import TextGeneration
>>> model = TextGeneration(model="deployment")
>>> out = model("Once upon a time,", max_new_tokens=100)
>>> out.generations[0].text
'there was a young woman named Lily. She was a kind and gentle soul, with a heart full of love and compassion. Lily had always been fascinated by the natural world, and she spent most of her free time exploring the forests and fields around her home.\n\nOne day, as she was wandering through the woods, Lily stumbled upon a small clearing. In the center of the clearing, she saw a beautiful butterfly fluttering its wings. The butterfly was unlike any she had'

HF Baseline :)

[dsikka]

Can you add an explanation/comment explaining why this is needed and how this differs than the other llama models?

Also, will this work for the 70b model? In that case, can we update the name of the config to cover what group of llama models this will cover?

[dsikka]

A warning will show up every time about the number of updated nodes if they aren't the standard, just an fyi

[mgoin]

Yeah, we should fix this for all Llama models

[dbogunowicz]

@dsikka could you elaborate?

[dbogunowicz]

@dsikka it should work for 70b model, but I am not sure whether @mgoin has the capacity to test it. I'd take a leap of faith and merge it regardless.

[mgoin]

@dsikka it should work for 70b model, but I am not sure whether @mgoin has the capacity to test it. I'd take a leap of faith and merge it regardless.

Mistral actually uses a "completely" new model/config class so the config.json actually holds a "MistralForCausalLM" architecture. Here is an example config. There is only one pretrained Mistral arch model and it is 7b.

Based on how we're currently architecting this, I assume we need to make a new config entry for every new "model_type" we see in config.json files. This makes the Mistral models not exactly connected to the Llama model support for GQA, aka the 70b model y'all are talking about. So it is disconnected in practice, but I will put a comment that Mistral is very connected to Llama in origin.

We should make a separate PR in order to support Llama models with GQA enabled.

[dsikka]

@mgoin from what we've seen though, it seems like the config you added could also be used for GQA?

[mgoin]

@dsikka yes, it should work for Llama models with GQA if the name is changed to "llama". My position is that Llama GQA support is a different issue than Mistral support, but if y'all want to add it into this diff I'm good with that. I'm not sure how to structure the change without affecting non-GQA Llama models