[Feature Request] Support grouped-query attention

[abhinavkulkarni]

Hi,

The recent release of LLaMA 2 from Meta AI uses grouped-query attention (GQA) as opposed to multi-head attention (MHA) for the 70B model and the current AWQ search fails. Considering it is the best open-source model, please support GQA.

Thanks!

[casper-hansen]

Did you try with newest transformers 4.31.0? They only just added LLaMa 2 support in this PR:
huggingface/transformers#24891

Guide says the following: num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used

Also, see https://github.com/huggingface/transformers/pull/24891/files#diff-110a445233a8b15a0875998eeaf75cb8607b38a5daa736291dd058766879bbddR104

[abhinavkulkarni]

Closing this as this is in the works: #53