Request for the Llama-2-13B with AQLM (2x8 scheme)

[ChenMnZ]

Hello,

Thanks for your outstanding works. I want to do a compressive comparison of recent quantization methods.

Due to that latest lm-eval can obtain higher accuracy than the ones reported in paper, I have to re-evaluate each quantized models.

I found that there is not model about Llama-2-13B with AQLM (2x8 scheme), can you share it on huggingface?

Thank you!

[Vahe1994]

Hello!
Sorry for late response. It was very busy several weeks.
Thanks for interest in the work. Currently 13B 2x8gs8 quantization is running. When it is done, I will put it in HF hub and tell you.

P.s. Additionally you can check out this PR #93 for new LM eval code. Thanks to fast dequantization kernels, you can run lm eval from HF as any other models.
For ppl please check out #91.

[Vahe1994]

@ChenMnZ The model is ready please see https://huggingface.co/ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

[ChenMnZ]

Thanks for your time!!! I will try this model.