-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KV Cache Quantization #85
Comments
Hi! I am not an author, but a contributor, and I have some familiarity with the issue. As you correctly describe, the AQLM does not do cache quantization, relying on standard transformers code. One way you can do this is by extending transformers Cache class roughly as follows:
This should give you the expected memory saving since only one attention layer is de-quantized at a time. Since the neurips deadline is soon, it is unlikely that paper authors will be able to write this anytime soon. However, if you try this and share your observations, we'd be glad to take a look. In turn, if you have any issues with AQLM while doing so, please tell us. |
This issue is stale because it has been open for 30 days with no activity. |
For anyone curious, huggingface has done it: https://huggingface.co/blog/kv-cache-quantization |
TWIMC: @Vahe1994 recently also tested that their standard kv quantization works with aqlm-quantized models He modified this notebook with the following code:
This is not using AQLM for KV caches, but using AQLM for weights in combination of default cache quantization from quanto. |
Hey, thanks for your work. I saw https://huggingface.co/ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16/discussions/2 about how 8-bit KV cache quantization can be enabled on vLLM. I am not too sure of how exactly KV cache is implemented for AQLM using Transformers, but would KV cache quantization be theoretically possible? It might address some of the concerns regarding high vram usage for context from https://www.reddit.com/r/LocalLLaMA/comments/1clinlb/bringing_2bit_llms_to_production_new_aqlm_models/.
To be specific, would 4-bit cache quantization be possible? Turboderp managed to achieve negligable ppl loss somehow: https://github.com/turboderp/exllamav2/blob/master/doc/qcache_eval.md. For reference, turboderp/exllamav2@324404e.
Thanks!
The text was updated successfully, but these errors were encountered: