New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for bnb nf4 quant? #18

Open

col-in-coding opened this issue Nov 27, 2023 · 1 comment

col-in-coding commented Nov 27, 2023 •

edited

Loading

It seems that there's no decreasing of lattency on llama-13b model using nf4 quant

Collaborator

Viol2000 commented Nov 27, 2023

I'm not sure about its compatibility with bnb.
But it is already integrated into llama.cpp. You can check it if you want to run quant models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment