Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWQ? #119

Closed
themrzmaster opened this issue Jul 3, 2024 · 4 comments
Closed

AWQ? #119

themrzmaster opened this issue Jul 3, 2024 · 4 comments

Comments

@themrzmaster
Copy link

HI!
WOuld be nice to have AWQ quantization, so we can run on vLLM with a lower end GPU.
Thanks for your work!

@sanjay920
Copy link
Collaborator

hey @themrzmaster what model are you looking for awq quants? i can get that for you

@themrzmaster
Copy link
Author

themrzmaster commented Jul 3, 2024

70b would be nice @sanjay920

@sanjay920
Copy link
Collaborator

@themrzmaster https://huggingface.co/rubra-ai/Meta-Llama-3-70B-Instruct-AWQ

Let me know if you have any problems with it. If it looks good feel free to close this issue.

@themrzmaster
Copy link
Author

Worked! Thank you!! Any plan on releasing the dataset?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants