-
Notifications
You must be signed in to change notification settings - Fork 459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend support for Phi-3 models #651
base: main
Are you sure you want to change the base?
Conversation
@davidgxue Did you have a stable avg losses and/or did you do ppl of pre-quant model vs post-quant model to see if the fused layers posed issue to quantizer? I know from dbrx tests that fused layers are really bad for quantization. |
Yes, let me upload it. I only tested on 8 bit. I can do some more testing on 4 bits as well and come back to this. |
Some delays have been encountered due to this #657. I am unable to get around the |
Hey @davidgxue, |
@bhardwajsapna Try my packing fix Pr #642. If you have lots of cores |
GPTQModel has validated the code by @davidgxue and have merged this in PR ModelCloud/GPTQModel#27 If you want a AutoGPTQ compatible pkg that supports the latest models and bug fixes, checkout GPTQModel |
Just tested but hit |
Description
Technical Details
qkv_proj
is just ann.Linear
. However, I could be wrong, and would be great if someone wants to nudge me in the right direction.Tests
Side Note
Related Issues
closes #652