-
Notifications
You must be signed in to change notification settings - Fork 189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement exllama q4_matmul kernel as alternative #3
Comments
https://github.com/qwopqwop200/AutoAWQ-exllama awq kernel
[======] Model summary: opt-125m-awq [======] exllama kernel
[======] Model summary: opt-125m-awq [======] It was tested in the following. wsl (window 11) |
This is good work @qwopqwop200. I was working on the same thing on the exllama branch. It seems there could be a modest boost in speed of around 10% from your initial testing. Do you want to open a PR or can I copy your work into the exllama branch? |
Copy it to exllama branch. I'm not sure yet, but it seems that exllama and awq kerenl have different weight storage methods. This may be why exllama is not working. |
I have gone through your implementation now and unfortunately, it seems it runs into the same issues around the shapes of the in_features and out_features. I have fixed these for now in the exllama branch, but I still need to make the fused modules work. If you have time to spare @qwopqwop200 and want to help with the exllama integration, I would appreciate it if you could work from this branch. A few issues:
|
Draft PR #30 is now open. |
ExLlama has implemented very optimized CUDA kernels. We should import the kernels to see just how efficient it could be in AWQ.
https://github.com/turboderp/exllama/blob/master/exllama_ext/exllama_ext.cpp#L199
The text was updated successfully, but these errors were encountered: