Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suppprt static groups and fix bug #236

Merged
merged 4 commits into from
Aug 8, 2023
Merged

Suppprt static groups and fix bug #236

merged 4 commits into from
Aug 8, 2023

Conversation

qwopqwop200
Copy link
Collaborator

Adds static groups, a new feature of GPTQ.
Also fix disable exllama not working on fused attention.

@qwopqwop200
Copy link
Collaborator Author

qwopqwop200 commented Aug 7, 2023

I will be benchmarking llama soon. wait

@qwopqwop200
Copy link
Collaborator Author

qwopqwop200 commented Aug 7, 2023

All quantized from wikitext2.
Experiment with gptq code

LLaMA2-7B Bits group-size Wikitext2 PTB C4
GPTQ 4 128 5.7229 nan 7.1896
GPTQ(static groups) 4 128 5.7343 15849 7.1757
GPTQ(act-order) 4 128 5.5847 54.5089 7.1423
GPTQ(act-order&static groups) 4 128 5.5802 41.3790 7.1391

Experiment with AutoGPTQ code

LLaMA2-7B Bits group-size Wikitext2
GPTQ 4 128 5.2595
GPTQ(static groups) 4 128 5.2692
GPTQ(act-order) 4 128 5.2388
GPTQ(act-order&static groups) 4 128 5.2542

@qwopqwop200
Copy link
Collaborator Author

PanQiWei
Experiments with static groups all seem to be ok. Now I think can merge them.

Copy link
Collaborator

@PanQiWei PanQiWei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Thank you so much for this fast implementation!

@PanQiWei PanQiWei merged commit 5981f15 into main Aug 8, 2023
@qwopqwop200 qwopqwop200 deleted the suppprt-static_groups branch August 8, 2023 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants