Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qwen1.5-72B-Chat-GPTQ-Int4 #16

Closed
ChuanhongLi opened this issue Apr 1, 2024 · 2 comments
Closed

Qwen1.5-72B-Chat-GPTQ-Int4 #16

ChuanhongLi opened this issue Apr 1, 2024 · 2 comments

Comments

@ChuanhongLi
Copy link

请问下，能直接跑 Qwen1.5-72B-Chat-GPTQ-Int4 模型吗？
@guyan364
Copy link
Collaborator

guyan364 commented Apr 2, 2024

你好，我使用 Qwen1.5-7B-Chat-GPTQ-Int4 测试可以推理。load model 的时候去掉 dtype=torch.bfloat16 即可。

@guyan364 guyan364 closed this as completed Apr 2, 2024
@ChuanhongLi ChuanhongLi mentioned this issue Apr 15, 2024
Closed
@huliangbing
Copy link

您好！修改哪个文件？@guyan364

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@huliangbing @guyan364 @ChuanhongLi