-
Notifications
You must be signed in to change notification settings - Fork 404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AssertionError #486
Comments
TheBlokeAI/Mixtral-tiny-GPTQ is meant only for testing AutoGPTQ and does not have the modules set in the config necessary for transformers to load it. If you want to load it you need to use AutoGPTQ directly. You also need the latest transformers and optimum if you want to load the full model with transformers. |
Using the model Code:
Full list of packages:
Error:
|
Make sure you have the updated config https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ/blob/main/config.json |
Yes, the config file downloaded (on my machine - see below), matches the details of the config you've mentioned:
|
Updated the transformers library, works now... Thanks for your inputs on this @LaaZa 👍, really appreciate it.
|
Oh, I though that was in 4.36.1 but it seems it just barely missed that version. |
@virentakia can you share the fix? got the same error and can't make it work |
@luisfrentzen-cc - Downloaded and installed the latest Dev version of transformers (4.37? Not released yet I guess).
4.36.* is the latest one released ( and that does not seem to work) |
pip install -U git+https://github.com/huggingface/transformers.git |
well, i got the same error and updated transformers to 4.37.0.dev0 but still wrong.
|
@MarseusFu I'm not sure how you would get the same error, but you need optimum. |
@MarseusFu Try to run |
@cckuailong Still got the AssetionError :(
|
@MarseusFu |
@cckuailong Yes, I installed both of them and still got the AssertionError. |
@MarseusFu any solution to the problem? I'm facing the same issue with transformers and optimum from pip, auto-gptq compiled from source https://github.com/AutoGPTQ/AutoGPTQ |
@paolovic Sadly, no. I gave up. |
This should not be closed. Quantizing Mixtral with AutoGPTQ writes a config out that AutoGPTQ itself is not compatible with. I can confirm that it does work if you manually add |
AutoGPTQ does not use that config, It's for transformers and optimum. Modules are defined in the code for AutoGPTQ, not a config. This is likely not an AutoGPTQ issue but issue with optimum. |
It seems like if you use AutoGPTQ/AutoAWQ directly you can get something working.
Source: |
The code below throws an assertion error:
Error:
The text was updated successfully, but these errors were encountered: