-
Notifications
You must be signed in to change notification settings - Fork 26.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when QWEN1.5 32B/9B de-quantizing GGUF #32526
Comments
cc @SunMarc |
Hi @kunger97, can you share with me the version of gguf that you are using ? Also, did it work in the past or you tried it for the first time ? |
This is my first attempt, i'm useing gguf 0.9.1 |
Seems that the model config was created incorrectly. I ran the reproduction code, and the vocab size in config is incorrect: from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
model_id = "Qwen/Qwen1.5-32B-Chat-GGUF"
filename = "qwen1_5-32b-chat-q4_k_m.gguf"
config = AutoConfig.from_pretrained(model_id, gguf_file=filename)
print(config)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
print(tokenizer) Outputs Qwen2Config {
"_model_name_or_path": "Qwen2-beta-32B-Chat-AWQ-fp16",
"_name_or_path": "Qwen/Qwen1.5-32B-Chat-GGUF",
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 27392,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 40,
"num_hidden_layers": 64,
"num_key_value_heads": 8,
"pad_token_id": 151643,
"rms_norm_eps": 9.999999974752427e-07,
"rope_theta": 1000000.0,
"sliding_window": 4096,
"tie_word_embeddings": false,
"transformers_version": "4.42.3",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}
Qwen2TokenizerFast(name_or_path='Qwen/Qwen1.5-32B-Chat-GGUF', vocab_size=152064, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
} And the correct In fact, the current gguf config extraction is dependent on the |
Is there currently a solution to this problem? |
System Info
transformers
version: 4.45.0.dev0Who can help?
@ArthurZucker @Isotr0py
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
run my script to save gguf model to pytorch model
Expected behavior
The text was updated successfully, but these errors were encountered: