Failed at model.save_pretrained_gguf #341

ch-tseng · 2024-04-16T15:31:55Z

I use the model: https://huggingface.co/taide/TAIDE-LX-7B-Chat to fine-tune, but always got the error. training is OK, but model.save_pretrained_gguf failed.

==((====))== Unsloth: Fast Llama patching release 2024.4
\ /| GPU: NVIDIA GeForce RTX 3090. Max memory: 23.691 GB. Platform = Linux.
O^O/ _/ \ Pytorch: 2.2.2+cu121. CUDA = 8.6. CUDA Toolkit = 12.1.
\ / Bfloat16 = TRUE. Xformers = 0.0.25.post1. FA = False.
"--" Free Apache license: http://github.com/unslothai/unsloth
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.02s/it]
You set add_prefix_space. The tokenizer needs to be converted from the slow tokenizers
Unsloth 2024.4 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
Setting pad_token_id to eos_token_id:2 for open-end generation.
['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n\n\n### Input:\n上上禮拜持續出現頭痛、噁心、頭暈的症狀，有時睡一下起來還是沒有緩解，大概都痛在太陽穴上面一點，有時痛在頭腦勺（較少），躺著起來頭暈頻率越來越高（本身有貧血，但近期只要姿勢一轉換就會頭暈眼前接近黑色），容易疲累，想問一下這些症狀有需要到醫院去檢查嗎？\n\n### Response:\n\n您好：\n根據您的描述，您可能有以下幾種可能的原因：\n1. 貧血：貧血是常見的問題，若沒有定期檢查，可能會導致頭暈、頭痛、疲累等症狀。\n2. 內耳問題：內耳有平衡器官，若內耳有問題，可能會導致頭暈、頭痛、噁心等症狀。\n3. 其他疾病：如甲狀腺疾病、心臟疾病、糖尿病、高血壓等，都可能會引起頭暈、頭痛、噁心等症狀。\n建議您前往醫院，讓醫師為您做詳細的檢查，以確定病因，並接受適當的治療。\n祝健康！ ']
Unsloth: Merging 4bit and LoRA weights to 16bit...
Unsloth: Will use up to 46.9 out of 62.57 RAM for saving.
100%|█████████████████████████████████████████████████████████████████████████████████| 32/32 [00:00<00:00, 90.79it/s]
Unsloth: Saving tokenizer... Done.
Unsloth: Saving model... This might take 5 minutes for Llama-7b...
Done.
Unsloth: Converting llama model. Can use fast conversion = True.
==((====))== Unsloth: Conversion from QLoRA to GGUF information
\ /| [0] Installing llama.cpp will take 3 minutes.
O^O/ _/ \ [1] Converting HF to GUUF 16bits will take 3 minutes.
\ / [2] Converting GGUF 16bits to q4_k_m will take 20 minutes.
"--" In total, you will have to wait around 26 minutes.

Unsloth: [0] Installing llama.cpp. Unsloth: [1] Converting model The output location will be ./ch_taide This will take 3 minutes...
Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g params = Params(n_vocab=56064, Loaded vocab file PosixPath('ch_taide_ Vocab info: <LlamaHfVocab Special vocab info: <SpecialVocab Permuting layer 0
Permuting layer 1
Permuting layer 2
Permuting layer 3
Permuting layer 4
Permuting layer 5
Permuting layer 6
Permuting layer 7
Permuting layer 8
Permuting layer 9
Permuting layer 10
Permuting layer 11
Permuting layer 12
Permuting layer 13
Permuting layer 14
Permuting layer 15
Permuting layer 16
Permuting layer 17
Permuting layer 18
Permuting layer 19
Permuting layer 20
Permuting layer 21
Permuting layer 22
Permuting layer 23
Permuting layer 24
Permuting layer 25
Permuting layer 26
Permuting layer 27
Permuting layer 28
Permuting layer 29
Permuting layer 30
Permuting layer 31
model.embed_tokens.weight model.layers.0.input_layernorm.weight model.layers.0.mlp.down_proj.weight model.layers.0.mlp.gate_proj.weight model.layers.0.mlp.up_proj.weight model.layers.0.post_attention_layernorm.weight model.layers.0.self_attn.k_proj.weight model.layers.0.self_attn.o_proj.weight model.layers.0.self_attn.q_proj.weight model.layers.0.self_attn.v_proj.weight model.layers.1.input_layernorm.weight model.layers.1.mlp.down_proj.weight model.layers.1.mlp.gate_proj.weight model.layers.1.mlp.up_proj.weight model.layers.1.post_attention_layernorm.weight model.layers.1.self_attn.k_proj.weight model.layers.1.self_attn.o_proj.weight model.layers.1.self_attn.q_proj.weight model.layers.1.self_attn.v_proj.weight model.layers.10.input_layernorm.weight model.layers.10.mlp.down_proj.weight model.layers.10.mlp.gate_proj.weight model.layers.10.mlp.up_proj.weight model.layers.10.post_attention_layernorm.weight model.layers.10.self_attn.k_proj.weight model.layers.10.self_attn.o_proj.weight model.layers.10.self_attn.q_proj.weight model.layers.10.self_attn.v_proj.weight model.layers.11.self_attn.k_proj.weight model.layers.11.self_attn.q_proj.weight model.layers.2.input_layernorm.weight model.layers.2.mlp.down_proj.weight model.layers.2.mlp.gate_proj.weight model.layers.2.mlp.up_proj.weight model.layers.2.post_attention_layernorm.weight model.layers.2.self_attn.k_proj.weight model.layers.2.self_attn.o_proj.weight model.layers.2.self_attn.q_proj.weight model.layers.2.self_attn.v_proj.weight model.layers.3.input_layernorm.weight model.layers.3.mlp.down_proj.weight model.layers.3.mlp.gate_proj.weight model.layers.3.mlp.up_proj.weight model.layers.3.post_attention_layernorm.weight model.layers.3.self_attn.k_proj.weight model.layers.3.self_attn.o_proj.weight model.layers.3.self_attn.q_proj.weight model.layers.3.self_attn.v_proj.weight model.layers.4.input_layernorm.weight model.layers.4.mlp.down_proj.weight model.layers.4.mlp.gate_proj.weight model.layers.4.mlp.up_proj.weight model.layers.4.post_attention_layernorm.weight model.layers.4.self_attn.k_proj.weight model.layers.4.self_attn.o_proj.weight model.layers.4.self_attn.q_proj.weight model.layers.4.self_attn.v_proj.weight model.layers.5.input_layernorm.weight model.layers.5.mlp.down_proj.weight model.layers.5.mlp.gate_proj.weight model.layers.5.mlp.up_proj.weight model.layers.5.post_attention_layernorm.weight model.layers.5.self_attn.k_proj.weight model.layers.5.self_attn.o_proj.weight model.layers.5.self_attn.q_proj.weight model.layers.5.self_attn.v_proj.weight model.layers.6.input_layernorm.weight model.layers.6.mlp.down_proj.weight model.layers.6.mlp.gate_proj.weight model.layers.6.mlp.up_proj.weight model.layers.6.post_attention_layernorm.weight model.layers.6.self_attn.k_proj.weight model.layers.6.self_attn.o_proj.weight model.layers.6.self_attn.q_proj.weight model.layers.6.self_attn.v_proj.weight model.layers.7.input_layernorm.weight model.layers.7.mlp.down_proj.weight model.layers.7.mlp.gate_proj.weight model.layers.7.mlp.up_proj.weight model.layers.7.post_attention_layernorm.weight model.layers.7.self_attn.k_proj.weight model.layers.7.self_attn.o_proj.weight model.layers.7.self_attn.q_proj.weight model.layers.7.self_attn.v_proj.weight model.layers.8.input_layernorm.weight model.layers.8.mlp.down_proj.weight model.layers.8.mlp.gate_proj.weight model.layers.8.mlp.up_proj.weight model.layers.8.post_attention_layernorm.weight model.layers.8.self_attn.k_proj.weight model.layers.8.self_attn.o_proj.weight model.layers.8.self_attn.q_proj.weight model.layers.8.self_attn.v_proj.weight model.layers.9.input_layernorm.weight model.layers.9.mlp.down_proj.weight model.layers.9.mlp.gate_proj.weight model.layers.9.mlp.up_proj.weight model.layers.9.post_attention_layernorm.weight model.layers.9.self_attn.k_proj.weight model.layers.9.self_attn.o_proj.weight model.layers.9.self_attn.q_proj.weight model.layers.9.self_attn.v_proj.weight model.layers.11.input_layernorm.weight model.layers.11.mlp.down_proj.weight model.layers.11.mlp.gate_proj.weight model.layers.11.mlp.up_proj.weight model.layers.11.post_attention_layernorm.weight model.layers.11.self_attn.o_proj.weight model.layers.11.self_attn.v_proj.weight model.layers.12.input_layernorm.weight model.layers.12.mlp.down_proj.weight model.layers.12.mlp.gate_proj.weight model.layers.12.mlp.up_proj.weight model.layers.12.post_attention_layernorm.weight model.layers.12.self_attn.k_proj.weight model.layers.12.self_attn.o_proj.weight model.layers.12.self_attn.q_proj.weight model.layers.12.self_attn.v_proj.weight model.layers.13.input_layernorm.weight model.layers.13.mlp.down_proj.weight model.layers.13.mlp.gate_proj.weight model.layers.13.mlp.up_proj.weight model.layers.13.post_attention_layernorm.weight model.layers.13.self_attn.k_proj.weight model.layers.13.self_attn.o_proj.weight model.layers.13.self_attn.q_proj.weight model.layers.13.self_attn.v_proj.weight model.layers.14.input_layernorm.weight model.layers.14.mlp.down_proj.weight model.layers.14.mlp.gate_proj.weight model.layers.14.mlp.up_proj.weight model.layers.14.post_attention_layernorm.weight model.layers.14.self_attn.k_proj.weight model.layers.14.self_attn.o_proj.weight model.layers.14.self_attn.q_proj.weight model.layers.14.self_attn.v_proj.weight model.layers.15.input_layernorm.weight model.layers.15.mlp.down_proj.weight model.layers.15.mlp.gate_proj.weight model.layers.15.mlp.up_proj.weight model.layers.15.post_attention_layernorm.weight model.layers.15.self_attn.k_proj.weight model.layers.15.self_attn.o_proj.weight model.layers.15.self_attn.q_proj.weight model.layers.15.self_attn.v_proj.weight model.layers.16.input_layernorm.weight model.layers.16.mlp.down_proj.weight model.layers.16.mlp.gate_proj.weight model.layers.16.mlp.up_proj.weight model.layers.16.post_attention_layernorm.weight model.layers.16.self_attn.k_proj.weight model.layers.16.self_attn.o_proj.weight model.layers.16.self_attn.q_proj.weight model.layers.16.self_attn.v_proj.weight model.layers.17.input_layernorm.weight model.layers.17.mlp.down_proj.weight model.layers.17.mlp.gate_proj.weight model.layers.17.mlp.up_proj.weight model.layers.17.post_attention_layernorm.weight model.layers.17.self_attn.k_proj.weight model.layers.17.self_attn.o_proj.weight model.layers.17.self_attn.q_proj.weight model.layers.17.self_attn.v_proj.weight model.layers.18.input_layernorm.weight model.layers.18.mlp.down_proj.weight model.layers.18.mlp.gate_proj.weight model.layers.18.mlp.up_proj.weight model.layers.18.post_attention_layernorm.weight model.layers.18.self_attn.k_proj.weight model.layers.18.self_attn.o_proj.weight model.layers.18.self_attn.q_proj.weight model.layers.18.self_attn.v_proj.weight model.layers.19.input_layernorm.weight model.layers.19.mlp.down_proj.weight model.layers.19.mlp.gate_proj.weight model.layers.19.mlp.up_proj.weight model.layers.19.post_attention_layernorm.weight model.layers.19.self_attn.k_proj.weight model.layers.19.self_attn.o_proj.weight model.layers.19.self_attn.q_proj.weight model.layers.19.self_attn.v_proj.weight model.layers.20.input_layernorm.weight model.layers.20.mlp.down_proj.weight model.layers.20.mlp.gate_proj.weight model.layers.20.mlp.up_proj.weight model.layers.20.post_attention_layernorm.weight model.layers.20.self_attn.k_proj.weight model.layers.20.self_attn.o_proj.weight model.layers.20.self_attn.q_proj.weight model.layers.20.self_attn.v_proj.weight model.layers.21.input_layernorm.weight model.layers.21.mlp.down_proj.weight model.layers.21.mlp.gate_proj.weight model.layers.21.mlp.up_proj.weight model.layers.21.post_attention_layernorm.weight model.layers.21.self_attn.k_proj.weight model.layers.21.self_attn.o_proj.weight model.layers.21.self_attn.q_proj.weight model.layers.21.self_attn.v_proj.weight model.layers.22.input_layernorm.weight model.layers.22.mlp.down_proj.weight model.layers.22.mlp.gate_proj.weight model.layers.22.mlp.up_proj.weight model.layers.22.post_attention_layernorm.weight model.layers.22.self_attn.k_proj.weight model.layers.22.self_attn.o_proj.weight model.layers.22.self_attn.q_proj.weight model.layers.22.self_attn.v_proj.weight model.layers.23.self_attn.k_proj.weight model.layers.23.self_attn.o_proj.weight model.layers.23.self_attn.q_proj.weight model.layers.23.self_attn.v_proj.weight lm_head.weight model.layers.23.input_layernorm.weight model.layers.23.mlp.down_proj.weight model.layers.23.mlp.gate_proj.weight model.layers.23.mlp.up_proj.weight model.layers.23.post_attention_layernorm.weight model.layers.24.input_layernorm.weight model.layers.24.mlp.down_proj.weight model.layers.24.mlp.gate_proj.weight model.layers.24.mlp.up_proj.weight model.layers.24.post_attention_layernorm.weight model.layers.24.self_attn.k_proj.weight model.layers.24.self_attn.o_proj.weight model.layers.24.self_attn.q_proj.weight model.layers.24.self_attn.v_proj.weight model.layers.25.input_layernorm.weight model.layers.25.mlp.down_proj.weight model.layers.25.mlp.gate_proj.weight model.layers.25.mlp.up_proj.weight model.layers.25.post_attention_layernorm.weight model.layers.25.self_attn.k_proj.weight model.layers.25.self_attn.o_proj.weight model.layers.25.self_attn.q_proj.weight model.layers.25.self_attn.v_proj.weight model.layers.26.input_layernorm.weight model.layers.26.mlp.down_proj.weight model.layers.26.mlp.gate_proj.weight model.layers.26.mlp.up_proj.weight model.layers.26.post_attention_layernorm.weight model.layers.26.self_attn.k_proj.weight model.layers.26.self_attn.o_proj.weight model.layers.26.self_attn.q_proj.weight model.layers.26.self_attn.v_proj.weight model.layers.27.input_layernorm.weight model.layers.27.mlp.down_proj.weight model.layers.27.mlp.gate_proj.weight model.layers.27.mlp.up_proj.weight model.layers.27.post_attention_layernorm.weight model.layers.27.self_attn.k_proj.weight model.layers.27.self_attn.o_proj.weight model.layers.27.self_attn.q_proj.weight model.layers.27.self_attn.v_proj.weight model.layers.28.input_layernorm.weight model.layers.28.mlp.down_proj.weight model.layers.28.mlp.gate_proj.weight model.layers.28.mlp.up_proj.weight model.layers.28.post_attention_layernorm.weight model.layers.28.self_attn.k_proj.weight model.layers.28.self_attn.o_proj.weight model.layers.28.self_attn.q_proj.weight model.layers.28.self_attn.v_proj.weight model.layers.29.input_layernorm.weight model.layers.29.mlp.down_proj.weight model.layers.29.mlp.gate_proj.weight model.layers.29.mlp.up_proj.weight model.layers.29.post_attention_layernorm.weight model.layers.29.self_attn.k_proj.weight model.layers.29.self_attn.o_proj.weight model.layers.29.self_attn.q_proj.weight model.layers.29.self_attn.v_proj.weight model.layers.30.input_layernorm.weight model.layers.30.mlp.down_proj.weight model.layers.30.mlp.gate_proj.weight model.layers.30.mlp.up_proj.weight model.layers.30.post_attention_layernorm.weight model.layers.30.self_attn.k_proj.weight model.layers.30.self_attn.o_proj.weight model.layers.30.self_attn.q_proj.weight model.layers.30.self_attn.v_proj.weight model.layers.31.input_layernorm.weight model.layers.31.mlp.down_proj.weight model.layers.31.mlp.gate_proj.weight model.layers.31.mlp.up_proj.weight model.layers.31.post_attention_layernorm.weight model.layers.31.self_attn.k_proj.weight model.layers.31.self_attn.o_proj.weight model.layers.31.self_attn.q_proj.weight model.layers.31.self_attn.v_proj.weight model.norm.weight Writing ch_taide_medicine.gguf-unsloth.F16.gguf, Traceback (most recent call last):
File "/GPUData/working/unsloth/convert if True: model.save_pretrained_gguf("c File "/home/chtseng/envs/LM2/lib/pytho file_location = save_to_gguf(model_type, File "/home/chtseng/envs/LM2/lib/pytho raise RuntimeError(
RuntimeError: Unsloth: Quantization You might have to compile llama.cpp You do not need to close this You must run this in the same git clone h_taide_medicine.gguf", tokenizer, quantization_method = "quantized")
n3.10/site-packages/unsloth/save.py", line 1333, in unsloth_save_pretrained_gguf
new_save_directory, quantization_method, first_conversion, makefile)
n3.10/site-packages/unsloth/save.py", line 957, in save_to_gguf
failed for ./ch_taide_medicine.gguf-unsloth.F16.gguf
yourself, then run this again.
Python program. Run the following commands in a new terminal:
folder as you're saving your model.
ggerganov/llama.cpp">https://github.com/ggerganov/llama.cpp
clean && LLAMA_CUDA=1 make all -j
n.

The text was updated successfully, but these errors were encountered:

danielhanchen · 2024-04-16T17:54:16Z

@ch-tseng Sorry on the issue - you'll have to unfortunately manually convert it to GGUF via https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf.

danielhanchen · 2024-04-21T19:34:51Z

I'm unsure if the latest release fixes this

sozercan mentioned this issue Apr 19, 2024

[REQ] update unsloth to llama3 release sozercan/aikit#199

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed at model.save_pretrained_gguf #341

Failed at model.save_pretrained_gguf #341

ch-tseng commented Apr 16, 2024

danielhanchen commented Apr 16, 2024

danielhanchen commented Apr 21, 2024

Failed at model.save_pretrained_gguf #341

Failed at model.save_pretrained_gguf #341

Comments

ch-tseng commented Apr 16, 2024

danielhanchen commented Apr 16, 2024

danielhanchen commented Apr 21, 2024