convert.py and safetensors

[SrVill]

Greetings!
Tell me if it is possible to convert .safetensors to .gguf using convert.py? Or just from .bin?

[KerfuffleV2]

It should support safetensors. I'm not sure if that functionality has been tested recently though. Give it a shot and create an issue if you encounter issues.

[SrVill]

I failed to convert safetensors:
Traceback (most recent call last): File "E:\koboldcpp\convert.py", line 1224, in <module> main() File "E:\koboldcpp\convert.py", line 1165, in main model_plus = load_some_model(args.model) File "E:\koboldcpp\convert.py", line 1076, in load_some_model raise Exception(f"Can't find model in directory {path}") Exception: Can't find model in directory E:\koboldcpp\M

[KerfuffleV2]

That's really not enough information to help you. You also seem to be using koboldcpp not llama.cpp. Please try to reproduce the issue with llama.cpp and provide more information about the model you used, the parameters, etc. Or you can try making an issue at the koboldcpp repo instead.

Even though koboldcpp is derived llama.cpp, it has various modifications.

[SrVill]

Converted as you suggested. The script works with safetensors.

Initially, I was confused that this format was not specified when calling --help. It would probably be nice to add *.safetensors to line 1137 convert.py: help="directory containing model file, or model file itself (*.pth, *.pt, *.bin)"

Thank you for your help!

[takosalad]

How does this supposedly work?

First:
$ python convert-hf-to-gguf.py models/Mixtral-8x7B-exl2_3.00bpw
Loading model: Mixtral-8x7B-exl2_3.00bpw
gguf: This GGUF file is for Little Endian only
Set model parameters
Set model tokenizer
gguf: Setting special token type bos to 1
gguf: Setting special token type eos to 2
gguf: Setting special token type unk to 0
gguf: Setting add_bos_token to True
gguf: Setting add_eos_token to False
gguf: Setting chat_template to {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}
Exporting model to 'models/Mixtral-8x7B-exl2_3.00bpw/ggml-model-f16.gguf'
gguf: loading model part 'model-00001-of-00003.safetensors'
Traceback (most recent call last):
File "/home/takosalad/llama2/llama.cpp/convert-hf-to-gguf.py", line 1046, in
model_instance.write()
File "/home/takosalad/llama2/llama.cpp/convert-hf-to-gguf.py", line 136, in write
self.write_tensors()
File "/home/takosalad/llama2/llama.cpp/convert-hf-to-gguf.py", line 97, in write_tensors
for name, data_torch in self.get_tensors():
File "/home/takosalad/llama2/llama.cpp/convert-hf-to-gguf.py", line 60, in get_tensors
ctx = cast(ContextManager[Any], safe_open(self.dir_model / part_name, framework="pt", device="cpu"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: No such file or directory: "models/Mixtral-8x7B-exl2_3.00bpw/model-00001-of-00003.safetensors"

Then after I renamed the three files manually from output-0000... to model-0000...:
$ python convert-hf-to-gguf.py models/Mixtral-8x7B-exl2_3.00bpw
Loading model: Mixtral-8x7B-exl2_3.00bpw
gguf: This GGUF file is for Little Endian only
Set model parameters
Set model tokenizer
gguf: Setting special token type bos to 1
gguf: Setting special token type eos to 2
gguf: Setting special token type unk to 0
gguf: Setting add_bos_token to True
gguf: Setting add_eos_token to False
gguf: Setting chat_template to {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}
Exporting model to 'models/Mixtral-8x7B-exl2_3.00bpw/ggml-model-f16.gguf'
gguf: loading model part 'model-00001-of-00003.safetensors'
token_embd.weight, n_dims = 2, torch.float16 --> float16
Can not map tensor 'model.layers.0.block_sparse_moe.experts.0.w1.q_groups'

Nothing works.

[cebtenzzre]

How does this supposedly work?

Based on the "exl2_3.00bpw" part, you are trying to load a model that has been quantized for Exllama. You need an original FP16 model such as this one to use it with convert.py - then you must quantize it for llama.cpp afterwards. Of course, TheBloke has already converted and quantized Mixtral if you'd like to take the easy route: https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF

[takosalad]

Oh thanks, I didn't know that you can't convert quantized models. That explains a lot. :|
It's like one of these things that isn't explained anywhere and you just can't know it unless someone tells you, typical for llama space.

[592319702]

I want to convert safetensors to gguf with convert.py
But I hit the issue as follow,

(mlx) I542464@DY4GPKX1J0 llama.cpp % python convert.py ../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned
Loading model file ../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/model-00001-of-00003.safetensors
Loading model file ../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/model-00001-of-00003.safetensors
Loading model file ../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/model-00002-of-00003.safetensors
Loading model file ../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/model-00003-of-00003.safetensors
params = Params(n_vocab=32000, n_embd=4096, n_layer=32, n_ctx=32768, n_ff=14336, n_head=32, n_head_kv=8, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=10000.0, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned'))
Found vocab files: {'tokenizer.model': None, 'vocab.json': None, 'tokenizer.json': PosixPath('../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/tokenizer.json')}
Loading vocab file '../mlx-examples/lora/lora_fused_model/My-Mistral-7B-fine-tuned/tokenizer.json', type 'spm'
Traceback (most recent call last):
  File "/Users/I542464/mlx-test/llama.cpp/convert.py", line 1474, in <module>
    main()
  File "/Users/I542464/mlx-test/llama.cpp/convert.py", line 1442, in main
    vocab, special_vocab = vocab_factory.load_vocab(args.vocab_type, model_parent_path)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/I542464/mlx-test/llama.cpp/convert.py", line 1328, in load_vocab
    vocab = SentencePieceVocab(
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/I542464/mlx-test/llama.cpp/convert.py", line 394, in __init__
    self.sentencepiece_tokenizer = SentencePieceProcessor(str(fname_tokenizer))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/I542464/miniconda3/envs/mlx/lib/python3.11/site-packages/sentencepiece/__init__.py", line 447, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "/Users/I542464/miniconda3/envs/mlx/lib/python3.11/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/I542464/miniconda3/envs/mlx/lib/python3.11/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Internal: /Users/runner/work/sentencepiece/sentencepiece/src/sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 

[Runeblue]

same here x_x

[cebtenzzre]

@592319702 @Runeblue Check out #5821, I think it should solve this issue.