Quantization error - model.save_pretrained_gguf(new_model, tokenizer, quantization_method = "q4_k_m")

[dynamite9999]

Hello All,
I have been saving llama3 in gguf for weeks and was working fine.
Only today, I started getting the error, I tried everything including the suggestion git clone and make clean / make all with the flags.

Any suggestions / hints to get past this issue, very much appreciated.
Traceback (most recent call last):
File "/home/d/hp/dev/syslog/syslog_scraper/t59_nie_func_data/t13.py", line 1276, in
main()
File "/home/d/hp/dev/syslog/syslog_scraper/t59_nie_func_data/t13.py", line 1231, in main
model.save_pretrained_gguf(TRAINED_GGUF_MODEL, tokenizer, quantization_method = "q4_k_m")
File "/home/d/.local/lib/python3.11/site-packages/unsloth/save.py", line 1340, in unsloth_save_pretrained_gguf
file_location = save_to_gguf(model_type, new_save_directory, quantization_method, first_conversion, makefile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/d/.local/lib/python3.11/site-packages/unsloth/save.py", line 964, in save_to_gguf
raise RuntimeError(
RuntimeError: Unsloth: Quantization failed for ./TRAINED_GGUF_MODEL-unsloth.F16.gguf
You might have to compile llama.cpp yourself, then run this again.
You do not need to close this Python program. Run the following commands in a new terminal:
You must run this in the same folder as you're saving your model.
git clone --recursive https://github.com/ggerganov/llama.cpp
cd llama.cpp && make clean && LLAMA_CUDA=1 make all -j
Once that's done, redo the quantization.

[danielhanchen]

@dynamite9999 Can you try reinstalling and updating Unsloth?

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

[dynamite9999]

Yes! Thank you , it worked. Appreciate it.

…

On Sun, Jun 2, 2024 at 11:36 PM Daniel Han ***@***.***> wrote: @dynamite9999 <https://github.com/dynamite9999> Can you try reinstalling and updating Unsloth? pip uninstall unsloth -y pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git — Reply to this email directly, view it on GitHub <#579 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANVUTBAYISSOJ2KPJUEP6TLZFQFHVAVCNFSM6AAAAABIVLXSZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBUGM4DMNZTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[danielhanchen]

Great!

[devzzzero]

I think I just got bit by this as well. Is there a way to get unsloth to use a preinstalled version of llama.cpp?
It's dying on llama.cpp compile, after a clean uninstall/reinstall of unsloth via directions above.

I got this error
p  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_OPENMP -DGGML_USE_LLAMAFILE  -c common/grammar-parser.cpp -o grammar-parser.o
c++: error: unrecognized command line option ‘-Wextra-semi’; did you mean ‘-Wextra’?
c++: error: unrecognized command line option ‘-Wextra-semi’; did you mean ‘-Wextra’?
c++: error: unrecognized command line option ‘-Wextra-semi’; did you mean ‘-Wextra’?
make: *** [Makefile:766: llama.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:769: common.o] Error 1
make: *** [Makefile:778: grammar-parser.o] Error 1
c++: error: unrecognized command line option ‘-Wextra-semi’; did you mean ‘-Wextra’?
make: *** [Makefile:772: sampling.o] Error 1
make: Leaving directory '/home/ai/LLM/PEFT/llama.cpp'

[danielhanchen]

Oh weird - what's your GCC version? There is a way to install a specific version of llama.cpp

[devzzzero]

Oh weird - what's your GCC version? There is a way to install a specific version of llama.cpp

gcc-13 (opensuse Leap 15.5).

I'm actually not too fussed about this actually.

I think having 27 different copies of llama.cpp across my 18 different conda envs, is just wasteful!

The work around is to manually convert the merged model to gguf (using a locally installed llama.cpp). Which works for me, for now :-)

Thank you!
LOL!

[danielhanchen]

This should be fixed now - sorry on the delay - please update with uv pip install --upgrade unsloth unsloth_zoo --force-reinstall --no-cache-dir - if it's not fixed yet, it'll be in the next pip / PyPI release - if it's still not working after that, please open another issue - appreciate it again!