Understanding "wrong number of tensors" error when loading Llama3 model.

[eltonjohnfanboy]

Hi,

Recently, I've been experimenting with GGUF quantization, but I'm fairly new to it. While quantizing the model Llama3 8B, I encountered the following error: error loading model: done_getting_tensors: wrong number of tensors; expected 578, got 470.

I understand that there seems to be a mismatch between the expected tensors and the actual tensors received when loading the quantized model. However, I would like to understand more deeply why this error occurs and what the root cause is. This issue is particularly puzzling because I was able to successfully quantize the model using llama.cpp, but the error only appears when I attempt to load the model for inference.

If anyone could help me understand this better, I would be really grateful. Thanks!

[slaren]

It means that the gguf file contains more tensors than expected, which indicates that something went wrong during model conversion to gguf. During quantization this check is not done. You can use gguf-py/scripts/gguf-dump.py to see the list of tensors in the gguf file. The root cause would need to be investigated on a case by case basis, it may mean the model architecture is wrong, or that there is a bug in the conversion script.

[sinand99]

Getting the same error with the new LLM "granite-8b-code-instruct-GGUF" on HuggingFace. None of them work.

[eltonjohnfanboy]

@sinand99 The community just updated the llama.cpp for the granite 8B and 3B models yesterday. So any previously quantized model might encounter the same error/issue. I suggest re-quantizing the model using the latest convert-hf-to-gguf.py script :))

[10thfloor]

If you're here because of a problem in LMStudio. Try restarting LMStudio.
That worked for me. (Models were breaking after update).

[orenong]

If you're here because of a problem in LMStudio. Try restarting LMStudio. That worked for me. (Models were breaking after update).

True solution is to update, updates require a restart to be applied