Bug: Couldn't load GGUF file into Transformers

[PayteR]

What happened?

Hi, Im trying to load GGUF model into transformers and I'm doing it like it is in example here https://huggingface.co/docs/transformers/en/gguf

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "thirteenbit/madlad400-10b-mt-gguf"
filename = "model-q8_0.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

inputs = tokenizer("<2pt> I love pizza!", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

But I'm getting this error

Traceback (most recent call last):
  File "....path.../translate-run-madlad.py", line 14, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
  File "......path....../lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 851, in from_pretrained
    config_dict = load_gguf_checkpoint(gguf_path, return_tensors=False)["config"]
  File "......path....../lib/python3.10/site-packages/transformers/modeling_gguf_pytorch_utils.py", line 83, in load_gguf_checkpoint
    reader = GGUFReader(gguf_checkpoint_path)
  File "......path....../lib/python3.10/site-packages/gguf/gguf_reader.py", line 94, in __init__
    if self._get(offs, np.uint32, override_order = '<')[0] != GGUF_MAGIC:
  File "......path....../lib/python3.10/site-packages/gguf/gguf_reader.py", line 151, in _get
    .newbyteorder(override_order or self.byte_order)
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

It looks like you just need to substitute newbyteorder by arr.view(arr.dtype.newbyteorder(order)) , thx

Name and Version

It's i think on all versions https://github.com/ggerganov/llama.cpp/blob/master/gguf-py/gguf/gguf_reader.py

What operating system are you seeing the problem on?

No response

Relevant log output

No response

[superskirv]

I ran this to fix the issue, though I doubt its the ideal solution.

pip install numpy==1.26.3

Linux Mint(Ubuntu 22.04), 64gb ram, 3090ti

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[erasurecom]

Those who want to stay on NumPy>=2.0, e.g. to avoid "Dependency Hell", could change the code in gguf_reader.py file at line 148:

arr: npt.NDArray[Any] = self.data[offset:end_offs].view(dtype = dtype)[:count] # <- Add 'arr' variable for readability
return (
#self.data[offset:end_offs] <- comment out original line
#.view(dtype = dtype)[:count] <- comment out original line
#.newbyteorder(override_order or self.byte_order) <- comment out original line
arr.view(arr.dtype.newbyteorder(override_order or self.byte_order)) # <- Define new notation according to NumPy 2.0
)

[NeyroDesigner]

Those who want to stay on NumPy>=2.0, e.g. to avoid "Dependency Hell", could change the code in gguf_reader.py file at line 148:

arr: npt.NDArray[Any] = self.data[offset:end_offs].view(dtype = dtype)[:count] # <- Add 'arr' variable for readability return ( #self.data[offset:end_offs] <- comment out original line #.view(dtype = dtype)[:count] <- comment out original line #.newbyteorder(override_order or self.byte_order) <- comment out original line arr.view(arr.dtype.newbyteorder(override_order or self.byte_order)) # <- Define new notation according to NumPy 2.0 )

Pls, guys help!
i know that it's not for this theme, but i faced with next error after top.
'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte

[PJR1987]

Those who want to stay on NumPy>=2.0, e.g. to avoid "Dependency Hell", could change the code in gguf_reader.py file at line 148:

arr: npt.NDArray[Any] = self.data[offset:end_offs].view(dtype = dtype)[:count] # <- Add 'arr' variable for readability return ( #self.data[offset:end_offs] <- comment out original line #.view(dtype = dtype)[:count] <- comment out original line #.newbyteorder(override_order or self.byte_order) <- comment out original line arr.view(arr.dtype.newbyteorder(override_order or self.byte_order)) # <- Define new notation according to NumPy 2.0 )

This just f'd everything up for me even more:
File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\nodes.py", line 258, in load_unet
sd = gguf_sd_loader(unet_path)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\nodes.py", line 39, in gguf_sd_loader
reader = gguf.GGUFReader(path)
^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\gguf\gguf_reader.py", line 94, in init
if self._get(offs, np.uint32, override_order = '<')[0] != GGUF_MAGIC:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\gguf\gguf_reader.py", line 152, in _get
arr.view(arr.dtype.newbyteorder(override_order or self.byte_order)) # <- Define new notation according to NumPy 2.0
^^^
NameError: name 'arr' is not defined