Bug in tranformers/modeling_utils.py

[nightandweather]

System Info

I use torch 1.13.1 / and following bitsandbytes cmake install

------------------------------------------------------------------------------------

Training...
  0%|                                                                                                                                                             | 0/3000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/kanghoun/1.Language_Model/amc_emr_oncology_LLM/llama_test.py", line 394, in <module>
    train(model, tokenizer, dataset, file_path)
  File "/home/kanghoun/1.Language_Model/amc_emr_oncology_LLM/llama_test.py", line 352, in train
    train_result = trainer.train()
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1780, in train
    return inner_training_loop(
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2134, in _inner_training_loop
    self.current_flos += float(self.floating_point_ops(inputs))
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 3813, in floating_point_ops
    return self.model.floating_point_ops(inputs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1141, in floating_point_ops
    return 6 * self.estimate_tokens(input_dict) * self.num_parameters(exclude_embeddings=exclude_embeddings)
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1089, in num_parameters
    param.numel() * 2 * self.hf_quantizer.quantization_config.bnb_4bit_quant_storage.itemsize
AttributeError: 'torch.dtype' object has no attribute 'itemsize'
----------------------------------------------------------------------------------------

I changed self.hf_quantizer.quantization_config.bnb_4bit_quant_storage.itemsize 
-> torch_dtype_itemsize(self.hf_quantizer.quantization_config.bnb_4bit_quant_storage)

`import torch

def torch_dtype_itemsize(dtype):
    """
    Get the element size of the given torch.dtype object.

    Parameters:
        dtype (torch.dtype): The torch data type.

    Returns:
        int: The element size of the given data type.
    """
    # Dictionary mapping torch dtype to their respective sizes
    dtype_sizes = {
        torch.uint8: 1,
        torch.int8: 1,
        torch.int16: 2,
        torch.int32: 4,
        torch.int64: 8,
        torch.float16: 2,
        torch.float32: 4,
        torch.float64: 8,
    }

    # Retrieve the size from the dictionary, or return None if not found
    return dtype_sizes.get(dtype, None)

Example usage

tensor = torch.tensor([1, 2, 3], dtype=torch.uint8)
itemsize = torch_dtype_itemsize(tensor.dtype)
print("itemsize:", itemsize)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. I use torch 1.13.1 / and following bitsandbytes cmake install

2. in torch 1.13.1, there is no torch.dtype.itemsize

3. but tranformers/modeling_utils.py

   transformers/src/transformers/modeling_utils.py

   Line 1089 in c5f0288

   param.numel() * 2 * self.hf_quantizer.quantization_config.bnb_4bit_quant_storage.itemsize

   
   use itemsize()

so I make new function to work in

Expected behavior

---

[scouzi1966]

EDIT AFTER INITIAL POST: Problem seems to be related to Python 3.11. Managed to run with Python 3.10 and torch-2.2.1-cp310-cp310-manylinux1_x86_64.whl.metadata

I have encountered the same issue.

My params are:

bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_compute_dtype=compute_dtype,
bnb_4bit_use_double_quant=True,
)

Error:

File ~/miniconda3/envs/311_gpu/lib/python3.11/site-packages/transformers/modeling_utils.py:1089, in ModuleUtilsMixin.num_parameters(self, only_trainable, exclude_embeddings)
1084 if param.requires_grad or not only_trainable:
1085 # For 4bit models, we need to multiply the number of parameters by 2 as half of the parameters are
1086 # used for the 4bit quantization (uint8 tensors are stored)
1087 if is_loaded_in_4bit and isinstance(param, bnb.nn.Params4bit):
1088 total_numel.append(
-> 1089 param.numel() * 2 * self.hf_quantizer.quantization_config.bnb_4bit_quant_storage.itemsize
1090 )
1091 else:
1092 total_numel.append(param.numel())

AttributeError: 'torch.dtype' object has no attribute 'itemsize'

[solablue]

THANK YOU!!

[amyeroberts]

cc @SunMarc @younesbelkada as it looks like it's quantization code which is triggering this, and should be compatible with multiple versions of python and torch

[younesbelkada]

Indeed ! #30162 should fix it - we've seen a simiar issue on PEFT recently huggingface/peft#1635