transformers import_utils.py module 'torch' has no attribute 'fx'

[R0k1e]

System Info

env: transformers=4.35.2, torch = 1.12.1+cu113, Ubuntu 20.04.6 LTS python=3.10
when I write this sentence "from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM", It tell me it cant find fx in torch. I inspect the doc of torch version and found there is fx module in this version. And when I use the console to import torch.fx, there is no fault happened. And the I add the import torch.fx before the "from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM", the error misses.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. install the llama according to its readme
2. install the torch = 1.12.1+cu113
3. write a footprint including following sentences
   import argparse
   import json
   import os
   import time

import pandas as pd
import tensor_parallel as tp
import torch

from tqdm import tqdm
from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM

and then the error occur
from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM
File "", line 1075, in _handle_fromlist
File "/miniconda3/envs/llama-7b/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1344, in getattr
value = getattr(module, name)
File "/miniconda3/envs/llama-7b/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1343, in getattr
module = self._get_module(self._class_to_module[name])
File "/miniconda3/envs/llama-7b/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1355, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
module 'torch' has no attribute 'fx'

Expected behavior

no error occur

[amyeroberts]

Hi @R0k1e, thanks for raising this issue!

So that we can best help you could you provide some more information:

• The running environment: run transformers-cli env in the terminal and copy-paste the output

install the llama according to its readme

• Could you link to the README? Which llama is being referred to here?

I inspect the doc of torch version and found there is fx module in this version. And when I use the console to import torch.fx,

Could you open a python session, run the following and reply with what is printed out?:

print("Importing torch")
import torch
print(torch.__version__)

print("Importing torch.fx")
import torch.fx

print("Importing transformers")
import transformers
print(transformers.__version__)

print("Importing transformers models")
from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM

[R0k1e]

transformers-cli env

• transformers version: 4.35.2
• Platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.31
• Python version: 3.10.0
• Huggingface_hub version: 0.19.3
• Safetensors version: 0.4.0
• Accelerate version: 0.24.1
• Accelerate config: not found
• PyTorch version (GPU?): 1.12.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Yes, using tensor_parallel to fuifil this function. To be more precise, the script is like the following format:

import tensor_parallel as tp
model = LlamaForCausalLM.from_pretrained(ckpt_dir, low_cpu_mem_usage = True, torch_dtype=torch.float16)
model = tp.tensor_parallel(model, [i for i in range(n_gpus)])

llama README

https://github.com/facebookresearch/llama
According to this README, I create a conda env with python3.10, and then use the pip install -e . to build up the env. And then replace the torch 2.1.1 to torch 1.12.1 to suit the cuda version.

python session

>>> print("Importing torch")
Importing torch
>>> import torch
>>> print(torch.__version__)
1.12.1
>>> 
>>> print("Importing torch.fx")
Importing torch.fx
>>> import torch.fx
>>> 
>>> print("Importing transformers")
Importing transformers
>>> import transformers
>>> print(transformers.__version__)
4.35.2
>>> 
>>> print("Importing transformers models")
Importing transformers models
>>> from transformers import LlamaForCausalLM, LlamaTokenizer, AutoTokenizer, AutoModelForCausalLM
>>> 

If I ignore the import torch.fx, the error occurs as before

[jessyford]

same here, temporarily solved by downgrading transformers 4.35.2 to 4.34.0

[jessyford]

more tests:
4.35.0-4.35.2: same error
4.34.0-4.34.1: succeed

[amyeroberts]

@jessyford @R0k1e OK - thanks for testing and reporting back. I've open a PR which should resolve this issue for you.

As a side note - you don't need to follow installation instructions from the original Llama repo to use the model. You can just install transformers and use the model directly with the latest pytorch versions.

[patrickocal]

Hi, I'm getting the same error.

Traceback (most recent call last):
  File "/scratch/project_mnt/S0066/unlimiformer-11-dec-kg/src/run.py", line 25, in <module>
    from unlimiformer import Unlimiformer
  File "/scratch/project_mnt/S0066/unlimiformer-11-dec-kg/src/unlimiformer.py", line 6, in <module>
    from transformers import BartModel, BartForConditionalGeneration, \
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/uqpocall/micromamba/envs/unlimiformer11/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1344, in __getattr__
    value = getattr(module, name)
  File "/home/uqpocall/micromamba/envs/unlimiformer11/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1343, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/uqpocall/micromamba/envs/unlimiformer11/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1355, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
module 'torch' has no attribute 'fx'

Here is the output from transformers-cli env:

• transformers version: 4.35.2
• Platform: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.28
• Python version: 3.10.13
• Huggingface_hub version: 0.17.3
• Safetensors version: 0.3.3
• Accelerate version: 0.25.0
• Accelerate config: not found
• PyTorch version (GPU?): 1.12.1.post200 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

I am trying to install this particular version as these are the instructions I found here: https://github.com/abertsch72/unlimiformer and I am trying to reproduce their results. Also, I got very different results just by changing environments (sequence of sequences ... ... were treated differently by bart-base in 4.34.1 vs 4.35.2)

[patrickocal]

fixed it by adding from torch.fx import symbolic_trace to the relevant module.

[amyeroberts]

Hi @patrickocal, the error would have still be present if running from 4.35.x as the fix had been merged but was not yet part of a release i.e. to have it requires installing from source. We've just (1 hour ago!) released 4.36 which contains the fix. If after installing 4.36 the issue persists, please let us know in a comment.