cannot import name 'is_torch_xla_available' from 'transformers'

[YooSungHyun]

System Info

transformers==4.38.2
torch==2.2.1

rtx 3090 * 4 and cuda 12.1

Who can help?

@muellerz @pacman100 @stevhliu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. run https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling 's run_clm.py
2. use this script

python run_clm.py \
    --model_name_or_path openai-community/gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm

3. raised error like cannot import name 'is_torch_xla_available' from 'transformers' at 49 line

Expected behavior

import is success

[amyeroberts]

Hi @YooSungHyun, this would indicate to me that there's an issue with the transformers version installed in your environment e.g. the python path and the pip path are not aligned.

I'm able to run the following without issue:

from transformers import is_torch_xla_available

[borodache]

There is a problem with version: transformers==4.38.2
try version 2.2

[amyeroberts]

try version 2.2

@borodache Could you clarify? Version 2.2 of what (pytorch?)

[borodache]

of transformers wasn't that clear?

[amyeroberts]

No - this is why I asked. I'm not sure why you'd suggest using v2.2 of transformers, which was released 2 major versions and 5 years ago.

[borodache]

You have a point. look at this post: #1956
it solves the issue with version 2.2. However, it doesn't mean that newer versions do not work with it. You can try and let me know...

[davies-w]

I have exactly the same issue, trying to run exactly the same script. This must be a bug in HuggingFace code, as it is LITERALLY in their script (run_clm) and even after running pip install requirements, it still fails.

[davies-w]

Ok, I think I figured out the problem, and I think it stems from the instructions that we probably both found (https://github.com/marshmellow77/nlp-model-shakespeare/blob/main/train-nlp-model.ipynb). By cloning out the current version, you mess up the dependencies with transformers, which will probably not be the same version as is installed.

I forced this to the following (you can of course change what you want about the version, I just checked out a recent stable release).

!git clone --single-branch --branch v4.39-release https://github.com/huggingface/transformers
!pip install transformers==4.39.1
print(transformers.version)
!pip install -r transformers/examples/pytorch/language-modeling/requirements.txt

[ozamaahmed]

Ok, I think I figured out the problem, and I think it stems from the instructions that we probably both found (https://github.com/marshmellow77/nlp-model-shakespeare/blob/main/train-nlp-model.ipynb). By cloning out the current version, you mess up the dependencies with transformers, which will probably not be the same version as is installed.

I forced this to the following (you can of course change what you want about the version, I just checked out a recent stable release).

!git clone --single-branch --branch v4.39-release https://github.com/huggingface/transformers !pip install transformers==4.39.1 print(transformers.version) !pip install -r transformers/examples/pytorch/language-modeling/requirements.txt

i did exactly this and im having this error

2024-03-26 10:57:33.635984: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-26 10:57:33.636052: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-26 10:57:33.638226: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-26 10:57:36.295064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "/content/drive/MyDrive/mlm_code/run_mlm.py", line 58, in <module>
    check_min_version("4.40.0.dev0")
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py", line 251, in check_min_version
    raise ImportError(
ImportError: This example requires a source install from HuggingFace Transformers (see `[https://huggingface.co/docs/transformers/installation#install-from-source`](https://huggingface.co/docs/transformers/installation#install-from-source%60)), but the version found is 4.39.1.
Check out https://github.com/huggingface/transformers/tree/main/examples#important-note for the examples corresponding to other versions of HuggingFace Transformers.```

[minyang-chen]

hi, I ran into same error with setup of 4060-ti x 2 and cuda 12.1, manage to fix the import error.

BEFORE
transformers==4.38.2
torch==2.2.1

ERROR:

CUDA_VISIBLE_DEVICES=0,1 python examples/pytorch/language-modeling/run_clm.py --model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
/miniconda/envs/llama_factory/lib/python3.10/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
Traceback (most recent call last):
  File "/home/pop/hosting/colab/bonito_dataset/transformers/examples/pytorch/language-modeling/run_clm.py", line 59, in <module>
    **check_min_version("4.40.0.dev0")**
  File "/miniconda/envs/llama_factory/lib/python3.10/site-packages/transformers/utils/__init__.py", line 251, in check_min_version
    raise ImportError(
ImportError: This example requires a source install from HuggingFace Transformers (see `https://huggingface.co/docs/transformers/installation#install-from-source`), but the version found is 4.39.1.
Check out https://github.com/huggingface/transformers/tree/main/examples#important-note for the examples corresponding to other versions of HuggingFace Transformers.

INVESTIGATION:
according to error message, required minimal transformer version: 4.40.0.dev0

WORKAROUND:
install latest main branch

git clone https://github.com/huggingface/transformers.git
pip install -e .

install the rest requirements

pip install -r transformers/examples/pytorch/language-modeling/requirements.txt

AFTER:

Grouping texts in chunks of 1024: 100%|███████████████████████████████████████████████████████████████████| 3760/3760 [00:00<00:00, 22833.65 examples/s]
[INFO|trainer.py:561] 2024-03-27 02:15:29,602 >> max_steps is given, it will override any value given in num_train_epochs
[INFO|trainer.py:1995] 2024-03-27 02:15:29,727 >> ***** Running training *****
[INFO|trainer.py:1996] 2024-03-27 02:15:29,727 >>   Num examples = 2,318
[INFO|trainer.py:1997] 2024-03-27 02:15:29,727 >>   Num Epochs = 1
[INFO|trainer.py:1998] 2024-03-27 02:15:29,727 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2000] 2024-03-27 02:15:29,727 >>   Training with DataParallel so batch size has been adjusted to: 8
[INFO|trainer.py:2001] 2024-03-27 02:15:29,727 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2002] 2024-03-27 02:15:29,727 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2003] 2024-03-27 02:15:29,727 >>   Total optimization steps = 200
[INFO|trainer.py:2004] 2024-03-27 02:15:29,727 >>   Number of trainable parameters = 124,439,808

hope this helps!

[InzamamAnwar]

@YooSungHyun you can replace

from transformers import is_torch_xla_available

with

from transformers import is_torch_tpu_available

but this has been deprecated and will be removed in 4.41.0. Find more details in the following function

transformers/src/transformers/utils/import_utils.py

Line 518 in 536ea2a

def is_torch_tpu_available(check_device=True):

[minyang-chen]

@YooSungHyun you can replace

from transformers import is_torch_xla_available

with

from transformers import is_torch_tpu_available

but this has been deprecated and will be removed in 4.41.0. Find more details in the following function

transformers/src/transformers/utils/import_utils.py

Line 518 in 536ea2a

def is_torch_tpu_available(check_device=True):

Nice finding, thanks for the info.