Versions greater than 4.49 are not compatible with Ascend NPU

[1737686924]

System Info

910B1,python==3.10,transformers==4.51.0

Who can help?

When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

Expected behavior

When torch==2.2.0 and torch-npu==2.2.0 and transformers>4.49, the error "AssertionError: Torch not compiled with CUDA enabled" will appear. <=4.49 has no problem. Upgrading to torch==2.4.0 and torch-npu==2.4.0 and transformers>4.49 allows training, but the graphics card cannot be specified. By default, it starts from card 0.

[Rocketknight1]

cc @ivarflakstad, with apologies!

[ivarflakstad]

Hey!
Just as a general note when you decide to use an older version of one library (torch in this instance) it is not completely unexpected that you would get issues with newer versions of another (transformers in this instance). Transformers supports a fairly wide range of torch, but when using torch-npu I'd recommend a smaller range as the entire ecosystem around npus is less mature. That's true for any software btw, not specific to transformers, torch, etc.

Have you tried torch* > 2.4.0?

[1737686924]

I discovered the problem. I upgraded torch*==2.5.1 and solved this problem