fix: compatibility with torch<=2.7 by andylin-hao · Pull Request #46393 · huggingface/transformers

andylin-hao · 2026-06-04T03:53:20Z

What does this PR do?

This PR fixes #46389, which is transformers' compatibility with torch<=2.7 which does not have FP8 support (missing torch.float8_e8m0fnu).

When installing 5.10.1 with torch<=2.7, importing AutoProcessor from transformers will import torch.float8_e8m0fnu in this line:

transformers/src/transformers/integrations/finegrained_fp8.py

Line 45 in d3f0591

_UE8M0_SF_DTYPE = torch.float8_e8m0fnu

As a result, the import will fail with ModuleNotFoundError with the following stack:

>>> from transformers import AutoProcessor
Traceback (most recent call last):
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2254, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2488, in _get_module
    raise e
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2486, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/.python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/models/auto/processing_auto.py", line 25, in <module>
    from ...image_processing_utils import ImageProcessingMixin
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/image_processing_utils.py", line 34, in <module>
    from .processing_utils import ImagesKwargs, Unpack
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/processing_utils.py", line 83, in <module>
    from .modeling_utils import PreTrainedAudioTokenizerBase
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/modeling_utils.py", line 69, in <module>
    from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/integrations/finegrained_fp8.py", line 45, in <module>
    _UE8M0_SF_DTYPE = torch.float8_e8m0fnu
                      ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/openvla/lib/python3.11/site-packages/torch/__init__.py", line 2681, in __getattr__
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'float8_e8m0fnu'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2342, in __getattr__
    raise ModuleNotFoundError(
ModuleNotFoundError: Could not import module 'AutoProcessor'. Are this object's requirements defined correctly?
>>> ^D
^[[A#                                                                                                                                                                                                     
(openvla) ➜  transformers git:(bugfix/torch_compat) python
Python 3.11.14 (main, Jan 27 2026, 23:58:49) [Clang 21.1.4 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoProcessor
Traceback (most recent call last):
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2254, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2488, in _get_module
    raise e
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2486, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/.python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/models/auto/processing_auto.py", line 25, in <module>
    from ...image_processing_utils import ImageProcessingMixin
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/image_processing_utils.py", line 34, in <module>
    from .processing_utils import ImagesKwargs, Unpack
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/processing_utils.py", line 83, in <module>
    from .modeling_utils import PreTrainedAudioTokenizerBase
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/modeling_utils.py", line 69, in <module>
    from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS
  File "/opt/venv/openvla/lib/python3.11/site-packages/transformers/integrations/finegrained_fp8.py", line 45, in <module>
    _UE8M0_SF_DTYPE = torch.float8_e8m0fnu
                      ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/openvla/lib/python3.11/site-packages/torch/__init__.py", line 2681, in __getattr__
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'float8_e8m0fnu'

This can be reproduced by installing torch==2.7 and transformers 5.10.1. Since transformers' dependency only requires torch>=2.4, I believe it's necessary to maintain support for these non-FP8 torch versions.

The fix uses getattr to check if FP8 is supported in the current torch and modifies deepgemm_compatible based on the check.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

vasqu

LGTM but cc @IlyasMoutawwakil just in case

transformers 5.10.x (released 2026-06-03) evaluates torch.float8_e8m0fnu at import time (integrations/finegrained_fp8.py), which requires torch >= 2.7. The diffusion pool installs torch from the cu124 index, which tops out at torch 2.6.0, so the worker's `from diffusers import StableDiffusionPipeline` crashes with AttributeError: module 'torch' has no attribute 'float8_e8m0fnu'. This made test_batch_diffusion time out (45 min) in 4/4 attempts of the nightly --aws build 11109: the mapper dies before the first batch is fed, the managed job stays RUNNING at 0% instead of failing (separate framework issue), and the test hits its timeout. Upstream: huggingface/transformers#46389 (fix PR huggingface/transformers#46393 not yet released). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

IlyasMoutawwakil

LGTM ! thanks for the fix

vasqu · 2026-06-04T10:04:50Z

@andylin-hao could you sync with main (merge/rebase)? I think CI should be green then 🤗 would merge after

Signed-off-by: Hao Lin <linhaomails@gmail.com>

andylin-hao · 2026-06-04T11:58:26Z

@andylin-hao could you sync with main (merge/rebase)? I think CI should be green then 🤗 would merge after

Hi @vasqu, I just rebased main. Thanks!

IlyasMoutawwakil · 2026-06-04T12:14:23Z

 _FP8_MIN = torch.finfo(_FP8_DTYPE).min
 _FP8_MAX = torch.finfo(_FP8_DTYPE).max
-_UE8M0_SF_DTYPE = torch.float8_e8m0fnu
+_UE8M0_SF_DTYPE = getattr(torch, "float8_e8m0fnu", None)


sorry just noticed the default is None, hmm this will break the parameter creation no ? there's also some to(_UE8M0_SF_DTYPE) in the code, can you check please

Yes. I think all the usages _UE8M0_SF_DTYPE require scale_fmt to be "ue8m0", and passing None to tensor.to() is basically a no-op with no side effect. But on the other hand, this may be a silent failure when the tensor should be converted to FP8 but really is not. So I added a function to raise with clear error message when this occurs (users request scale_fmt to be ue8m0 but FP8 is not supported by the torch version). What do you think?

HuggingFaceDocBuilderDev · 2026-06-04T12:25:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Signed-off-by: Hao Lin <linhaomails@gmail.com>

github-actions · 2026-06-04T13:46:32Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46393&sha=086be5

transformers 5.10.x (released 2026-06-03) evaluates torch.float8_e8m0fnu at import time (integrations/finegrained_fp8.py), which requires torch >= 2.7. The diffusion pool installs torch from the cu124 index, which tops out at torch 2.6.0, so the worker's `from diffusers import StableDiffusionPipeline` crashes with AttributeError: module 'torch' has no attribute 'float8_e8m0fnu'. This made test_batch_diffusion time out (45 min) in 4/4 attempts of the nightly --aws build 11109: the mapper dies before the first batch is fed, the managed job stays RUNNING at 0% instead of failing (separate framework issue), and the test hits its timeout. Upstream: huggingface/transformers#46389 (fix PR huggingface/transformers#46393 not yet released). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-05T06:39:15Z

CI Dashboard: View test results in Grafana

andylin-hao changed the title ~~fix: compatibility with no-fp8 torch~~ fix: compatibility with torch<=2.7 Jun 4, 2026

zpoint mentioned this pull request Jun 4, 2026

[Test] Pin transformers<5.10 in batch diffusion example skypilot-org/skypilot#9799

Merged

vasqu approved these changes Jun 4, 2026

View reviewed changes

IlyasMoutawwakil approved these changes Jun 4, 2026

View reviewed changes

Rocketknight1 mentioned this pull request Jun 4, 2026

AutoProcessor import fails with NVIDIA torch 2.7 missing torch.float8_e8m0fnu #46389

Closed

fix: compatibility with no-fp8 torch

12c96ac

Signed-off-by: Hao Lin <linhaomails@gmail.com>

andylin-hao force-pushed the bugfix/torch_compat branch from 37602b2 to 12c96ac Compare June 4, 2026 11:57

IlyasMoutawwakil enabled auto-merge June 4, 2026 12:12

IlyasMoutawwakil disabled auto-merge June 4, 2026 12:13

IlyasMoutawwakil reviewed Jun 4, 2026

View reviewed changes

andylin-hao and others added 5 commits June 4, 2026 20:28

Merge branch 'main' into bugfix/torch_compat

bbfc568

fix: raise if scale_fmt is fp8 without fp8 support

4e02667

Signed-off-by: Hao Lin <linhaomails@gmail.com>

Merge branch 'main' into bugfix/torch_compat

8b10fe3

fix: lint error

78a4fac

Signed-off-by: Hao Lin <linhaomails@gmail.com>

Merge branch 'main' into bugfix/torch_compat

086be55

IlyasMoutawwakil approved these changes Jun 5, 2026

View reviewed changes

IlyasMoutawwakil enabled auto-merge June 5, 2026 06:42

IlyasMoutawwakil added this pull request to the merge queue Jun 5, 2026

Merged via the queue into huggingface:main with commit 10ed83f Jun 5, 2026
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: compatibility with torch<=2.7#46393

fix: compatibility with torch<=2.7#46393
IlyasMoutawwakil merged 6 commits into
huggingface:mainfrom
andylin-hao:bugfix/torch_compat

andylin-hao commented Jun 4, 2026

Uh oh!

vasqu left a comment

Uh oh!

IlyasMoutawwakil left a comment

Uh oh!

vasqu commented Jun 4, 2026

Uh oh!

andylin-hao commented Jun 4, 2026

Uh oh!

IlyasMoutawwakil Jun 4, 2026

Uh oh!

andylin-hao Jun 4, 2026 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andylin-hao commented Jun 4, 2026

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu commented Jun 4, 2026

Uh oh!

andylin-hao commented Jun 4, 2026

Uh oh!

IlyasMoutawwakil Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

andylin-hao Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andylin-hao Jun 4, 2026 •

edited

Loading