Unable to load google/siglip2-base-patch16-naflex

[EricWiener]

System Info

Using git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2.

Who can help?

@ariG23498 @qubvel

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the example from the docs:

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("google/siglip2-base-patch16-naflex")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

candidate_labels = ["2 cats", "2 dogs"]
texts = [f"This is a photo of {label}." for label in candidate_labels]

inputs = processor(text=texts, images=image, max_num_patches=256, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

logits_per_image = outputs.logits_per_image
probs = torch.sigmoid(logits_per_image) # these are the probabilities
print(f"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'")

https://huggingface.co/docs/transformers/main/model_doc/siglip2

This results in:

self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.

Expected behavior

The probabilities would be printed out.

[qubvel]

Hey @EricWiener, thanks for opening the issue! Can you please try installing the latest main branch to check if the error still exists for your env?

pip install -U git+https://github.com/huggingface/transformers

[EricWiener]

Thanks @qubvel ! Just tried that and now I get:

E           RuntimeError: Failed to import transformers.models.siglip2.modeling_siglip2 because of the following error (look up to see its traceback):
E           module 'torch.distributed' has no attribute 'device_mesh'

which seems like it may be an issue with the version of torch (2.2.0a0+git6c8c5ad) I am using. Will investigate and follow up

[EricWiener]

I was able to get torch.distributed.device_mesh to load using 2.6.0 but it failed on 2.2.1 and 2.2.2. Given that the setup.py file requires torch>=2.0, maybe it would be best to lazy import torch.distributed.device_mesh to avoid this causing problems for users who don't need to train SigLip?

[EricWiener]

Even if I monkey patch that to work around it I still get:

self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.

[qubvel]

Thanks for reporting issue with device_mesh, it's indeed a bug, however, I'm not able to reproduce the following error with typing typing.Optional requires a single type. Got <module 'PIL.Image' .... Can you send your env params? transformers-cli env

[qubvel]

Also can you send full traceback to get the idea where this type error occurs?

[EricWiener]

Hi @qubvel , I'm running via bazel so a bit difficult to get transformers-cli env working but here is env info:

• Python version: 3.9.19
• PyTorch version: 2.2.0a0+git6c8c5ad
• Transformers version: 4.50.0.dev0
• CUDA available: True
• CUDA version: 12.4
• GPU device: NVIDIA GeForce RTX 3090

Here is longer stack trace:

>       model = AutoModel.from_pretrained("google/siglip2-base-patch16-naflex")

/path/../embedding_models_test.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../dd_py_deps_transformers/site-packages/transformers/models/auto/processing_auto.py:343: in from_pretrained
    return processor_class.from_pretrained(
../dd_py_deps_transformers/site-packages/transformers/processing_utils.py:1070: in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
../dd_py_deps_transformers/site-packages/transformers/processing_utils.py:1134: in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
../dd_py_deps_transformers/site-packages/transformers/models/auto/image_processing_auto.py:523: in from_pretrained
    image_processor_class = get_image_processor_class_from_name(image_processor_type)
../dd_py_deps_transformers/site-packages/transformers/models/auto/image_processing_auto.py:194: in get_image_processor_class_from_name
    return getattr(module, class_name)
../dd_py_deps_transformers/site-packages/transformers/utils/import_utils.py:1937: in __getattr__
    module = self._get_module(self._class_to_module[name])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <module 'transformers.models.siglip2' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_...e_mining/embedding_models_test.runfiles/dd_py_deps_transformers/site-packages/transformers/models/siglip2/__init__.py'>
module_name = 'image_processing_siglip2_fast'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
>           raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e
E           RuntimeError: Failed to import transformers.models.siglip2.image_processing_siglip2_fast because of the following error (look up to see its traceback):
E           typing.Optional requires a single type. Got <module 'PIL.Image' from '/root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/execroot/_.

../dd_py_deps_transformers/site-packages/transformers/utils/import_utils.py:1951: RuntimeError

[EricWiener]

Hi, any updates on this? Anything I can do to assist?