Florence 2 _supports_sdpa missing in remote code

[Znerual]

System Info

• transformers version: 4.57.0.dev0
• Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
• Python version: 3.12.11
• Huggingface_hub version: 1.0.0.rc5
• Safetensors version: 0.6.2
• Accelerate version: 1.10.1
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU

Who can help?

@yonigozlan @molbap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Running the example script:

import requests

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=False)

prompt = "<OD>"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=4096,
    num_beams=3,
    do_sample=False
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

Leads to the following error:

`torch_dtype` is deprecated! Use `dtype` instead!
Traceback (most recent call last):
  File "/home/ruzickal/Code/entity-analyzer-backend/audio-llm-api/test.py", line 11, in <module>
    model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/models/auto/auto_factory.py", line 378, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 270, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 4473, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/.cache/huggingface/modules/transformers_modules/microsoft/Florence_hyphen_2_hyphen_large/21a599d414c4d928c9032694c424fb94458e3594/modeling_florence2.py", line 2535, in __init__
    super().__init__(config)
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 1822, in __init__
    self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 2416, in _check_and_adjust_attn_implementation
    applicable_attn_implementation = self.get_correct_attn_implementation(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 2450, in get_correct_attn_implementation
    self._sdpa_can_dispatch(is_init_check)
  File "/home/ruzickal/Code/entity-analyzer-backend/transformers/src/transformers/modeling_utils.py", line 2314, in _sdpa_can_dispatch
    if not self._supports_sdpa:
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ruzickal/.cache/pypoetry/virtualenvs/audio-llm-api-nPR59pGu-py3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1962, in __getattr__
    raise AttributeError(
AttributeError: 'Florence2ForConditionalGeneration' object has no attribute '_supports_sdpa'

Expected behavior

Output the detections

[zucchini-nlp]

You need to use the transformers implementation without trust_remote_code (docs: https://huggingface.co/docs/transformers/en/model_doc/florence2)

[Shippdenn]

You need to use the transformers implementation without trust_remote_code (docs: https://huggingface.co/docs/transformers/en/model_doc/florence2)

sorry for asking but how should I do that ?

[zucchini-nlp]

You need to delete trust_remote_code=True when loading the model and use this checkpoint florence-community/Florence-2-base. The models in community org are the ones which are compatible with transformers

[Znerual]

Thank you for your help!

[a7md5aliid]

You need to delete trust_remote_code=True when loading the model and use this checkpoint florence-community/Florence-2-base. The models in community org are the ones which are compatible with transformers

Thank you man, this thing was going to drive me crazy! Error fixed and finally a new error popped up <3

[JingHHj]

Hey @zucchini-nlp I came across the same issue earlier today, then I found this post. But it still not working after I take trust_remote_code=False off

**Here is the code I am running: **

import requests

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=False).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=False)

prompt = "<OD>"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=4096,
    num_beams=3,
    do_sample=False
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

**Here is the result I got: **

(venv) jing@jing-Lenovo-Legion-Y7000-2021:~/file/learn/deployment/florence-2$ python official_florence2.py 
config.json: 2.44kB [00:00, 5.45MB/s]
`torch_dtype` is deprecated! Use `dtype` instead!
Traceback (most recent call last):
  File "/home/jing/file/learn/deployment/florence-2/official_florence2.py", line 11, in <module>
    model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=False).to(device)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 573, in from_pretrained
    trust_remote_code = resolve_trust_remote_code(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 782, in resolve_trust_remote_code
    raise ValueError(
ValueError: The repository microsoft/Florence-2-large contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/microsoft/Florence-2-large .
 You can inspect the repository content at https://hf.co/microsoft/Florence-2-large.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

**And after I take them off, I am running with: ***

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large")

*it just back to the original bug

(venv) jing@jing-Lenovo-Legion-Y7000-2021:~/file/learn/deployment/florence-2$ python official_florence2.py 
`torch_dtype` is deprecated! Use `dtype` instead!
The repository microsoft/Florence-2-large contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/microsoft/Florence-2-large .
 You can inspect the repository content at https://hf.co/microsoft/Florence-2-large.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
modeling_florence2.py: 127kB [00:00, 17.7MB/s]
configuration_florence2.py: 15.1kB [00:00, 22.2MB/s]
A new version of the following files was downloaded from https://huggingface.co/microsoft/Florence-2-large:
- configuration_florence2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/microsoft/Florence-2-large:
- modeling_florence2.py
- configuration_florence2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
model.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1.55G/1.55G [01:30<00:00, 17.1MB/s]
Traceback (most recent call last):
  File "/home/jing/file/learn/deployment/florence-2/official_florence2.py", line 11, in <module>
    model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype).to(device)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 597, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4971, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/.cache/huggingface/modules/transformers_modules/microsoft/Florence_hyphen_2_hyphen_large/21a599d414c4d928c9032694c424fb94458e3594/modeling_florence2.py", line 2535, in __init__
    super().__init__(config)
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2076, in __init__
    self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2686, in _check_and_adjust_attn_implementation
    applicable_attn_implementation = self.get_correct_attn_implementation(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2722, in get_correct_attn_implementation
    self._sdpa_can_dispatch(is_init_check)
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2573, in _sdpa_can_dispatch
    if not self._supports_sdpa:
           ^^^^^^^^^^^^^^^^^^^
  File "/home/jing/file/learn/deployment/florence-2/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1940, in __getattr__
    raise AttributeError(
AttributeError: 'Florence2ForConditionalGeneration' object has no attribute '_supports_sdpa'

[zucchini-nlp]

Hey @JingHHj , you need to use the hf-format checkpoints in https://huggingface.co/florence-community for that

[JingHHj]

@zucchini-nlp Thanks!