Very slow model instantiation

[rvorias]

System Info

transformers version: 4.51.3

• Platform: Linux-6.8.0-57-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.30.2
• Safetensors version: 0.5.3
• Accelerate version: not installed
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA RTX 4000 SFF Ada Generation

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

this is the culprit:

transformers/src/transformers/safetensors_conversion.py

Line 26 in 9ec8be5

safetensors_convert_space_url = "https://safetensors-convert.hf.space"

update: when interrupting the process, it seems to be more likely it's stuck here: File "/home/vd/.venv/lib/python3.10/site-packages/torch/serialization.py", line 1888, in load_tensor zip_file.get_storage_from_record(name, numel, torch.UntypedStorage) KeyboardInterrupt

Rollback to 4.46.1 does not have this issue.

Expected behavior

NOT take 5 mins to create model instance.

[Rocketknight1]

Issue confirmed, we're working on a fix right now! Expect a PR today

[doublex]

@Rocketknight1
Thanks a lot!

[Rocketknight1]

Hi @doublex @rvorias we have a PR open here which should help a bit: #37751

But the main cause of the slow loading is the weights on the Florence-2 repos being in a legacy format. We've opened PRs to update them and also fix some modeling issues (like the deprecation warning it causes when loading from timm)

PR list:
https://huggingface.co/microsoft/Florence-2-large/discussions/103
https://huggingface.co/microsoft/Florence-2-base/discussions/27
https://huggingface.co/microsoft/Florence-2-large-ft/discussions/38
https://huggingface.co/microsoft/Florence-2-base-ft/discussions/29

While you're waiting for them to be merged, you can load the PR branches like this:

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", revision="pr/103")
model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", revision="pr/27")
model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", revision="pr/38")
model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base-ft", revision="pr/29")

[doublex]

Thank you for your merits!

[Rocketknight1]

Double-checking the PRs now, it's possible weight-tying issues lost the LM head weights. One sec!

[Rocketknight1]

@doublex PRs should now be working correctly!

[rvorias]

Thanks for picking up 🙏

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.