`use_safetensors` is not passed to `load_sub_model`

[AnthonyBarbier]

Describe the bug

In DiffusionPipeline.from_pretrained() (pipeline_utils.py) the use_safetensors option is only passed to the downloader but not to load_sub_model() which means if the safetensors are in the cached folder then they will be loaded.

Reproduction

from diffusers import StableDiffusionPipeline

model_id = "stabilityai/stable-diffusion-2"
StableDiffusionPipeline.from_pretrained( model_id ) # Will add the safetensors model to the cache

StableDiffusionPipeline.from_pretrained(
            model_id,
            use_safetensors=False, # Parameter ignored -> model.safetensors is still loaded
 )

TRANSFORMERS_VERBOSITY=debug python3 repro.py

loading weights file /home/anthonyb/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2/snapshots/1e128c8891e52218b74cde8f26dbfc701cb99d79/text_encoder/model.safetensors

Changing the default value for use_safetensors from None to False in PreTrainedModel.from_pretrained() (in transformers/modeling_utils.py) to emulate what would happen if the value was propagated to the loader shows that the safetensors file is discarded and the model is loaded from bin:

loading weights file /home/anthonyb/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2/snapshots/1e128c8891e52218b74cde8f26dbfc701cb99d79/text_encoder/pytorch_model.bin

Logs

System Info

Version: 0.30.3

Who can help?

@sayakpaul @DN6

[AnthonyBarbier]

FYI: the other way around doesn't work either:

from diffusers import StableDiffusionPipeline
model_id = "stabilityai/stable-diffusion-2"
pipe = StableDiffusionPipeline.from_pretrained(
            model_id,
            use_safetensors=False,
)
print(f"{pipe.text_encoder.text_model.encoder.layers[0].self_attn.q_proj.bias.data_ptr()/ 4.0}")
print(f"{pipe.text_encoder.text_model.encoder.layers[0].self_attn.q_proj.weight.data_ptr()/ 4.0}")
pipe = StableDiffusionPipeline.from_pretrained(
            model_id,
            use_safetensors=True,
)
print(f"{pipe.text_encoder.text_model.encoder.layers[0].self_attn.q_proj.bias.data_ptr()/ 4.0}")
print(f"{pipe.text_encoder.text_model.encoder.layers[0].self_attn.q_proj.weight.data_ptr()/ 4.0}")

Will use model.bin in both cases.

[DN6]

Hi @AnthonyBarbier I'm unable to reproduce this behaviour using your snippets. Here's the logs from running with safetensors=True

loading configuration file /home/dhruv/.cache/hub/models--stabilityai--stable-diffusion-2/snapshots/1e128c8891e52218b74cde8f26dbfc701cb99d79/text_encoder/config.json
Model config CLIPTextConfig {
  "architectures": [
    "CLIPTextModel"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "dropout": 0.0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_size": 1024,
  "initializer_factor": 1.0,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 77,
  "model_type": "clip_text_model",
  "num_attention_heads": 16,
  "num_hidden_layers": 23,
  "pad_token_id": 1,
  "projection_dim": 512,
  "torch_dtype": "float32",
  "transformers_version": "4.51.3",
  "vocab_size": 49408
}

loading weights file /home/dhruv/.cache/hub/models--stabilityai--stable-diffusion-2/snapshots/1e128c8891e52218b74cde8f26dbfc701cb99d79/text_encoder/model.safetensors
All model checkpoint weights were used when initializing CLIPTextModel.

And then setting it use_safetensors=False downloads the model.bin file

Detected torch version: 2.7.0
pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 1.36G/1.36G [00:03<00:00, 412MB/s]

And uses it in the model

loading weights file /home/dhruv/.cache/hub/models--stabilityai--stable-diffusion-2/snapshots/1e128c8891e52218b74cde8f26dbfc701cb99d79/text_encoder/pytorch_model.bin
All model checkpoint weights were used when initializing CLIPTextModel.

We also pass use_safetensors to load_sub_model here

diffusers/src/diffusers/pipelines/pipeline_loading_utils.py

Line 784 in 4af76d0

loading_kwargs["use_safetensors"] = use_safetensors

Setting a break point just before loading the model you can see that it's set correctly.

> /home/dhruv/diffusers/src/diffusers/pipelines/pipeline_loading_utils.py(796)load_sub_model()
    795
--> 796     if isinstance(loaded_sub_model, torch.nn.Module) and isinstance(device_map, dict):
    797         # remove hooks

ipdb> is_transformers_model
True
ipdb> loading_kwargs
{'torch_dtype': None, 'device_map': None, 'max_memory': None, 'offload_folder': None, 'offload_state_dict': None, 'use_safetensors': False, 'low_cpu_mem_usage': True}
ipdb>

Is this happening when you try to load the model directly?

from transformers import AutoModel
model_id = "stabilityai/stable-diffusion-2"

AutoModel.from_pretrained(model_id, subfolder="text_encoder", use_safetensors=False)

[AnthonyBarbier]

Doh!

Looks like it's been fixed last year (v0.31): 1287822

I hadn't realised we had pinned our tests to 0.30.3

Sorry for wasting your time, and thank you for looking into this.