Runtime error with from_single_file

[suzukimain]

Describe the bug

Versioning diffusers to 0.29.0.dev0 did not solve the problem.

Reproduction

pip install diffusers
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_single_file("https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors")

Logs

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict, assign)
   2187 
   2188         if len(error_msgs) > 0:
-> 2189             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   2190                                self.__class__.__name__, "\n\t".join(error_msgs)))
   2191         return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

System Info

• 🤗 Diffusers version: 0.28.2
• Platform: Ubuntu 22.04.3 LTS - Linux-6.1.85+-x86_64-with-glibc2.35
• Running on a notebook?: No
• Running on Google Colab?: No
• Python version: 3.10.12
• PyTorch version (GPU?): 2.3.0+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
• Jax version: 0.4.26
• JaxLib version: 0.4.26
• Huggingface_hub version: 0.23.2
• Transformers version: 4.41.2
• Accelerate version: not installed
• PEFT version: not installed
• Bitsandbytes version: not installed
• Safetensors version: 0.4.3
• xFormers version: not installed
• Accelerator: Tesla T4, 15360 MiB VRAM
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[sayakpaul]

Cc: @DN6

[asomoza]

Hi, thanks for reporting. I tested this and It works with the 0.28.2 version. It's weird that it doesn't work for you. I suggest you try uninstalling diffusers and then install it again.

- 🤗 Diffusers version: 0.28.2

from diffusers import StableDiffusionPipeline


pipe = StableDiffusionPipeline.from_single_file(
    "https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors"
).to("cuda")

image = pipe(prompt="a happy capybara jumping on the street").images[0]
image.save("result.png")

[suzukimain]

Thank you for your swift response. I tried again, but it fails.

bug.mp4

Detailed log.

---

RuntimeError Traceback (most recent call last)
in <cell line: 6>()
4
5
----> 6 pipe = StableDiffusionPipeline.from_single_file(
7 "https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors"
8 ).to("cuda")

4 frames
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py in _inner_fn(*args, **kwargs)
112 kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.name, has_token=has_token, kwargs=kwargs)
113
--> 114 return fn(*args, **kwargs)
115
116 return _inner_fn # type: ignore

/usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file.py in from_single_file(cls, pretrained_model_link_or_path, **kwargs)
489 else:
490 try:
--> 491 loaded_sub_model = load_single_file_sub_model(
492 library_name=library_name,
493 class_name=class_name,

/usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file.py in load_single_file_sub_model(library_name, class_name, name, checkpoint, pipelines, is_pipeline_module, cached_model_config_path, original_config, local_files_only, torch_dtype, is_legacy_loading, **kwargs)
109
110 elif is_transformers_model and is_clip_model_in_single_file(class_obj, checkpoint):
--> 111 loaded_sub_model = create_diffusers_clip_model_from_ldm(
112 class_obj,
113 checkpoint=checkpoint,

/usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file_utils.py in create_diffusers_clip_model_from_ldm(cls, checkpoint, subfolder, config, torch_dtype, local_files_only, is_legacy_loading)
1380
1381 else:
-> 1382 model.load_state_dict(diffusers_format_checkpoint)
1383
1384 if torch_dtype is not None:

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict, assign)
2187
2188 if len(error_msgs) > 0:
-> 2189 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2190 self.class.name, "\n\t".join(error_msgs)))
2191 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

[asomoza]

oh yeah, you're right, in colab it gives an error but locally it's just a warning:

local:

Some weights of the model checkpoint were not used when initializing CLIPTextModel: 
 ['text_model.embeddings.position_ids']

colab:

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	Unexpected key(s) in state_dict: "text_model.embeddings.position_ids". 

I have the same pytorch and cuda version as in colab, the only difference is the python version. We'll have to wait for @DN6 for a solution it seems.

[asomoza]

Interesting, I installed transformers from source !pip install git+https://github.com/huggingface/transformers.git in the colab and I got a different error:

SingleFileComponentError: Failed to load CLIPTextModel. Weights for this component appear to be missing in the checkpoint.
Please load the component before passing it in as an argument to `from_single_file`.

[asomoza]

in the meantime, since that repo has a version in the diffusers format, why not use it like this?

pipe = StableDiffusionPipeline.from_pretrained("gsdf/Counterfeit-V2.5").to("cuda")

[suzukimain]

The same issue was also discussed here.

[suzukimain]

However, I'm not used to making pull requests, so if I'm wrong somewhere, please point it out.

[suzukimain]

Fixed in #8462 and I'll close it.
Thank you!