[0.28.0]: from_single_file doesn't work with custom checkpoints

[putdanil]

Describe the bug

Can't load custom SDXL checkpoint with from_single_file with custom checkpoints since 0.28.0 version. All SDXL pipelines don't work, with or without controlnet.
Triedsetting low_cpu_mem_usage=False and ignore_mismatched_sizes=True, it doesn't change anything.

Reproduction

pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(
"checkpoint_sdxl_generation.safetensors",
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
controlnet=[controlnet_midas_sdxl, controlnet_canny_sdxl],
vae=vae
).to('cuda')

Logs

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(
      2             "checkpoint_sdxl_generation.safetensors",
      3             torch_dtype=torch.float16,
      4             variant="fp16",
      5             use_safetensors=True,
      6             controlnet=[controlnet_midas_sdxl, controlnet_canny_sdxl],
      7             safety_checker= None,
      8             load_safety_checker=False,
      9             vae=vae,
     10             add_watermarker=False
     11         ).to('cuda')

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:114, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    111 if check_use_auth_token:
    112     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 114 return fn(*args, **kwargs)

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/diffusers/loaders/single_file.py:491, in FromSingleFileMixin.from_single_file(cls, pretrained_model_link_or_path, **kwargs)
    489 else:
    490     try:
--> 491         loaded_sub_model = load_single_file_sub_model(
    492             library_name=library_name,
    493             class_name=class_name,
    494             name=name,
    495             checkpoint=checkpoint,
    496             is_pipeline_module=is_pipeline_module,
    497             cached_model_config_path=cached_model_config_path,
    498             pipelines=pipelines,
    499             torch_dtype=torch_dtype,
    500             original_config=original_config,
    501             local_files_only=local_files_only,
    502             is_legacy_loading=is_legacy_loading,
    503             **kwargs,
    504         )
    505     except SingleFileComponentError as e:
    506         raise SingleFileComponentError(
    507             (
    508                 f"{e.message}\n"
   (...)
    514             )
    515         )

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/diffusers/loaders/single_file.py:100, in load_single_file_sub_model(library_name, class_name, name, checkpoint, pipelines, is_pipeline_module, cached_model_config_path, original_config, local_files_only, torch_dtype, is_legacy_loading, **kwargs)
     97     if original_config:
     98         cached_model_config_path = None
--> 100     loaded_sub_model = load_method(
    101         pretrained_model_link_or_path_or_dict=checkpoint,
    102         original_config=original_config,
    103         config=cached_model_config_path,
    104         subfolder=name,
    105         torch_dtype=torch_dtype,
    106         local_files_only=local_files_only,
    107         **kwargs,
    108     )
    110 elif is_transformers_model and is_clip_model_in_single_file(class_obj, checkpoint):
    111     loaded_sub_model = create_diffusers_clip_model_from_ldm(
    112         class_obj,
    113         checkpoint=checkpoint,
   (...)
    118         is_legacy_loading=is_legacy_loading,
    119     )

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:114, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    111 if check_use_auth_token:
    112     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 114 return fn(*args, **kwargs)

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/diffusers/loaders/single_file_model.py:273, in FromOriginalModelMixin.from_single_file(cls, pretrained_model_link_or_path_or_dict, **kwargs)
    270     model = cls.from_config(diffusers_model_config)
    272 if is_accelerate_available():
--> 273     unexpected_keys = load_model_dict_into_meta(model, diffusers_format_checkpoint, dtype=torch_dtype)
    274     if model._keys_to_ignore_on_load_unexpected is not None:
    275         for pat in model._keys_to_ignore_on_load_unexpected:

File ~/anaconda3/envs/openvino/lib/python3.10/site-packages/diffusers/models/model_loading_utils.py:120, in load_model_dict_into_meta(model, state_dict, device, dtype, model_name_or_path)
    118 if empty_state_dict[param_name].shape != param.shape:
    119     model_name_or_path_str = f"{model_name_or_path} " if model_name_or_path is not None else ""
--> 120     raise ValueError(
    121         f"Cannot load {model_name_or_path_str}because {param_name} expected shape {empty_state_dict[param_name]}, but got {param.shape}. If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example."
    122     )
    124 if accepts_dtype:
    125     set_module_tensor_to_device(model, param_name, device, value=param, dtype=dtype)

ValueError: Cannot load because down_blocks.1.attentions.0.proj_in.weight expected shape tensor(..., device='meta', size=(640, 640, 1, 1)), but got torch.Size([640, 640]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.

System Info

ValueError: Cannot load because down_blocks.1.attentions.0.proj_in.weight expected shape tensor(..., device='meta', size=(640, 640, 1, 1)), but got torch.Size([640, 640]). If you want to instead overwrite randomly initialized weights, please make sure to pass both low_cpu_mem_usage=False and ignore_mismatched_sizes=True. For more information, see also: #1619 (comment) as an example.

Who can help?

No response

[DN6]

@putdanil can you install diffusers from main and try again? Is it possible for you to host the checkpoint you're trying to load on the Hugging Face hub and share it here?

[DN6]

@putdanil By any chance is this a merged inpainting checkpoint?

[putdanil]

@putdanil By any chance is this a merged inpainting checkpoint?

it is, I'm in the process of uploading it

[putdanil]

@DN6 here is the link dputilov/juggernautxlinpaint
https://huggingface.co/datasets/dputilov/juggernautxlinpaint/resolve/main/checkpoint_sdxl_generation.safetensors

[DN6]

@putdanil Could you try passing in the config argument when loading the model?

pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(
    "checkpoint_sdxl_generation.safetensors",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
    controlnet=[controlnet_midas_sdxl, controlnet_canny_sdxl],
    vae=vae,
    config="diffusers/stable-diffusion-xl-1.0-inpainting-0.1"
).to('cuda')

[putdanil]

@DN6 Thanks! Everything works now.