TypeError: PeftAdapterMixin.load_adapter() got an unexpected keyword argument 'adapter_state_dict'

[tin2tin]

Describe the bug

When loading LoRAs with the new multiple LoRAs adapter, some SDXL LoRAs fail loading with this error:
TypeError: PeftAdapterMixin.load_adapter() got an unexpected keyword argument 'adapter_state_dict'
Ex.
https://civitai.com/models/57437/andrei-tarkovsky-style
https://civitai.com/models/107140/weegee-style

However, this one works without that error:
https://civitai.com/models/133900/lora-marvin-the-martian-looney-tunes

Reproduction

Load some of the LoRA models from above with the multiple LoRAs adapter.

Logs

`TypeError: PeftAdapterMixin.load_adapter() got an unexpected keyword argument 'adapter_state_dict'`

System Info

Win 11

Who can help?

@younesbelkada @sayakpaul

[younesbelkada]

hi @tin2tin for that you need to update transformers - can you try to build transformers from source?

pip install -U git+https://github.com/huggingface/transformers.git

What is the transformers version you are using?

[younesbelkada]

Can you also share the full traceback of the issue 🙏

[tin2tin]

What is the transformers version you are using?

4.33.2

[younesbelkada]

the feature to load peft adapters through state dict has been added only on 4.34: https://github.com/huggingface/transformers/releases/tag/v4.34.0 with huggingface/transformers#25721
can you try with that version instead?
@sayakpaul I think we should force users to use at least 4.34 by changing this line: https://github.com/huggingface/diffusers/blob/main/src/diffusers/utils/constants.py#L49 WDYT?

[tin2tin]

Cool, updating transformers made it work on SD 1.5 too.

NB. trying to load a 1.5 LoRA on SD XL results in a long error message, maybe incompatible LoRAs cound be catched, and a short error could be printed instead?

The current error:

Traceback (most recent call last):
  File "C:\Users\45239\AppData\Roaming\Blender Foundation\Blender\3.5\scripts\addons\Generative_AI-main\__init__.py", line 2798, in execute
    pipe.load_lora_weights(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\loaders.py", line 3226, in load_lora_weights
    self.load_lora_into_unet(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\loaders.py", line 1637, in load_lora_into_unet
    incompatible_keys = set_peft_model_state_dict(unet, state_dict, adapter_name)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\peft\utils\save_and_load.py", line 154, in set_peft_model_state_dict
    load_result = model.load_state_dict(peft_model_state_dict, strict=False)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for down_blocks.1.attentions.0.proj_in.lora_A.KlausKinski-06.weight: copying a param with shape torch.Size([16, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([16, 640]).
        size mismatch for down_blocks.1.attentions.0.proj_in.lora_B.KlausKinski-06.weight: copying a param with shape torch.Size([640, 16, 1, 1]) from checkpoint, the shape in current model is torch.Size([640, 16]).
        size mismatch for down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_k.lora_A.KlausKinski-06.weight: copying a param with shape torch.Size([16, 768]) from checkpoint, the shape in current model is torch.Size([16, 2048]).
        size mismatch for down... etc.

[sayakpaul]

@sayakpaul I think we should force users to use at least 4.34 by changing this line: https://github.com/huggingface/diffusers/blob/main/src/diffusers/utils/constants.py#L49 WDYT?

@younesbelkada sounds good to me!

NB. trying to load a 1.5 LoRA on SD XL results in a long error message, maybe incompatible LoRAs cound be catched, and a short error could be printed instead?

@tin2tin I don't think we should cache incorrect LoRAs. I prefer the full error trace too because it gives more information to the developers and also helps with issue reporting.