[LoRA] Fix SDXL text encoder LoRAs

[sayakpaul]

This PR fixes the LoRA loading part for the text encoders, especially the ones we see for SDXL 0.9.

Additionally, it also refactors some of the loading logic to help make the state_dict parsing easier.

I tested with the following:

from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
import safetensors
import torch 

base_model_id = "stabilityai/stable-diffusion-xl-base-0.9"
pipeline = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=torch.float16).to("cuda:0")
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
    pipeline.scheduler.config, algorithm_type="sde-dpmsolver++"
)
pipeline.load_lora_weights(".", weight_name="daiton.safetensors")

prompt = "alien, monster, daiton, high resolution, 4k, photorealistic, in a spaceship, gloomy"
generator = torch.manual_seed(22)
num_inference_steps = 50
guidance_scale = 5

image = pipeline(
    prompt=prompt, num_inference_steps=num_inference_steps,
    generator=generator, guidance_scale=guidance_scale
).images[0]
image.save("image.png")

And got:

LoRA is from here: https://civitai.com/models/108448/daiton-sdxl-test

@patrickvonplaten I see we don't do SLOW tests for SDXL, probably because we run the SLOW tests on a V100, which might fall short of running SDXL. I am bringing it up because I wanted to add a couple of slow tests to test_lora_layers.py to test the SDXL LoRAs from CivitAI.

Some comments are in line.

Also, I have run the SLOW tests and they pass successfully.

[sayakpaul]

Like: https://github.com/huggingface/diffusers/pull/4371/files#r1278549857

[adhikjoshi]

Can we show ValueError as warning log (non supported lora ) and move on instead of raise error?

[patrickvonplaten]

I think a ValueError is great here

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[adhikjoshi]

Lycoris still not working for SDXL

https://civitai.com/models/108011/fcbodybuildingxl-10-for-sdxl

[sayakpaul]

Lycoris still not working for SDXL

https://civitai.com/models/108011/fcbodybuildingxl-10-for-sdxl

I think we talked about it in the other thread too. It might be happening because LyCORIS adapters can contain LoHA modules too which we don't support yet.

Let's try not to duplicate these things :-)

[patrickvonplaten]

Very nice! Only left a couple of minor suggestions

[sayakpaul]

Merging a potentially impactful one. Fingers crossed!