SDXL LoRA key not loaded "lora_te2_text_projection.*" (LoRA from OneTrainer)

[tin2tin]

Describe the bug

When trying to use a LoRA in Diffusers, created in OneTrainer:

These keys are not loaded:

Python\Python311\site-packages\diffusers\loaders\lora_conversion_utils.py", line 273, in _convert_kohya_lora_to_diffusers
    raise ValueError(f"The following keys have not been correctly be renamed: \n\n {', '.join(state_dict.keys())}")
ValueError: The following keys have not been correctly be renamed:

 lora_te2_text_projection.lora_down.weight, lora_te2_text_projection.lora_up.weight

Reading up on the topic here: Nerogar/OneTrainer#47 reveals that it is not considered a OneTrainer bug, and something which should be fixed externally.

This is how the bug is described:

So the question is: can the attempt to load the extra text encoder, also be ignored, if not found, so we can use OneTrainer LoRAs in Diffusers?

Reproduction

Import a SDXL LoRA from OneTrainer into Diffusers.
(The one I've got is far too big to share)

Logs

No response

System Info

Windows 11, Diffusers latest alpha version.

Who can help?

@sayakpaul

[tin2tin]

Crosspost: Nerogar/OneTrainer#314

[sayakpaul]

You have to provide a fully reproducible code snippet for us to look into this further.

[tin2tin]

from diffusers import DiffusionPipeline
import torch

pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("C:/Users/user_name/Downloads/LORA/", weight_name="lora.safetensors", adapter_name="engraving")
lora_scale= 0.9
prompt = "Engravings Henri Meyer"
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
).images[0]
image

The LoRA file: https://filetransfer.io/data-package/SKG6lXd3#link

[sayakpaul]

If at all possible, please also host the LoRA file on the Hub so that is easily retrievable.

[tin2tin]

Thank you to all involved for prioritizing and resolving this issue superfast!