Add support for SDXL LoRA models in diffusers format.

[RyanJDick]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update
• Community Node Submission

Have you discussed this change with the InvokeAI team?

• Yes
• No, because:

Have you updated all relevant documentation?

• Yes
• No

Description

This PR introduces 2 related changes:

1. This PR adds support for SDXL LoRA models in the diffusers format. There are 2 popular SDXL LoRA weight formats: Stability AI format and diffusers format (produced by diffusers, kohya_ss, and soon invoke-training). Prior to this change, InvokeAI only supported the Stability AI format, and trying to load a LoRA model in diffusers format would raise an exception.
2. This PR fixes a bug in the current version of InvokeAI that causes all text encoder LoRA weights to be ignored. In other words, prior to this change InvokeAI was only loading UNet LoRA weights for SDXL models.

Related Tickets & Documents

Relevant Discord conversation where someone encountered this issue: https://discord.com/channels/1020123559063990373/1133929466763161650/1139504370627661896

QA Instructions, Screenshots, Recordings

I tested the following models:

Voxel XL

• https://civitai.com/models/118536
• Trigger word: "voxel style"
• Format: Stability AI
• This LoRA was already 'working' before this PR. But the performance seems to be improved after fixing the text encoder bug.
• Before change:
  
• After change (Same seed and configs):
  

SDXL William Eggleston Style

• https://civitai.com/models/119667
• Trigger words: "by william eggleston"
• Format: diffusers
• Before this change: raised Exception
• After change:
  

Angelina Jolie SDXL

• https://civitai.com/models/125123/angelina-jolie-sdxl
• Trigger words: "ohwx woman"
• Format: diffusers
• Before this change: raised Exception
• After change:
  

Embroidery Style - SDXL

• https://civitai.com/models/122308
• Trigger words: None?
• Format: diffusers
• Before this change: raised Exception
• After change: The LoRA runs, but does not match the results on CivitAI. I'm still trying to figure out why.
  

Added/updated tests?

• Yes
• No : From what I can see, there is 0 test coverage of the LoRA loading code right now. Do we have any plans to change this?

[optional] Are there any post deployment tasks we need to perform?

[RyanJDick]

If anyone has ideas as to why the Embroidery Style LoRA isn't working as expected, I'd love to hear them. I'm still investigating this, but not sure if it's worth holding up this PR over it.