Enable LoRAs to patch the text_encoder as well as the unet #3214
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
damian0815
requested review from
keturn,
blessedcoolant and
psychedelicious
as code owners
damian0815
changed the title
psychedelicious
approved these changes
Apr 16, 2023
lstein
approved these changes
Apr 18, 2023
There was a problem hiding this comment.
Looks good. I appreciate the cleanup.
I think ultimately we need to come up with a way of registering modules that modify, the textencoder, tokenizer, unet, etc. The code now has a lot of different places where there is a check for a LoRA being present and different actions are taken depending on that. Maybe better to have a series of modules registered somewhere and then applied at points where they might want to do something. What do you think?
lstein
added a commit
that referenced
this pull request
Apr 19, 2023
It's like LoHA but use Kronecker product instead of Hadamard product. https://github.com/KohakuBlueleaf/LyCORIS#lokr I tested it on this 2 LoKR's: https://civitai.com/models/34518/unofficial-vspo-yakumo-beni https://civitai.com/models/35136/mika-pikazo-lokr More tests hard to find as it's new format) Better to test with #3214 Also a bit refactor forward function. //LyCORIS also have (IA)^3 format, but I can't find examples in this format and even on LyCORIS page it's marked as experimental. So, until there some test examples I prefer not to add this.
|
@GreggHelt2 and @blessedcoolant Could one of you have a look at this PR? It fixes LoRA issues and I'm hoping to put it into a 2.3.5 release. |
GreggHelt2
approved these changes
Apr 24, 2023
|
Yep. Quite a bit of work involved here, but I'll have it in in time for 3.0
L
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Load LoRAs during compel's text embedding encode pass in case there are requested LoRAs which also want to patch the text encoder.
Also generally cleanup the attention processor patching stuff. It's still a mess, but at least now it's a stateless mess.