Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: LoRa's wont work #10036

Closed
1 task done
Muskelmagier opened this issue May 3, 2023 · 26 comments
Closed
1 task done

[Bug]: LoRa's wont work #10036

Muskelmagier opened this issue May 3, 2023 · 26 comments
Labels
bug-report Report of a bug, yet to be confirmed cannot-reproduce I can't reproduce this, so I can't fix it. Add steps for reproduction and remove this tag.

Comments

@Muskelmagier
Copy link

Is there an existing issue for this?

  • I have searched the existing issues and checked the recent builds/commits

What happened?

I have this error code when I use a LoRa, and they are not applied to the prompt

Steps to reproduce the problem

Using any lora

What should have happened?

LoRa's should be used

Commit where the problem happens

5ab7f21

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--deepdanbooru --api --no-half-vae --xformers

List of extensions

DreamArtist-sd-webui-extension https://github.com/7eu7d7/DreamArtist-sd-webui-extension.git 12f80775 (Mon Apr 24 05:53:26 2023) unknown
 
DreamArtist-sd-webui-extension https://github.com/7eu7d7/DreamArtist-sd-webui-extension.git [12f80775 (Mon Apr 24 05:53:26 2023)](https://github.com/7eu7d7/DreamArtist-sd-webui-extension.git/commit/12f8077517b11199802f8d448d36ea573debae96) unknown a1111-sd-webui-tagcomplete https://github.com/DominikDoom/a1111-sd-webui-tagcomplete [a2e7b6b (Tue May 2 10:30:04 2023)](https://github.com/DominikDoom/a1111-sd-webui-tagcomplete/commit/a2e7b6bf6c8cbdff031b5b5929de150bf548c582) unknown multi-subject-render https://github.com/Extraltodeus/multi-subject-render.git [03427e26 (Mon Mar 6 14:11:30 2023)](https://github.com/Extraltodeus/multi-subject-render.git/commit/03427e26bebdc6da0ccfb749bf3c4e7e33d7458b) unknown openOutpaint-webUI-extension https://github.com/zero01101/openOutpaint-webUI-extension [5e84d6d5 (Mon Apr 10 23:01:41 2023)](https://github.com/zero01101/openOutpaint-webUI-extension/commit/5e84d6d5b1057f837eeecaa49a92a235dd589bc5) unknown sd-webui-ar https://github.com/alemelis/sd-webui-ar.git [9df49dc2 (Wed Apr 12 09:23:17 2023)](https://github.com/alemelis/sd-webui-ar.git/commit/9df49dc2d7da7333ac918fbce926c2370a3b8b53) unknown sd-webui-controlnet https://github.com/Mikubill/sd-webui-controlnet [a482867 (Tue May 2 23:13:18 2023)](https://github.com/Mikubill/sd-webui-controlnet/commit/a482867ee5e82b08b221c53662ff0c70c2f18d09) unknown sd-webui-infinite-image-browsing https://github.com/zanllp/sd-webui-infinite-image-browsing.git [6bc7f4ca (Tue May 2 19:52:50 2023)](https://github.com/zanllp/sd-webui-infinite-image-browsing.git/commit/6bc7f4ca1e10e932e34453fb744d1bd006640b09) unknown

Console logs

Traceback (most recent call last):
  File "C:\Users\jesus\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 215, in load_loras
    lora = load_lora(name, lora_on_disk.filename)
  File "C:\Users\jesus\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 176, in load_lora
    module.weight.copy_(weight)
RuntimeError: output with shape [32, 320, 1, 1] doesn't match the broadcast shape [32, 320, 3, 3]

Additional information

No response
@Muskelmagier Muskelmagier added the bug-report Report of a bug, yet to be confirmed label May 3, 2023
@justinh24
Copy link

+1

1 similar comment
@CRCODE22
Copy link

CRCODE22 commented May 3, 2023

+1

@Muskelmagier
Copy link
Author

with further probing i found out that only certain older LoRa's no longer work. Newer ones still work

@kornerson
Copy link

with further probing i found out that only certain older LoRa's no longer work. Newer ones still work

Wrong. I've been trying to make a Lora made by myself to load with no success. Another Lora made by me loads with no problem.

@CRCODE22
Copy link

CRCODE22 commented May 3, 2023

I made a Lora today, same way I did in the past. with the latest version of stable diffusion webui. It no longer works throwing these errors.
ding Lora E:\stable-diffusion-webui\models\Lora\Models2023V1.safetensors: RuntimeError | 0/4000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "E:\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 215, in load_loras
lora = load_lora(name, lora_on_disk.filename)
File "E:\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 176, in load_lora
module.weight.copy_(weight)
RuntimeError: output with shape [128, 320, 1, 1] doesn't match the broadcast shape [128, 320, 3, 3]

@thojmr
Copy link

thojmr commented May 3, 2023 •
edited
Loading

Might be due to some LoRAs using Conv2d 1x1 and others using Conv2d 3x3, but this repo only handles 1x1 from what I can see. Was it just skipping 3x3 layers in the past?

See Kohya LoRa for Conv2d 3x3 code
https://github.com/kohya-ss/sd-scripts/blob/ad5f318d066c52e5b27306b399bc87e41f2eef2b/networks/lora.py#L91

For now this change seems to fix the issue for txt2img at least:
lora.py:168

        # replace the last elif with this
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (1, 1):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (3, 3):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (3, 3), bias=False)

lora.py:237

        # insert this
         elif up.shape[2:] == (3, 3) or down.shape[2:] == (3, 3):
            updown = torch.nn.functional.conv2d(down.permute(1, 0, 2, 3), up).permute(1, 0, 2, 3)

I'm too lazy to open a PR for this but maybe someone with more motivation will see this.

@retronomi
Copy link

Confirmed this works, thanks!

@ClashSAN
Copy link
Collaborator

ClashSAN commented May 4, 2023

related #9979

This was referenced May 4, 2023
Closed
Closed
@ashllay
Copy link

ashllay commented May 4, 2023

Might be due to some LoRAs using Conv2d 1x1 and others using Conv2d 3x3, but this repo only handles 1x1 from what I can see. Was it just skipping 3x3 layers in the past?

See Kohya LoRa for Conv2d 3x3 code https://github.com/kohya-ss/sd-scripts/blob/ad5f318d066c52e5b27306b399bc87e41f2eef2b/networks/lora.py#L91

For now this change seems to fix the issue for txt2img at least: lora.py:168

        # replace the last elif with this
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (1, 1):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (3, 3):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (3, 3), bias=False)

lora.py:237

        # insert this
         elif up.shape[2:] == (3, 3) or down.shape[2:] == (3, 3):
            updown = torch.nn.functional.conv2d(down.permute(1, 0, 2, 3), up).permute(1, 0, 2, 3)

I'm too lazy to open a PR for this but maybe someone with more motivation will see this.

For me it not works i have installed a1111-sd-webui-locon extension and it did the trick.

@Mozoloa Mozoloa mentioned this issue May 4, 2023
Merged
@imago10
Copy link

imago10 commented May 4, 2023

Can someone post working lora.py with all that corrections? I tried to put it myself and I only get error an LORA wont loading :/

@SafakB
Copy link

SafakB commented May 8, 2023

Might be due to some LoRAs using Conv2d 1x1 and others using Conv2d 3x3, but this repo only handles 1x1 from what I can see. Was it just skipping 3x3 layers in the past?

See Kohya LoRa for Conv2d 3x3 code https://github.com/kohya-ss/sd-scripts/blob/ad5f318d066c52e5b27306b399bc87e41f2eef2b/networks/lora.py#L91

For now this change seems to fix the issue for txt2img at least: lora.py:168

        # replace the last elif with this
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (1, 1):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
        elif type(sd_module) == torch.nn.Conv2d and weight.shape[2:] == (3, 3):
            module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (3, 3), bias=False)

lora.py:237

        # insert this
         elif up.shape[2:] == (3, 3) or down.shape[2:] == (3, 3):
            updown = torch.nn.functional.conv2d(down.permute(1, 0, 2, 3), up).permute(1, 0, 2, 3)

I'm too lazy to open a PR for this but maybe someone with more motivation will see this.

Working for me, thanks

@brehiner
Copy link

I did the changes but doesn't work for me

@kornerson
Copy link

Doesn't work for me either.

getting this error on LORAs that were working beofre

module.weight.copy_(weight)
RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

@WS-HS
Copy link

WS-HS commented May 12, 2023

Work For Me!

lora.py:237

    # insert this
     elif up.shape[2:] == (3, 3) or down.shape[2:] == (3, 3):
        updown = torch.nn.functional.conv2d(down.permute(1, 0, 2, 3), up).permute(1, 0, 2, 3)

this not insert 237
you need insert before the " else "

My Fix File
lora Fix.zip

@jacquesfeng123
Copy link

Doesn't work for me either.

getting this error on LORAs that were working beofre

module.weight.copy_(weight) RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

i think we are running into a different issue, their broadcast error is different.

ours is [A,B] does not match with [A,B,A,B],

theirs is [A,B,1,1] does not match with [A,B,3,3]

I can confirm you can fix with https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris extension. but this is an ugly fix.

@Joeyd348039
Copy link

Doesn't work for me either.
getting this error on LORAs that were working beofre
module.weight.copy_(weight) RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

i think we are running into a different issue, their broadcast error is different.

ours is [A,B] does not match with [A,B,A,B],

theirs is [A,B,1,1] does not match with [A,B,3,3]

I can confirm you can fix with https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris extension. but this is an ugly fix.

I tried that and still no luck. I'm still getting an error:
Traceback (most recent call last):
File "C:\ai\stable-diffusion-21\extensions-builtin\Lora\lora.py", line 253, in load_loras
lora = load_lora(name, lora_on_disk)
File "C:\ai\stable-diffusion-21\extensions-builtin\Lora\lora.py", line 211, in load_lora
module.weight.copy_(weight)
RuntimeError: output with shape [128, 320] doesn't match the broadcast shape [128, 320, 128, 320]

@kornerson
Copy link

kornerson commented Jun 16, 2023 via email

@suay1113
Copy link

Doesn't work for me either.
getting this error on LORAs that were working beofre
module.weight.copy_(weight) RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

i think we are running into a different issue, their broadcast error is different.
ours is [A,B] does not match with [A,B,A,B],
theirs is [A,B,1,1] does not match with [A,B,3,3]
I can confirm you can fix with https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris extension. but this is an ugly fix.

I tried that and still no luck. I'm still getting an error: Traceback (most recent call last): File "C:\ai\stable-diffusion-21\extensions-builtin\Lora\lora.py", line 253, in load_loras lora = load_lora(name, lora_on_disk) File "C:\ai\stable-diffusion-21\extensions-builtin\Lora\lora.py", line 211, in load_lora module.weight.copy_(weight) RuntimeError: output with shape [128, 320] doesn't match the broadcast shape [128, 320, 128, 320]

Can anyone solve this problem?
[A,B] does not match with [A,B,A,B].
I tried updating the extension(in https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris.), but the issue still persists.

@coucoul38
Copy link

Doesn't work for me either.
getting this error on LORAs that were working beofre
module.weight.copy_(weight) RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

i think we are running into a different issue, their broadcast error is different.

ours is [A,B] does not match with [A,B,A,B],

theirs is [A,B,1,1] does not match with [A,B,3,3]

I can confirm you can fix with https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris extension. but this is an ugly fix.

How ?
I installed the extension but it still doesnt work

@kornerson
Copy link

kornerson commented Jul 12, 2023 via email

@TongfanWeitf
Copy link

facing the same problem. my version is 1.5.1 68f336b.
My lora works, but only for the first few. I need to restart my web ui again and again. There is no bug report, it just doesn't work

@catboxanon
Copy link
Collaborator

catboxanon commented Aug 11, 2023 •
edited
Loading

I can't reproduce this. There's mention that this only happens for LoRAs extracted from checkpoints while running Torch 2.0, but those are also working for me, and I'm running Torch 2.0. #10089 was also supposed to fix this in the first place and that's been implemented since 1.2.0.

Could someone here experiencing this provide a link to a LoRA that causes this error? Also if you have any informational at all about when this did work, if at all, so a git bisect could be done if needed.

This fork has a few similar issues that were made, but the issue ended up being 1) users not using the locon/lycoris extension (which is no longer needed, as these extra networks are implemented natively now), 2) LoRAs made for SD2.1 (which the webui previously didn't support I believe, but very much should now), or 3) extension incompatibilities (composable lora in particular -- but all extra extensions should be disabled to verify if a LoRA is working or not).

vladmandic/automatic#120
vladmandic/automatic#139
vladmandic/automatic#509

@catboxanon catboxanon added the cannot-reproduce I can't reproduce this, so I can't fix it. Add steps for reproduction and remove this tag. label Aug 11, 2023
@0AwsD0
Copy link

0AwsD0 commented Aug 11, 2023 •
edited
Loading

For me @thojmr fix worked when I encountered this problem with this two lora's:

https://mega.nz/file/ReBhBSLD#rN-ZtKgaRTi5brxxW2MrHHgcDRYu_fEfcUL9pvaAGT8
https://mega.nz/file/cWBhRChb#DGsmXpbsG_8K0JBgsZpee1gTnBkLfszv9uUBxAC1ZEM

They worked normally on additional networks tab.

However - after a recent LORA update I don't need those fixes anymore and Lora's that I provide work out of the box.

@catboxanon
Copy link
Collaborator

Yeah, those were accounted for months ago by #10089. I guess I'm moreso referring to the error mentioned here. #10036 (comment)

RuntimeError: output with shape [200, 320] doesn't match the broadcast shape [200, 320, 200, 320]

It's a different broadcast shape than the one the PR accounts for.

@TongfanWeitf
Copy link

facing the same problem. my version is 1.5.1 68f336b. My lora works, but only for the first few. I need to restart my web ui again and again. There is no bug report, it just doesn't work

I am now using lyCORIS output channel for all my loras, seems work

@catboxanon
Copy link
Collaborator

Okay, since the few extra comments here back up what I initially had mentioned in my comment, where any further issues are likely user error, I'm going to go ahead and close this. If this issue persists, open a new issue with additional details and a link to the LoRA model that does not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug-report Report of a bug, yet to be confirmed cannot-reproduce I can't reproduce this, so I can't fix it. Add steps for reproduction and remove this tag.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests