Loading multiple LoRA weights during inference

[asadm]

During inference, is it possible to load multiple LoRA weights on top of the frozen model. Something like:

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16)
pipe.unet.load_attn_procs("a.bin")
pipe.unet.load_attn_procs("b.bin")
pipe.to("cuda")

This seems to not work currently due to current API design I think but is there a workaround for doing this?
Thanks

[krahnikblis]

you can add them together (c = a + b), or average them (c = a/2 + b/2), before loading. i.e. for each layer, add the lora_ups from your a and b models, and the lora_downs. it just ends up looking kinda fried (like overtraining) if you add more than ~4 together at once. the methods that do work fofr >~4 are a bit more labor-intensive to code and time-consuming to run

[asadm]

Thanks for confirming that! I guess the better way would be to redo dreambooth using the multi-subject method instead.

[patrickvonplaten]

@asadm given that the lora weights are so small it should also be fine to change them out on the fly no? E.g.:

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16)
pipe.to("cuda")

def diffuse_with_a(prompt):
    pipe.unet.load_attn_procs("a.bin")
    pipe(prompt)

def diffuse_with_b(prompt):
    pipe.unet.load_attn_procs("b.bin")
    pipe(prompt)

diffuse_with_a("....")
diffuse_with_b("...")
...

[asadm]

That's correct. But my usecase is to use both concepts in a single prompt/inference.
Example: "an astronaut on a horse", where both astronaut and horse are from different LoRA weight files.

But so far it looks like that will not work?

[evancasey]

If you want to mix LoRA's you can just average the weights together and then do a single call to pipe.unet.load_attn_procs

For example:

lora1 = torch.load("<my first lora>")
lora2 = torch.load("<my second lora>")
lora_merged = {}
for (k1, v1 ), (k2, v2) in zip(lora1.items(), lora2.items()):
    lora_merged[k1] = (v1 + v2) / 2
pipe.unet.load_attn_procs(lora_merged)

[asadm]

Thank you for the code. I will try out these approaches and see if they work. Closing this for now.