IP-Adapter + Lora , lora_weight != 1.0 warning logs

[Honey-666]

diffusers/src/diffusers/models/attention_processor.py

Line 2238 in 7d63182

if scale != 1.0:

hi
When I use lora + ip-adapter, lora_weight ! = 1.0 will print a lot of warning logs, can anyone tell me why?

os:

diffusers==0.26.0.dev0

this is my code:

import torch
from PIL import Image
from diffusers import StableDiffusionPipeline
from transformers import CLIPVisionModelWithProjection

model_path = 'C:\\work\\pythonProject\\aidazuo\\models\\Stable-diffusion\\stable-diffusion-v1-5'
ip_adapter_path = 'C:\\work\\pythonProject\\aidazuo\\models\\IP-Adapter'
ip_img_path = 'C:\\work\\pythonProject\\aidazuo\\jupyter-script\\test-img\\vermeer.png'
lora_path = 'C:\\work\\pythonProject\\aidazuo\\models\\Lora\\figma.safetensors'

image_encoder = CLIPVisionModelWithProjection.from_pretrained(ip_adapter_path,
                                                              subfolder='models/image_encoder',
                                                              torch_dtype=torch.float16).to('cuda')

pipe = StableDiffusionPipeline.from_pretrained(
    model_path,
    safety_checker=None,
    variant="fp16",
    torch_dtype=torch.float16,
    image_encoder=image_encoder).to("cuda")

pipe.load_ip_adapter(ip_adapter_path, subfolder="models", weight_name="ip-adapter-plus_sd15.bin")
pipe.set_ip_adapter_scale(0.6)

pipe.load_lora_weights(lora_path)

ip_adapter_img = Image.open(ip_img_path)

images = pipe(
    prompt='a beautiful girl',
    ip_adapter_image=ip_adapter_img,
    negative_prompt="",
    num_inference_steps=30,
    num_images_per_prompt=1,
    width=512,
    height=512,
    cross_attention_kwargs={"scale": 0.5}
).images

pipe.unload_lora_weights()
pipe.unload_ip_adapter()

for img in images:
    print(img)

[spezialspezial]

Remove the cross_attention_kwargs argument and try a combination of
pipe.load_lora_weights(path, weight_name=path.name, adapter_name=lora_key) pipe.set_adapters(adapter_names=lora_keys, adapter_weights=lora_scales)

[Honey-666]

Remove the cross_attention_kwargs argument and try a combination of pipe.load_lora_weights(path, weight_name=path.name, adapter_name=lora_key) pipe.set_adapters(adapter_names=lora_keys, adapter_weights=lora_scales)

Ok, I tried that and he didn't print the warning log Thank You!

[patrickvonplaten]

cc @yiyixuxu for visibility

[yiyixuxu]

I think we can remove this line

diffusers/src/diffusers/models/attention_processor.py

Line 2238 in 7d63182

if scale != 1.0:

[spezialspezial]

Pretty related I think it would be nice to generally have some better infrastructure/mechanism for getting values to attention processors. For example to supply ip-adapter with the current time step/iteration step one has to modify several core files as the cross_attention_kwargs dict is expanded using ** and has to match ALL function signatures that receive it. Just passing a dict as reference where every processor may pick what it needs not bothering with other items would be more flexible imo.

[yiyixuxu]

@spezialspezial do you want to open an issue as "Feature Request / Discussion"?

[spezialspezial]

@yiyixuxu Done

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

we removed this warning so it should be fixed
this is also related #6915