[1.1.445] IPAdapter Advanced Weighting [Instant Style]

[huchenlei]

IPAdapter advanced weighting control was first implemented in https://github.com/cubiq/ComfyUI_IPAdapter_plus. Now we also have it supported in sd-webui-controlnet!

News

• [2024-04-29] Academia SD published an awesome tutorial video on ipadapter advanced weighting: https://www.youtube.com/watch?v=r8Kn0KCSKMo

SDXL composition [instant style]

Infotext:

1girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 40, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1058757959, Size: 1000x1000, Model hash: aeb7e9e689, Model: juggernautXL_v8Rundiffusion, Clip skip: 2, ControlNet 0: "Module: ip-adapter-auto, Model: ip-adapter_sdxl [d5d53548], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.9.0-15-gff6f4680

SDXL style transfer [instant style]

Infotext:

1girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 40, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1058757959, Size: 1000x1000, Model hash: aeb7e9e689, Model: juggernautXL_v8Rundiffusion, Clip skip: 2, ControlNet 0: "Module: ip-adapter-auto, Model: ip-adapter_sdxl [d5d53548], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.9.0-15-gff6f4680

Free edit on weights

You can also freely adjust any weight number by directly editing the weight array.

API usage

Just set advanced_weighting field in the API to control weight on each block with cross attention. You can refer to scripts/ipadapter/weight.py for weight presets used on the UI.

{
  "image": realistic_girl_face_img,
  "module": "ip-adapter-auto",
  "model": "ip-adapter_sd15",
  "advanced_weighting": [1, 1, 1, ..., 1],
}

Note

You are welcome to post your generations with different ipadapter weight settings in this discussion thread.

[CCpt5]

This is incredible! I've used the Comfy implementation which is great, but it seems to really help/work well in A1111. Thanks so much for your continued efforts!!

[IPv6]

Really cool, thanks!
(sorry to ask - can it be used in Forge too?)

[huchenlei]

Unfortunately lllyasviel did not give us collaborators merge permission in forge repo, so any work need to wait until he comes back.

[Gushousekai195]

Why don't I see that happening anytime soon???

[INTstinkt]

Hey there,
just wanted to ask if there is any kind of documentation about each different weight in the transformer index. For now i mostly found that Output block 6 is mostly for style and Input Block 3 mostly for Composition. I tried to understand what each Block does by 0 ing the block but other then those 2 mentioned Blocks the generation is just a tad different each time.

Cheers

[ema7569]

It's more a style management than a composition management. The constant is the blocks are a progression from in00 to out11 in the image construction. By applying math formula to the blocks weights you can manage precisely the degrees of similarity in one way or other. It's like the a1111 use to merge two checkpoint at 25%, 50%, 75% to chose the power of the merge, When you have access to these individual weights you can apply any formula you want not just a simple 25% or 50%, You can create a cosine, a reverse cosine or any formula you want. .. You can also use DAAM to help a bit to know what the layers the ia used to create a particular image and chose to use just these layers for the final image by example but these layers will work differently for a portrait, a full body or a landscape. The thing you can really use is to apply various formulas to the 25 layers to merge the styles. Also for more info lora has 25 layers. Lycoris has more layers than lora, ect.. Another important point these layer are just the surface of the full prompt, It seem also we have 25 layers per word in the prompt but currently I never see an extension that also give access to all these sub layers. I think having access to these sub layers per word give a more composition approach than just applying the formula to the full prompt that is more a style approach. With DAAM you can use a different prompt in each layer, it's interesting but do not currently give a full access to the sub layers. To resume, the layers for the full prompt is the style and the sub layers per word is the composition. We have access just to the style currently.

[ema7569]

Normally it's 25 blocks (12 in, 12 outs and one mid). Here you use less 25 ?

[huchenlei]

Here indexes are transformer index, not block index. Only a subset of blocks have cross attn (Transformer) in the UNet.

[huchenlei]

See

sd-webui-controlnet/scripts/enums.py

Lines 87 to 146 in 59d5998

	@property
	@lru_cache(maxsize=None)
	def transformer_ids(self) -> List[TransformerID]:
	"""id of blocks that have cross attention"""
	if self in (
	StableDiffusionVersion.SD1x,
	StableDiffusionVersion.SD2x,
	StableDiffusionVersion.UNKNOWN,
	):
	transformer_index = 0
	input_ids = []
	for block_id in [1, 2, 4, 5, 7, 8]:
	input_ids.append(
	TransformerID(UnetBlockType.INPUT, block_id, 0, transformer_index)
	)
	transformer_index += 1
	middle_id = TransformerID(UnetBlockType.MIDDLE, 0, 0, transformer_index)
	transformer_index += 1
	output_ids = []
	for block_id in [3, 4, 5, 6, 7, 8, 9, 10, 11]:
	input_ids.append(
	TransformerID(UnetBlockType.OUTPUT, block_id, 0, transformer_index)
	)
	transformer_index += 1
	return TransformerIDResult(input_ids, output_ids, [middle_id])
	else:
	# SDXL
	transformer_index = 0
	input_ids = []
	for block_id in [4, 5, 7, 8]:
	block_indices = (
	range(2) if block_id in [4, 5] else range(10)
	) # transformer_depth
	for index in block_indices:
	input_ids.append(
	TransformerID(
	UnetBlockType.INPUT, block_id, index, transformer_index
	)
	)
	transformer_index += 1
	middle_ids = [
	TransformerID(UnetBlockType.MIDDLE, 0, index, transformer_index)
	for index in range(10)
	]
	transformer_index += 1
	output_ids = []
	for block_id in range(6):
	block_indices = (
	range(2) if block_id in [3, 4, 5] else range(10)
	) # transformer_depth
	for index in block_indices:
	output_ids.append(
	TransformerID(
	UnetBlockType.OUTPUT, block_id, index, transformer_index
	)
	)
	transformer_index += 1
	return TransformerIDResult(input_ids, output_ids, middle_ids)

for ref

[huchenlei]

BTW, 25 blocks is only for SD15/SD21, SDXL only has 19 blocks (9 in, 9 out and 1 mid)

[hrz009]

I'm using Stable Diffusion 1.5 and after this update, ip adapter is not working. Can someone explain how to use it for SD1.5 and not SDXL. All the pics generated from ipadapter are just green blurry images

[TheSlayonetta]

i think it's specifically for SDXL... I'm going to revert back to an old repository until it gets sorted cause it has become unusable for me with this feature, so you're not alone

[huchenlei]

The advanced weighting is for both SD15 and SDXL. Please file an issue to reproduce your issue, thanks!

[hrz009]

How to file an issue, can you please explain.

[huchenlei]

https://github.com/Mikubill/sd-webui-controlnet/issues/new/choose

[AhBumm]

This feature seems can't work with Multi-Inputs. report:
ValueError: controlnet is enabled but no input image is given

[AhBumm]

*** Error running process: E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "E:\stable-diffusion-webui\modules\scripts.py", line 825, in process
        script.process(p, *script_args)
      File "E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1247, in process
        self.controlnet_hack(p)
      File "E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1232, in controlnet_hack
        self.controlnet_main_entry(p)
      File "E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 982, in controlnet_main_entry
        controls, hr_controls, additional_maps = get_control(
      File "E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 233, in get_control
        input_image, resize_mode = Script.choose_input_image(p, unit, idx)
      File "E:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 793, in choose_input_image
        raise ValueError("controlnet is enabled but no input image is given")
    ValueError: controlnet is enabled but no input image is given

[huchenlei]

I don't think this is caused by advanced weighting. According to my local testing, advanced weighting can be used with multi-inputs.

This issue can be caused by improper sync of gr.State with UI state. This can happen if you do not reload the page after restart A1111 server.

[OoAnikioO]

what weight choice do you recommend for intant-id?

[TheSlayonetta]

I don't recommend weight choice lol it doesn't work with 1.5 and I can't run sdxl so idk

[huchenlei]

According to my test, for instant id, majority of work is done by the controlnet unit, so the advanced weighting on the ipadapter unit does not change the result much.

[huchenlei]

I might consider just remove advanced weighting for instant id later as it is not so useful.

[fugueblus]

I hope you can remove it, I think it has a big impact on the weight control of instantid，THX

[lleksa]

I'm using automatic1111 (with SD1.5 checkpoint model) with ControlNet v1.1.445 and there's no ip-adapter_clip_sd15 preprocessor. Is this normal? Thank you!

[huchenlei]

ip-adapter_clip_sd15 is alias of ip-adapter_clip_h.

BTW, Now you can just use ip-adapter-auto to automatically select correct preprocessor for you.

[3blackbar]

plus vit-h works amazing with this ,better than any other from what i tested

[mykeehu]

I have noticed that if any of the prompts are 75 tokens longer, they lose the style of delivery. I tested with two models (babes and comicbabes), both of which lost style as soon as I went over 75 tokens.

55 token in negative prompt:

74 token in negative prompt:

76 token in negative prompt:

And 183 token in negative prompt:

[huchenlei]

IPA's decoupled Cross Attention training will be affected by the length of CLIP part
if their training is under standard CLIP only (77token
than the concat trick for CLIP will affect the attention significantly

Answer from A1111 dev.

[3blackbar]

multi inputs do not work , so if i have more than one image reference of the style then i can use just one

[huchenlei]

If you want to set advanced weight for each image, you need to put each image in individual unit.

[a3nima]

Any suggestions for settings with ipadapter face id v2 (sdxl) ? (normally I've been using weight 1), but with some source images the results are not good and i guess it could be better with other settings (lora 0.6 like suggested from dev).