[1.1.439] Instant ID

[huchenlei]

Instant ID project

https://github.com/InstantID/InstantID

Instant ID uses a combination of ControlNet and IP-Adapter to control the facial features in the diffusion process. One unique design for Instant ID is that it passes facial embedding from IP-Adapter projection as crossattn input to the ControlNet unet. Normally the crossattn input to the ControlNet unet is prompt's text embedding.

Download models

You need to download following models and put them under {A1111_root}/models/ControlNet directory. It is also required to rename models to ip-adapter_instant_id_sdxl and control_instant_id_sdxl so that they can be correctly recognized by the extension.

• ipadapter model
• ControlNet model

How to use

InstantID takes 2 models on the UI. You should always set the ipadapter model as first model, as the ControlNet model takes the output from the ipadapter model. (ipadapter model should be hooked first)

Unit 0 Setting

You must set ip-adapter unit right before the ControlNet unit. The projected face embedding output of IP-Adapter unit will be used as part of input to the next ControlNet unit.

Unit 1 Setting

The ControlNet unit accepts a keypoint map of 5 facial keypoints. You are not restricted to use the facial keypoints of the same person you used in Unit 0. Here I use a different person's facial keypoints.

CFG

It is recommended to set CFG 4~5 to get best result. Depending on sampling method and base model this number may vary, but generally you need to use CFG scale a little bit less than normal CFG.

Output

Follow-up work

• Make sd-webui-openpose-editor able to edit the facial keypoints in preprocessor result preview.
• Currently even if you are using the same face for both model, the insightface preprocessor will run twice. We need to find a way to cache the result and only run the model once.
• Support multiple face inputs.
• Support high res fix.

Note

As the insightface's github release currently do not have antelopev2 model, we are downloading from a huggingface mirror https://huggingface.co/DIAMONIK7777/antelopev2. If you are in mainland China and don't have good internet connection to huggingface, you can manually download the model from somewhere else and place them under extensions/sd-webui-controlnet/annotators/downloads/insightface/models/antelopev2.

Known issues

• If setting width x height to 1024 x 1024, there is higher chance to generate a watermark, logo. This cannot be corrected by adding logo watermark keywords in negative prompt. You can avoid this by setting other width x height values.

[ivanoff13]

only XL?

[Ldaluz2]

Any plans to make it for 1.5?

[Jiudy]

Please tell me where to place the antelopev2 model in the Forge file path.

[rzw520]

E:\webui_forge\webui\models\insightface\models\antelopev2

[Jiudy]

Your help is godsent, you're the kindest soul around, thank you so much, blessings will come your way for sure.

[ramnagapuri]

where i put this path

[2blackbar]

It does work but yeah, it loads the models over and over and over which takes like over minute of waiting time on 3090, so each image takes almost 2 minutes to generate cause of loading times, even if you wont change any reference images.
Imo results are sometimes spectacular considering 0 training.
Id do a check if any ref images were changed and if they werent- dont reload anything and just generate again, imo this behaviou should be default for all cnet models so theres no constant reloading when using future models as well.
Id also do another check - if theres no separate instance for the pose (lipcorners dots and iris dots ), just use preprocessor image for the pose.
Also your reference image should not be so close that jaw touches the bottom of the image - this is too close and wont be recognized, you should have some neck showing.
-for some reason in lot of wider pics i have hand obscuring the low part of the face , its pretty strange, happens when shoulders are already visible in framing .

[stasha17]

The results I'm getting are very similar to the above. For some reason, they look quite "washed out" of color. Not sure why.

[2blackbar]

you mean the contrast? i do painting by vallejo and frazetta prompt , maybe try to up the cfg scale to get more contrast, i personally dont like it and i use levels later to bring contrast back.All those are CFG 3 and 20steps

[hansolocambo]

I also have an 3090RTX (i9-12900K, 32GB RAM) and I generate (with Instant_ID) a 1024x1024 image in 21sec, and with highres fix x2, it takes 1m15s. It's long, but far from 2 minutes.

[huzheng0419]

Why do all the images I generate that have an instant ID appear a bit yellowish? It's normal for images that don't have an instant ID. This is a comparison image, and all the parameters have not changed. Can someone answer this question

[hben35096]

Works great, thank you for your efforts.

Things to watch out for at the moment:
1.CFG needs to be lower than usual, depending on the SD model
2. Avoid using the size of 1024*1024
3. The step count needs to be lower than usual

[hben35096]

Haha, no, InstantID only detects face keypoints, and the body pose is random.
The description of the resolution is just whether there will be any abnormalities outside of the test 1024*1024.

[2blackbar]

well, my results has something obscuring the face (fingers or warrdobe element or hair ) way way too often compared to not using instantID, especially near lips/jaw which is superstrange and i dont think its a coincidence

[f0ster]

Thanks for sharing @hben35096 . may have meant up to 1024^2, these settings look good for me at (1024,1024)

🍻

[minhquang1189]

Could you please guide me on how to fix this error? Thank you.

[jamesstothard]

I'm gettting very strange behaviour if I set the size below 10241024 then controlnet doesn't even work!?

This one is at 10241024

[FurkanGozukara]

awesome this time i didnt make gradio and now implemented :D

if face input not changed you really should cache it

i did this for IP Adapter Face ID and it did speed up huge. caching the calculated face input vectors

https://youtu.be/rjXsJ24kQQg

[huchenlei]

We do have preprocessor result cache here:

sd-webui-controlnet/scripts/global_state.py

Lines 22 to 49 in a67d040

	def cache_preprocessors(preprocessor_modules: Dict[str, Callable]) -> Dict[str, Callable]:
	""" We want to share the preprocessor results in a single big cache, instead of a small
	cache for each preprocessor function. """
	CACHE_SIZE = getattr(shared.cmd_opts, "controlnet_preprocessor_cache_size", 0)
	# Set CACHE_SIZE = 0 will completely remove the caching layer. This can be
	# helpful when debugging preprocessor code.
	if CACHE_SIZE == 0:
	return preprocessor_modules
	logger.debug(f'Create LRU cache (max_size={CACHE_SIZE}) for preprocessor results.')
	@ndarray_lru_cache(max_size=CACHE_SIZE)
	def unified_preprocessor(preprocessor_name: str, *args, **kwargs):
	logger.debug(f'Calling preprocessor {preprocessor_name} outside of cache.')
	return preprocessor_modules[preprocessor_name](*args, **kwargs)
	# TODO: Introduce a seed parameter for shuffle preprocessor?
	uncacheable_preprocessors = ['shuffle']
	return {
	k: (
	v if k in uncacheable_preprocessors
	else functools.partial(unified_preprocessor, k)
	)
	for k, v
	in preprocessor_modules.items()
	}

. If you don't change preprocessor input, the cached result will be used. If you add --controlnet-loglevel DEBUG, you will see whether the cached preprocessor result is used.

[2blackbar]

so if models are cached, why it takes so long and models are reloaded each time it generates ? Nothing should be reloaded if nothing was changed.

[huchenlei]

If you are talking about load of ControlNet/IP-Adapter models, you should set model cache size to a value > 1. (By default that config is 1. As we are using 2 models together, the cache size of 1 means we miss the cache every time.)

[2blackbar]

DAMN! This is it, can you make it default controlnet setting ? Now i can generate non stop with like 8 seconds per image, its great. i dont think people know this and it changes A LOT from 2 minutes to 8 seconds.
And whats best - you can change the photo of the person and its still 8 seconds per image on 3090

[FurkanGozukara]

this info super important thanks

[zachysaur]

---

100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:43<00:00, 2.19s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:42<00:00, 2.14s/it]
2024-01-28 21:23:52,651 - ControlNet - INFO - unit_separate = False, style_align = False 20/20 [00:42<00:00, 1.97s/it]
*** Error running process: D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "D:\Webui1111\Webui\stable-diffusion-portable-main\modules\scripts.py", line 718, in process
script.process(p, *script_args)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1138, in process
self.controlnet_hack(p)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1123, in controlnet_hack
self.controlnet_main_entry(p)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 868, in controlnet_main_entry
Script.check_sd_version_compatible(unit)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 793, in check_sd_version_compatible
raise Exception(f"ControlNet model {unit.model}({cnet_sd_version}) is not compatible with sd model({sd_version})")
Exception: ControlNet model ip-adapter_instant_id_sdxl eb2d3ec0 is not compatible with sd model(StableDiffusionVersion.SD1x)

[huchenlei]

Please use SDXL base model. Currently there is no instant id for sd15.

[matrix4767]

Are there plans for sd15 support?

[huchenlei]

You can file a request in instant id repo here: https://github.com/InstantID/InstantID. I don't have the resource to train model.

[zachysaur]

*** Error running process: D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "D:\Webui1111\Webui\stable-diffusion-portable-main\modules\scripts.py", line 718, in process
script.process(p, *script_args)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1138, in process
self.controlnet_hack(p)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1123, in controlnet_hack
self.controlnet_main_entry(p)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 868, in controlnet_main_entry
Script.check_sd_version_compatible(unit)
File "D:\Webui1111\Webui\stable-diffusion-portable-main\extensions\sd-webui-controlnet\scripts\controlnet.py", line 793, in check_sd_version_compatible
raise Exception(f"ControlNet model {unit.model}({cnet_sd_version}) is not compatible with sd model({sd_version})")
Exception: ControlNet model control_instant_id_sdxl c5c25a50 is not compatible with sd model(StableDiffusionVersion.SD1x)

---

[ghzgod]

I get an assertion error using all the same settings as above with an SDXL model of course.

AttributeError: 'UNetModel' object has no attribute 'cond_mark'

01/28/2024
05:48:12 PM
*** Error running process: /config/02-sd-webui/webui/extensions/sd-webui-controlnet/scripts/controlnet.py
01/28/2024
05:48:12 PM
    Traceback (most recent call last):
01/28/2024
05:48:12 PM
      File "/config/02-sd-webui/webui/modules/scripts.py", line 718, in process
01/28/2024
05:48:12 PM
        script.process(p, *script_args)
01/28/2024
05:48:12 PM
      File "/config/02-sd-webui/webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1138, in process
01/28/2024
05:48:12 PM
        self.controlnet_hack(p)
01/28/2024
05:48:12 PM
      File "/config/02-sd-webui/webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1123, in controlnet_hack
01/28/2024
05:48:12 PM
        self.controlnet_main_entry(p)
01/28/2024
05:48:12 PM
      File "/config/02-sd-webui/webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1103, in controlnet_main_entry
01/28/2024
05:48:12 PM
        assert isinstance(param.hint_cond, torch.Tensor)
01/28/2024
05:48:12 PM
    AssertionError
01/28/2024
05:48:12 PM
01/28/2024
05:48:12 PM
---
01/28/2024
05:48:18 PM

  0%|          | 0/20 [00:00<?, ?it/s]
  0%|          | 0/20 [00:01<?, ?it/s]
01/28/2024
05:48:18 PM
*** Error completing request

[k8kiss]

Where you able to fix this? I've got the same error

[wicherek1]

The same error. Did you find the solution for that?

[vashtsp]

Try another XL model

[navytut2]

Working fine. Thank you.
One doubt: If I don't want to provide a 2nd image as a pose and instead want it to give a random pose image with the first face in Unit 0, how to go about it?
InstandID's replicate demo has this option with single face input.

[huchenlei]

Can you link to the implementation code to InstandID's replicate demo?

[Lalimec]

https://replicate.com/zsxkib/instant-id though im not sure i understand what he means. Uploading the same image for the control would do the trick already.

[huchenlei]

I don't think the demo site supports generating random head position when you don't specify a pose.

[a3nima]

Currently it seems the second image is defining not only the position but also the size of the head in the image. It would be good, if the very well working face reproduction could be used freely like in face-id or an edit function could be used :)

[Angtrim]

Does it work also in img2img? More or less like "roop" ?

[trygvebw]

Thanks for your work! However, after playing around with this for a bit, something seems off compared to using InstantId through the new Diffusers pipeline. Let me demonstrate with an example (note, however, that the same happens with every image combination I've tried). I'm using a picture of (Daniel Radcliffe as) Harry Potter in ControlUnit 0 (ip-adapter) and a picture of Obama in ControlUnit 1. Here's the result with the "Control Weight" of Unit 0 set equal to 1.0:

Obviously, the quality is rather poor. The CFG scale is set to 4.0 and I use the euler-a sampler. Now, if I turn the "Control Weight" of Unit 0 down to 0.0, I get this:

This looks just slightly less like Radcliffe's Potter than the first image, but the quality is obviously vastly better. If I try values of Unit 0's "Control Weight" between 0.0 and 1.0, the quality is always degraded (although not as much as at 1.0) – even at low values like 0.2, the output image is noticeably blurry. If I keep the "Control Weight" at 1.0 but decrease the CFG scale to 3.0 or 2.0, the quality improves somewhat but the blurriness is still there (and the contrast gets worse):

Across several different input image combinations I've found that I get the best compromise between quality and identity fidelity when ControlUnit 0's "Control Weight" is 0.0 or a very low value like 0.05 or 0.1, and the CFG scale is as high as I can push it without getting too much contrast (usually around 4.0). That's somewhat odd, and I don't experience the same when using InstantId directly – the best combination of weights is then something like 0.8 and 0.8.

Any thoughts?

[Lalimec]

20-30 seconds on a g5 ec2 instance and a rtx4090 rig. both with 24gb vram. Generation times are quite a bit high with instantid. Only problem was forgetting to put the models in the proper folder, other than that there are 2 different comfy node set. ZHO was a little more flexible but both are far more restricted than the a1111 versions since they are just diffusers wrappers.

[dasoncheng]

Excuse me, what is the cause of this clarity problem in the webui and is there a solution

[rubensicko]

Hi! Can you show me the prompt (or model) for this amazing retro pink style? Many thanks!!

[a3nima]

Prompt:
vaporwave style portrait of a woman, scenery in the background . retro aesthetic, cyberpunk, vibrant, neon colors, vintage 80s and 90s style, highly detailed,
Negative prompt: monochrome, muted colors, realism, rustic, minimalist, dark,
Steps: 15, Sampler: DPM++ 2M Karras, CFG scale: 5, Seed: 2, Size: 960x1280, Model hash: dda8c0514c, Model: sdxlUnstableDiffusers_v11Rundiffusion, ControlNet 0: "Module: instant_id_face_embedding, Model: ip-adapter_instant_id_sdxl [eb2d3ec0], Weight: 0.6, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: True, Control Mode: ControlNet is more important, Hr Option: Both, Save Detected Map: True", ControlNet 1: "Module: instant_id_face_keypoints, Model: control_instant_id_sdxl [c5c25a50], Weight: 0.6, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: True, Control Mode: ControlNet is more important, Hr Option: Both, Save Detected Map: True", Version: v1.7.0

Image Width: 960 px
Image Height: 1280 px

[Lalimec]

Man its been a while lol. But images should contain metada, try to drop them on comfy or a1111.

[pxEkin]

How to deal with multiple faces in one picture? i tested it on the webui, and instantid only handled one face

[pwcarney]

Seems to download the onnx files and then complains about the onnxruntime. I tried deleting and forcing it to reinstall the onnx files and the onnxruntime from the venv, but no luck. Thoughts?

Downloading: "https://huggingface.co/DIAMONIK7777/antelopev2/resolve/main/1k3d68.onnx" to D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\1k3d68.onnx

100%|███████████████████████████████████████████████████████████████████████████████| 137M/137M [00:10<00:00, 14.0MB/s]
Downloading: "https://huggingface.co/DIAMONIK7777/antelopev2/resolve/main/2d106det.onnx" to D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\2d106det.onnx

100%|█████████████████████████████████████████████████████████████████████████████| 4.80M/4.80M [00:00<00:00, 12.8MB/s]
Downloading: "https://huggingface.co/DIAMONIK7777/antelopev2/resolve/main/genderage.onnx" to D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\genderage.onnx

100%|█████████████████████████████████████████████████████████████████████████████| 1.26M/1.26M [00:00<00:00, 8.26MB/s]
Downloading: "https://huggingface.co/DIAMONIK7777/antelopev2/resolve/main/glintr100.onnx" to D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\glintr100.onnx

100%|███████████████████████████████████████████████████████████████████████████████| 249M/249M [00:19<00:00, 13.6MB/s]
Downloading: "https://huggingface.co/DIAMONIK7777/antelopev2/resolve/main/scrfd_10g_bnkps.onnx" to D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\scrfd_10g_bnkps.onnx

100%|█████████████████████████████████████████████████████████████████████████████| 16.1M/16.1M [00:01<00:00, 13.1MB/s]
2024-01-29 14:53:13.3130372 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\stable-diffusion-webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
2024-01-29 14:53:15.2392103 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\stable-diffusion-webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
2024-01-29 14:53:15.3354037 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\stable-diffusion-webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
2024-01-29 14:53:15.5136602 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\stable-diffusion-webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
2024-01-29 14:53:18.4024247 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1193 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\stable-diffusion-webui\venv\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
*** Error running process: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "D:\stable-diffusion-webui\modules\scripts.py", line 718, in process
script.process(p, script_args)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1143, in process
self.controlnet_hack(p)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1128, in controlnet_hack
self.controlnet_main_entry(p)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 969, in controlnet_main_entry
controls, hr_controls = list(zip([preprocess_input_image(img) for img in input_images]))
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 969, in
controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in input_images]))
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 936, in preprocess_input_image
detected_map, is_image = self.preprocessor[unit.module](
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 80, in decorated_func
return cached_func(*args, **kwargs)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 64, in cached_func
return func(*args, **kwargs)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\global_state.py", line 37, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\processor.py", line 806, in run_model_instant_id
raise Exception(f"Insightface: No face found in image.")
Exception: Insightface: No face found in image.

[FurkanGozukara]

it couldnt detect face in your input image

[2blackbar]

your face must have neck visible, otherwise it wont be detected

[scarbain]

Great feature, thanks a lot !!
When trying out multiple images as input of unit 0 and get a "No face found in image", it would be absolutely awesome to get the filename of the corresponding image ! Same if we use different image between unit 0 and unit 1, it'd be much easier to know which image should be removed.
I have not made lots of tests but I found that when no face is detected because it's too close, adding some black padding around the image helps the detector.

[saibai]

Hello, I have this error in Automatic1111 for Mac M2

*** Error running process: /Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/modules/scripts.py", line 718, in process
script.process(p, script_args)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1143, in process
self.controlnet_hack(p)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1128, in controlnet_hack
self.controlnet_main_entry(p)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py", line 969, in controlnet_main_entry
controls, hr_controls = list(zip([preprocess_input_image(img) for img in input_images]))
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py", line 969, in
controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in input_images]))
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/controlnet.py", line 936, in preprocess_input_image
detected_map, is_image = self.preprocessor[unit.module](
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/utils.py", line 80, in decorated_func
return cached_func(*args, **kwargs)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/utils.py", line 64, in cached_func
return func(*args, **kwargs)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/global_state.py", line 37, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/processor.py", line 801, in run_model_instant_id
self.load_model()
File "/Users/cinesaibai/pinokio/api/sd-webui.pinokio.git/automatic1111/extensions/sd-webui-controlnet/scripts/processor.py", line 739, in load_model
from insightface.app import FaceAnalysis
ModuleNotFoundError: No module named 'insightface'

[b4sh]

ModuleNotFoundError: No module named 'insightface'

[saibai]

I understood, but I have installed everything and I find automatic1111/extensions/sd-webui-controlnet/annotator/downloads/insightface/models with all the files

[b4sh]

Please read sd-webui-reactor README about install insightface on Windows - https://github.com/Gourieff/sd-webui-reactor

[saibai]

#2589 (reply in thread)

[2blackbar]

It works absolute best if your input face have half shadow on it so the volume of the face is read better by ipadapter , with flat light stallone was unrecognizable

[jamsdrak]

Amazing results, just how did you got that output?? checkpoint, did you add lora or just prompt, or was because of the control_instantID

[2blackbar]

I think this was nightvision xl but now i use helloworld sdxl from civit, xl is getting better but more slowly than 1.5, no refiner

movie still of man wearing tank top, covered in blood standing on street full of intestines meat blood ,exploding cars ,backlight ,haze
Negative prompt: blurry,artifacts
Steps: 12, Sampler: DPM++ 2M Karras, CFG scale: 3, Seed: 1278390959, Size: 1280x720, Model hash: 9aa0c3e54d, Model: nightvisionXL 7, VAE hash: 235745af8d, VAE: sdxl.vae.safetensors, ControlNet 0: "Module: instant_id_face_embedding, Model: ip-adapter_instant_id_sdxl [eb2d3ec0], Weight: 1, Resize Mode: Resize and Fill, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: True, Control Mode: ControlNet is more important, Hr Option: Both, Save Detected Map: True", ControlNet 1: "Module: instant_id_face_keypoints, Model: control_instant_id_sdxl [c5c25a50], Weight: 1, Resize Mode: Resize and Fill, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: True, Control Mode: ControlNet is more important, Hr Option: Both, Save Detected Map: True", Version: v1.7.0

[DavideAlidosi]

I have this error, but the image generation looks like ok.
ERROR - Unable to infer control type from module instant_id_face_embedding and model ip-adapter_instant_id_sdxl [eb2d3ec0]
Any idea?
Thanks

[huchenlei]

This is an UI error for preset management. It should not block you do generations.

[DavideAlidosi]

Thanks for quick reply.

[huzheng0419]

Why do all the images I generate that have an instant ID appear a bit yellowish? It's normal for images that don't have an instant ID. This is a comparison image, and all the parameters have not changed. Can someone answer this question

[huchenlei]

Might be related to #2735. You can try latest version and see if the issue persists.

[huzheng0419]

This is my SD-WEBUI version, which is the latest version updated today. I used 1.8.0 yesterday and encountered the problem described above. Contronet is also the latest version. I don't know where the problem is, but I am really anxious. I hope this problem can be resolved

[huchenlei]

Do you have the same color issue when using ComfyUI's impl or diffusers impl?

[huzheng0419]

Sorry, I haven't used FOMFUI yet, I only use WebUI https://github.com/huchenlei/sd-webui-controlnet/tree/fix_insight?tab=readme I downloaded your new scripts folder from the website and replaced it. Contranet and webui are both the latest versions, but there is still a color issue with the images. Below are my settings and the generated images. I'm quite curious about how this reason came about and I really want to fix it，

[huzheng0419]

[HonZuna]

Guys any idea where is problem ? Thank you so much.

DEBUG version:

[huchenlei]

You need to select a model when using ip-adapter-auto meta preprocessor. For instant id, please stick to instant_id_face_embedding preprocessor.

[HonZuna]

That's what I'm doing :(

UNIT1:

UNIT2:

[ashleykleynhans]

Are you using an SDXL model?

[huchenlei]

It's the unit2 causing issue. Fixed in #2751

[huzheng0419]

我看作者您是解决了输出图片颜色有问题的bug，请原谅我是小白，请问如何重新更新配置文件？

[huzheng0419]

I uninstalled the CN on my computer and removed it from the https://github.com/huchenlei/sd-webui-controlnet/tree/fix_insight?tab=readme -OV file has re downloaded the CN update, and also re downloaded two processors and models, but the problem still exists. May I ask if the steps are incorrect.

[pavnilschanda]

I tried this out but the results look wonky. Without the ControlNet (which has been updated), it (SDXL with its accompanying VAE) can generate images fine. Let me know what I should do.

watercolor painting
Steps: 22, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 5, Seed: 2532559336, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, ControlNet 0: "Module: instant_id_face_embedding, Model: ip-adapter_instant_id_sdxl [eb2d3ec0], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", ControlNet 1: "Module: instant_id_face_keypoints, Model: control_instant_id_sdxl [c5c25a50], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.9.0

I also got these if they may affect the result:

FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 

A tensor with all NaNs was produced in VAE.
Web UI will now convert VAE into 32-bit float and retry.
To disable this behavior, disable the 'Automatically revert VAE to 32-bit floats' setting.
To always start with 32-bit VAE, use --no-half-vae commandline flag.

[booraddley]

It ran pretty well for me. I used one image

and I generated this image:

I'm on a 16-inch 2021 MacBook Pro , Apple M1 Max, 64Gigs of Ram.
I used:

Model: FullyREALXL_v80Refocus 
Sampler: DPM++ 3M SDE Exponential
Size: 1024x1024
Steps: 40
Clip Skip: 2
CFG: 4

Looks better than anything I've done with ReActor.
Hope this helps move the project along!! 😉

[booraddley]

Now that I've worked with this a bit more, using all the same stuff. I think your going to have a heck of a time getting a more posable head and neck with only those 5 facial keypoints. You'll also need a way measure and alter lighting/light source, which is it's own area of expertise when it comes to cgi work in films and animated films.

For fun, I layered the: facial keypoints, over an opacity altered version of my AI generated image, over the original image I got from a random google search. Here's the result:

The light source, the chin, the eyes, the mouth, the neck, all appear in the same spot in each of 13 different images I've generated. Each image had a prompt to try and alter something to change the look of the "pose" of the chin in relation to the neck, as well as the light source. I tried high angle I tried low angle, no change. I suppose that may be the idea for now.

Anyway, just wanted to share my thoughts.

[fbauer-kunbus]

Is there still no way to edit the keypoints or even create keypoints from scratch without using a reference image? The is an 'edit' button which opens Photopea - but I couldn't figure out what this is supposed to do, since Photopea doesn't seem to treat the keypoints as editable vectors - and even if it did - how would I save the edit so that it is then used by A1111 again? Am I missing a major thing here?

[booraddley]

They said it in the main post:
Make sd-webui-openpose-editor able to edit the facial keypoints in preprocessor result preview.
It's on their todo list.

[fbauer-kunbus]

Yes, I read this - but it is a bit confusing that there is already an 'edit' button which in fact does nothing useful - without even a hint or a tooltip or - gosh - some kind of documentation. But what do you expect from people who think that the 'Gradio' GUI is 'fun' :-) The whole Open Source AI developer community got the focus completely wrong on 'features' and 'extensions' that are a dime a dozen - instead of taking the most obvious usability aspects into account.

[booraddley]

Oh, that's just the controllnet mask. The only thing you can edit about that is by drawing on it like a canvas I think, but that would ruin it. We're pretty lucky to have Gradio & A1111 & StableForgeUI & ComfyUI & StableSwarmUI. Otherwise, we'd have to do this all through CLI or CMD.

[fbauer-kunbus]

Guess it is debatable whether a bad GUI is better than a bunch of intelligent scripts - with good documentation and adequate examples. Being a software developer for decades myself I would very much prefer using image generation AI the way a 'normal' script with function calls works (heck - there is only a very minor 'visualization' part in the AI GUIs - the complete rest is a mere parameter and settings orgy and nested function calls, isn't it?). This would be WAY more transparent than what the GUIs now do - and don't get me started about the chaotic spaghetti nonsense of (Un-)ComfyUI. Look at how you normally build even complex web apps with (script) frameworks - include jQuery, and the stage is already set for most operations. It tells a lot that A1111 to the date isn't even capable to save and reload a project file which includes ALL settings and parameters and referenced file pathnames to recreate the very same image you created before. What are those people thinking who develop this? I have never in decades come across any reasonable GUI which couldn't do this. Or do you know of any system where you have to keep manual annotations on top of a project file to simply continue your work where you left off last week?
No, to my feel and experience the open source AI scene mainly suffers from still being to theoretical and 'academic' at its base and core. You totally feel, that most oft the main developers don't have usability and every day practical aspects in focus. Take for comparison a project like 'Obsidian.md' (a highly complex note-taking system) - the GUI is flexible and intelligently thought out by pros, which you can feel from the very start. But what was I expecting when 'Python' is the language of choice for AI projects ... a language who tried to invent the wheel again for the umpteenth time - and didn't even do it in a good way (sigh) ;-)

[chace20]

I found ip-adapter_instant_id_sdxl and control_instant_id_sdxl model need too much CPU to run, nearly 40 core, or inference speed will slow down. Does anyone have a solution?

[throbvr]

Has anyone been able to get this working with Oobabooga text generation web ui via the API? Seems there would be some amazing possibilities...

[denisov7078]

After the 1.1.447 update, Instant ID stopped working, it gives an error

[MINGSONG4]

I am facing same issue as well! Need to wait fix

[huchenlei]

New update should fix this. It is really wield why the full qualified name reaches here. Can I confirm the gradio version you folks used?

[denisov7078]

Thanks, new update fixed the problem. gradio version 3.41.2

[3blackbar]

this happened to me when i tried pulid and then i wanted to try instandID, im sure other ppl did the same scenario,something is not unloaded

[huchenlei]

There is a quick fix here: #2849.

A more proper fix & large refactoring on the way: #2847

[3blackbar]

I think somethin got borked recently in latest comits, it does not work in img2img inpaint only masked and insightface does not recognize faces in 99% cases, i think the cropping thing for masked area is broken now and when i mask the face, it does not send that masked face to instantid to detect face points

100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:03<00:00, 3.30it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 12/12 [00:03<00:00, 3.06it/s]
2024-05-07 05:55:28,479 - ControlNet - INFO - unit_separate = False, style_align = False 12/12 [00:03<00:00, 3.32it/s]
2024-05-07 05:55:28,480 - ControlNet - INFO - Loading model from cache: ip-adapter_instant_id_sdxl [eb2d3ec0]
2024-05-07 05:55:28,624 - ControlNet - INFO - Using preprocessor: instant_id_face_embedding
2024-05-07 05:55:28,624 - ControlNet - INFO - preprocessor resolution = 512
*** Error running process: F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "F:\stable-diffusion-webui\modules\scripts.py", line 784, in process
script.process(p, script_args)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1215, in process
self.controlnet_hack(p)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1200, in controlnet_hack
self.controlnet_main_entry(p)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 934, in controlnet_main_entry
controls, hr_controls, additional_maps = get_control(
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 290, in get_control
controls, hr_controls = list(zip([preprocess_input_image(img) for img in optional_tqdm(input_images)]))
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 290, in
controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in optional_tqdm(input_images)]))
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 242, in preprocess_input_image
result = preprocessor.cached_call(
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\supported_preprocessor.py", line 192, in cached_call
result = self._cached_call(input_image, *args, **kwargs)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 82, in decorated_func
return cached_func(*args, **kwargs)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 66, in cached_func
return func(*args, **kwargs)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\supported_preprocessor.py", line 205, in _cached_call
return self(*args, **kwargs)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\preprocessor\legacy\legacy_preprocessors.py", line 105, in call
result, is_image = self.call_function(
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\preprocessor\legacy\processor.py", line 745, in run_model_instant_id
face_info = InsightFaceModel.pick_largest_face(faces)
File "F:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\preprocessor\legacy\processor.py", line 652, in pick_largest_face
raise Exception("Insightface: No face found in image.")
Exception: Insightface: No face found in image.

[3blackbar]

ok i had cropping turned on for ipadapter part , now its working fine, its turning on automatically when in inpaint mode

[BenDes21]

Hi there! Any tips for reduce the blurriness / improve the quality of the image ? Im using :

• Epic realism SDXL
• CFG 3
• Steps 30
• Res 1016x1016
• Nothing has been modified on the controlnet side except " controlnet is more important " on both units

Only stuff I imagine can improve the quality is pixel perfect on both unit but my laptop is not powerful... worth to use it if I rent a gpu cloud ?

[ayush1268]

hey can we use this instant id in stable diffusion forge

[NewbieFish]

为什么我生成的效果这么差