[1.1.434] IP-Adapter faceid portrait

[huchenlei]

How to use

You can download the model here:
https://huggingface.co/h94/IP-Adapter-FaceID/blob/main/ip-adapter-faceid-portrait_sd15.bin
The model should be paired with ip-adapter_face_id preprocessor. No LoRA is required for this model.

Model comparisons

Following samples are generated by running

 python -m pytest -vv --junitxml=test/results.xml --cov .\extensions\sd-webui-controlnet\ --cov-report=xml --verify-base-url .\extensions\sd-webui-controlnet\tests\web_api\full_coverage\ipadapter_test.py

You need to set environment variable CONTROLNET_TEST_FULL_COVERAGE to any value in order for the test to be detected.
Some other preps are documented in https://github.com/Mikubill/sd-webui-controlnet/blob/main/tests/README.md

Checkpoint used: realisticVisionV20_v20.safetensors [e6415c4892]

Single input

Input image

FaceID

FaceID plus

FaceID plus v2

FaceID portrait

Multiple inputs

• Uses 6 images under tests/images/portrait
• After Real multi-inputs ControlNet unit #2539, you can use weight = 1.0 intead of 1/6.

FaceID

FaceID plus

FaceID plus v2

FaceID portrait

Conclusion

The new portrait model performs best when given multiple inputs.

Notes

• Realistic vision checkpoints performs better on western faces. I used it to test whether portrait model can correctly restore features of an asian face.

Update 2024-01-23

There was some bugs in our testing code previously. I have fixed those bugs. The model comparison samples are updated. Negative prompt has big impact on generation result according to our testing. You should avoid use a long list of negative prompt to obtain better id preservation.

[IndigoDosSantos]

I'm impressed with the new feature that allows us to view uploaded batch images directly in the controlnet. This improvement eliminates the risk of errors due to incorrect path syntax.
While I haven't tested it thoroughly, it seems like the portrait IP-adapter might be faster than others from the faceid family. Also, its ability to blend faces effectively and maintain consistency across various prompts and seeds is quite remarkable (have a look at the image below). It effectively acts like an 'instant LoRA' as @huchenlei pointed out in #2527 , and could possibly be used to generate image sets for actual LoRAs.

[yamfun]

Is it normal that the old IP adaptor like "ip-adapter_clip_sd15" / "ip-adapter_sd15" no longer work after the update?

and like either the Weight or XYZ plot "[ControlNet] Weight" are not making a difference...

[huchenlei]

Can you file an issue sharing your setup and results? There might be some issues regarding the ip-adapter implementation change.

[huchenlei]

See #2558.

[yamfun]

Updated the extension again upon seeing your reply, it works now. Thanks a lot.

[yaboosat]

@huchenlei
Unfortunately I'm not able to see ip-adapter_face_id in my preprocessor list. I believe it's because my controlnet is old version. But when I try to update it to latest version through automatic1111 extension nothing happens :(

ns.

[IndigoDosSantos]

I'd try a fresh ControlNet install. Just make sure you don't accidentally delete the models as well, as you probably don't want to download them again.

[sipie800]

great work! it just shows obvious improvement over face id. can we have sdxl version model?

[huchenlei]

https://huggingface.co/h94/IP-Adapter-FaceID/resolve/main/ip-adapter-faceid-portrait_sdxl.bin

[hungtooc]

I have 2 question:

1. In the original model (https://huggingface.co/h94/IP-Adapter-FaceID), only ip-adapter faceid portrait support multiple inputs. How your implement support multiple inputs for all version of ip-adapter faceid?

2. does this extension support concat two image of people into one image like this (image from comfyUI)
   

3. @huchenlei