Face Models

[xiaohu2015]

discussion on face adapter

[cubiq]

yes I think it's a good idea to move the discussion about face models here. Do you mean faceid or something else?

[xiaohu2015]

yes, faceid and face plus adapter

[cubiq]

I did some research on the layers for SD1.5 and face models. It seems output_3 and output_4 are the most active for the face but FaceID and full/plus face react to weights quite differently. For some reason FaceID reacts very well increasing the weight of input_7, but it could be just because the composition gets closer to the reference. Plus/Full face don't have the same behavior but output_11 seems to have some positive impact.

I wasn't able to find a clear best configuration, but by tinkering a little I was able to increase the COS similarity by 0.001-0.002. The good news is that I can now constantly get an EUC<0.27 COS<0.015 with FACEIDv2+FULLFACE (dlib).

Full face generally seems to perform better than Plus face.

The Ideal configuration is as follow:
FaceIDv2: weight 0.8, v2 weight: 2.0 + Lora weight 0.62 + Full Face weight: 0.5

In addition to that for FaceIDv2 I'm increasing the following layers
input_7: 1.12
output_3: 0.96
output_4: 0.96

For Full Face:
output_3: 0.55
output_11: 0.6

[xiaohu2015]

maybe We need a better full face model

[xiaohu2015]

I add a unnorm version (don't do l2_norm for the ID embedding) https://huggingface.co/h94/IP-Adapter-FaceID/blob/main/ip-adapter-faceid-portrait_sdxl_unnorm.bin, it will be a little different2 with norm version.

[cubiq]

I got it, it is working just fine 😄 give me one sec and I'll push the update

[xiaohu2015]

I would like to share my findings:
unnormed ID embedding is easier to learn than normed ID embedding, moreover, it also more stable (less likely to be affected by text prompts). However, multi-face condition may not have an enhancing effect

[cubiq]

I get very weird results actually, what does the PerceiverResampler do? or maybe I'm doing something wrong with the embeddings

[xiaohu2015]

I updated the model in the huggingface, and you can refer to the above code (in fact, it is same as face-plus, but use unnormed id embedding)

[cubiq]

yeah I was already converting the keys, it's just that it doesn't follow any of the previous patterns so I have to rewrite a bit of the logic (and the embeds don't need to be normed)

[cubiq]

I'm not sure I understand the new unnormed model. It really burns the image. I have to set CFG 3-5 or use a rescale CFG strategy like below.

The prompt is not followed very well and sometimes completely ignored

It's fun to use for sure

likeliness is fine I guess

[xiaohu2015]

yes, use a low cfg scale, e.g. 3.0

[xiaohu2015]

some cases with Juggernaut-XL-v9

[cubiq]

More tests. It's fun to use for sure but not easy.

[xiaohu2015]

yes, it is more strong compared to the normed version (text maybe hard to change the face style, you have to lower the weight). and need to use some base models to get better results

[cubiq]

Likeliness is so-so, at least on its own I haven't checked with other face models

[xiaohu2015]

yes, it is almost the same as the normed version for face likeness

[JorgeR81]

The image with the marble statue looks amazing !

@cubiq, in the future, maybe you could use that prompt and settings, in one of the workflow examples.

[cubiq]

forgot to update that the unnorm is now supported for everybody to use. It's definitely better than sdxl portrait but it's very difficult to use and burns the image a lot, so you have to lower the CFG and the image doesn't follow the prompt

[JorgeR81]

Another great tutorial from @cubiq :
https://www.youtube.com/watch?v=4jq6VQHyXjg

Attention Masking gives excellent results, when the characters are separated in the image.
But what would be the best way make characters interact with each other ( e.g. hugging or holding hands ).

Could it be possible to create a new type of IPAdapter model, trained in the various ways characters can interact ?
Or maybe SDXL is just not good enough for this ?
( Perhaps this could work better on SD3 ? )

[cubiq]

if the style between the characters doesn't change, it's totally possible, maybe adding controlnet

[flamed0g]

@xiaohu2015

I am currently playing around with just FaceIDV2 on it's own (parameters down below) but I am wondering if there are currently any workarounds for the following issue that I'm having. The overall quality of the picture is very dependant on the reference image, as shown below.

As you can see, the left image is a very good quality but the right image looks very 'flat' without any shadows and the lighting is also a bit strange.. I seem to have this problem very often and I guess you could possibly fix this issue by using a difference reference image of course but I was wondering if this is a known issue. Personally I'd expect to have the quality of the left picture since only the face embeds are used, but it looks like the overall lighting conditions/shadows are also very important.. which is quite unfortunate since that completely ruins the quality in my case.

I am not sure if this is expected behaviour so definitely correct me if I'm wrong but I am also sure that many others are experiencing the same issue, since some reference images might have 'too much lighting' which has a bad influence on the output image.

Or perhaps anyone else that could help me out with this? I already have the start_at set to 0.1 (also for the example above) which makes it a tiny bit better but still not great..

[flamed0g]

These are the settings that I've used for the above example

I have tried lower LoRa weights, lower FaceID weights etc..

The left picture looks much more 'alive'.

Edit: another example:

[StellarBeing25]

@cubiq Can you confirm whether ConsistentID ( https://github.com/JackAILab/ConsistentID ) is using InsightFace for face embeds like FaceID or if it is similar to Plus Face? I assumed that it was more similar to the Plus Face, so I tried to compare it to the Plus Face at weight 1.0 using face analysis. The problem is that face analysis scored the faces generated with ConsistentID but refused to score the faces generated using Plus Face. Below is the face analysis score from consistentID

[StellarBeing25]

Will FaceID give better results when combined with ConsistentID instead of Plus Face?

[cubiq]

I believe Consistentid uses FaceID plus and yeah I can confirm it uses insightface

[StellarBeing25]

Well, it is of no use then; FaceID v2 gives a better score.

[StellarBeing25]

One last question: Why all the Face IP Adapters perform so much better with Taylor Swift's face as compared to other faces?  The results I get with FaceID v2 using her face is amazing.

[cubiq]

depends on the checkpoint bias

[xiaohu2015]

a new face adapter https://huggingface.co/spaces/yanze/PuLID, maby be can added in your repo @cubiq

[CapsAdmin]

I did a crude implementation here https://github.com/CapsAdmin/ComfyUI_IPAdapter_plus/

The main differences it seems are that it uses some other "eva clip encoder" which takes in a mask cropped face with a white background. And embeddings from antelopev2 instead of buffalo_l, same as instantid iirc.

Annoyingly pulid seems to be based on ip adapter but deviated slightly. I transform the pulid.bin model on load to what the ip adapter code expects and try to make use of as much as possible in ipadapter.

I left the cross attention code the same, but in pulid there are some experimental stuff there it seems, similar to style presets perhaps.

[cubiq]

I think I'm gonna release it as a stand alone extension, it doesn't need all the sophistication in the cross attention... I'll think about it

[cubiq]

I tried a very quick implementation, everything seems fine but nothing really happens, like the weights are not applied at all. I'll check later if I have time

[cubiq]

okay here it is https://github.com/cubiq/PuLID_ComfyUI

[cubiq]

@xiaohu2015 do you mind contacting me privately on Discord (I'm matt3o) or email matt3o at gmail or twitter https://twitter.com/cubiq ?

I have been offered compute power to train a free face model and I would like to talk to you about it if you are interested.

[xiaohu2015]

hi, you can email me with huye@tencent.com

[POD319]

Is this discussion at all about what the impact of loading different models into a 'Load Checkpoint' is which is then hooked in to 'Apply InstantID'? I have that question myself and didn't want to start a new discussion if that is relevant to topics discussed here.

Just looking to see what impact (if any) when you load different checkpoint models that way.