ControlNet 1.1 Inpainting

[MoonMoon82]

How does ControlNet 1.1 Inpainting work in ComfyUI?

I already tried several variations of puttin a b/w mask into image-input of CN or encoding it into latent input, but nothing worked as expected.

[MoonMoon82]

@comfyanonymous could you please provide an example of ControlNet 1.1 Inpainting?

[MoonMoon82]

@comfyanonymous still don't have a clue how to use ControlNet inpainting model in your UI :-( .. could you please help?

[comfyanonymous]

I'm just guessing but maybe you have to do "Mask to Image" on the mask and pass that to the controlnet?

[MoonMoon82]

@comfyanonymous No, just simply passing an image mask into controlnet apply seems not to work.
On this website: https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint
I guess the code of the function should describe what has to happen:

def make_inpaint_condition(image, image_mask):
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask.convert("L"))
    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    image[image_mask < 128] = -1.0 # set as masked pixel
    image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
    image = torch.from_numpy(image)
    return image

But I'm not able to figure out what that could mean how to arrange nodes to get this thing working.
(I have no experience in pyhton/torch/numpy .. whatever)

[MoonMoon82]

@comfyanonymous Could you have a look at the code from the huggingface website above? Maybe you can figure out what needs to be done to get Controlnet Inpainting to work

[BlenderNeko]

I've made a PR to the comfy controlnet preprocessors repo for an inpainting preprocessor node. In the mean time you can also use the standalone node found in this gist.

see this image for an example workflow on how to use it:

[MoonMoon82]

@BlenderNeko Great work ! Thank you very much for this preprocessor node! It seems it's working as needed!
Do you also recognize some changes to areas that are not meant to be changed? I guess it has nothing to do with the preprocessor

[BlenderNeko]

I've not played with it extensively, but the only latents that should change when you use a setLatentNoiseMask node are those in the mask. I think those mostly affect up to 16x16 pixels in the image?