How do you use inpaint?

[huchenlei]

Currently ControlNet supports both the inpaint mask from A1111 inpaint tab, and inpaint mask on ControlNet input image. According to #1768, there are many use cases that require both inpaint masks to be present, and some use cases where one mask must be used.

This poll is to collect some data on how people use the ControlNet inpaint feature so that we can decide what is the best way to handle inpaint mask.

Here are some different usage of inpaint:

1. txt2img inpaint
2. img2img inpaint. img2img tab, inpaint on ControlNet's input (inpaint_global_harmonious)
3. img2img inpaint. img2img inpaint tab, inpaint on A1111's input (inpaint_only, inpaint+lama)
4. img2img inpaint. img2img inpaint tab, inpaint on both ControlNet's input and A1111's input (Use cases described img2img+inpaint broken #1768)

Note:
Some configurations essentially have same effect, e.g.

• using txt2img inpaint with inpaint global harmonious vs using img2img tab, inpaint on ControlNet's input with inpaint global harmonious.
• using txt2img inpaint with inpaint only and using img2img inpaint on A1111's input with inpaint only.

Some configurations are not intended, and will not provide good result:

• using img2img inpaint on A1111's input with inpaint global harmonious
• using img2img tab, inpaint on ControlNet's input with inpaint only
• using img2img inpaint, inpaint on both A1111's input and ControlNet's input with no overlapping of 2 masks
• upload different image to A1111 img2img input, and ControlNet's input

[2blackbar]

AFAIK most people use only masked cause it preserves original resolution of the image, other modes are pretty much inferior cause ytou need sometimes very large sizes to just inpaint like haircut on 4k photo , you can do this with only masked and use 512 res but if you wanna do it in other modes youre pretty much cant unless you want much lower resolution output than original image and on top of that if you want to use that 4k to preserve it then you have to wait a lot for image to be generated and you just wanted haircut change...
Im aware that in order for inpaint to work it has to see entire image and figure out what should go where and only masked by its nature only feeds it small part of the image, but maybe it would be better to feed it entire image but only work on that masked area after it alredy figured out what should be there after seeing entire image and not just "only masked " area.

[2blackbar]

i use firefly website for inpaintings for now, inpaint controlnet models have too many landmines , if you wont set your settings just right - it will generate poor result, i wish it was locked somehow under the hood to its best settings...

[huchenlei]

That is what I am doing now. I am trying to figure out how people are using it, and lock it to a state which people cannot mess up settings. However, according to #1768, there are many use cases of inpaint I don't even know about when I add the constraint on inpaint configuration.

[2blackbar]

if you would show people how much resolution they lose by not using only masked they would all switch...

[2blackbar]

Also, it would be really much much better to have separate inpaint annotator for removing stuff and separate annotator for inpainting new things, like for example letting lama do its thing and dont add stuff just refine it.

[catboxanon]

maybe it would be better to feed it entire image

Well, you can do this... but the issue is the base model (the ones currently in existence at least, base SD and the NAI leak) is only trained at a resolution of 512px or 768px. There's no hotfix for what is a limitation of the model. It is impossible to "feed it [the] entire image" and then "only work on that masked area after it alredy figured out what should be there", the inference process must be done in tandem. Not to mention most users would struggle trying to fit such high resolution images as you describe in VRAM. You either must inpaint within the masked area or settle for dealing with lower resolution.
The cascaded workflow is the best of both worlds but even that is limited.

[Fishkin]

The biggest problem with inpainting is that when using img2img inpainting or upscaling, the ControlNet's input image doesn't get cropped/resized appropriately to match what WebUI does to the inpaint area. By "ControlNet's input image" I mean the one you upload when you enable "Upload independent control image" checkbox.

When you inpaint with "Whole Picture", there is no issue, as the ControlNet input image gets appropriately applied to the entire img2img input image.

However, when you inpaint with "Only masked", the problem appears. WebUI will crop the masked area (plus padding around it), resize it to the dimensions specified, then generate for inpaint, resize it back to original and merge.

The ControlNet's "independent control image" does not receive this treatment: it doesn't get cropped, resized, or whatever. It's still the same one used from the Whole Picture, while the img2img input image is cropped, the entire ControlNet image is used for just that small "Only masked" portion, resulting in mismatch.

This happens only if you upload your own control image. If you let the preprocessor do its thing automatically, it will use the properly cropped and resized input image from WebUI without issue.

The same problem is when you use an upscaler script like Ultimate SD Upscale, but I guess that makes sense because it does inpaint on each tile under-the-hood, which reveals the problem.

Here's example pictures from inpainting with Ultimate SD Upscale, because it's much more obvious what happens. We won't actually upscale anything automatically, just do img2img in tiles.

Let's start with this base 512x512 image:

We then upscale it to 1024x1024 manually in Extras tab with ESRGAN (doesn't matter):

Now let's do a img2img in tiles with high denoising and Canny ControlNet (this is just for example purposes). Denoising Strength set to 0.75 default is fine.

Preprocessor: canny. Upload independent control image is OFF. Ultimate SD Upscale set to "Scale by 1" (aka just redraw in tiles, don't upscale):

As you can see, looks as expected, the canny edges provided enough control for it to actually draw similar, and they were all drawn properly for each of the 4 tiles.

Now, let's preprocess the image manually and then upload it as independent control image (with no preprocessor), the result should be the same, right? So first, we preprocess it:

Now upload it as "independent control image" and set Preprocessor to none. Then Generate:

The result is clearly messed up. This is because, for each tile, the entire ControlNet image is used, so we essentially have this as ControlNet image:

Which is bad and unexpected. What should happen during "Only masked" inpainting (the script just inpaints each tile) is that the ControlNet's custom uploaded image should also get cropped and resized to fit just that area, instead of using all of it for each mask/region/tile.