[Feature Request]: Inpainting should allow the user to select the colour of the overlay

[infinity0]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do?

Inpainting sucks at the moment, if you want to make very specific edits such as fixing broken limbs or other details. I get much better results when I draw in a stupid basic replacement in GIMP of what I want to generate, with 4-5 brush strokes, then give it to Vary (Subtle).

I guess the reason is because Inpainting starts from a blank transparent canvas. An easy way to improve on this is to allow the user to select the brush colour, and draw directly a crude replacement on top within the Fooocus UI, instead of doing the import-export dance with GIMP which wastes a couple of minutes.

Example below:

Input image

The picture makes no sense, ignore that for the purposes of this ticket - for now I just want to give the girl her leg back.

The overall prompt is "a girl reads, while she sits on the branch of a large tree. a boy does a cartwheel on the ground beneath her." - this was used to generate this image, and is kept for all the variations below.

Inpainting with mask

(The results are the same as for inpainting with a manual brush in the Web UI, I am providing a mask for easier verification and reproduction for the purposes of this ticket.)

Modify Content, additional prompt = "girl on the right has two legs"

Lighting Generate 8 times gives us unwanted stuff like this:

1/8 times I do get a leg, but this is very inefficient.

Manual edit plus vary (subtle)

OTOH if we edit the image ourselves in GIMP:

Then put this into "Upscale or Variation" -> Vary (Subtle)

Lighting Generate 8 times gives us a leg 8/8 times.

Many of the legs are weird, but that's the same with Inpainting too. We can fix this with higher quality variations, and so this is out-of-scope for this issue.

Proposed workflow

1. Inpainting should allow users to select the colour of the brush, including using a dropper to select colours already in the base image.

Additional information

I understand that Fooocus is not going to re-implement GIMP inside itself, but simply changing the colour of the brush gives a massive improvement. Having this option displayed prominently also gives a hint to the newbie user how to fix/improve upon their problem.

[mashb1t]

Currently a mask is binary, everything inpainted, no matter the color, is "modify", everything not inpainted is "not modify".
You can get better results when not writing prompts in natural language, as this is not how SD(XL) has been trained. Using "leg" as inpaint prompt instead of "girl on the right has two legs" will result in better results.

Currently there are no plans to implement editor features for inpainting, except for black to erase from a mask, see #2032 (comment). This may change in the future.

[infinity0]

My extremely crude understanding of how stuff works is that each "sample" step is a transition function from an initial state to a final state. Is that vaguely correct? So that means even the inpainting has to start from an initial state. My suggestion is to have this initial state be a colour rather than being blank or transparent.

[infinity0]

this is not how SD(XL) has been trained

Hm, would be good to put this info in a prominent place in the UI. Since the point of Fooocus is to "focus on prompting and generating", this obv includes understanding what to prompt.

[neckaros]

My extremely crude understanding of how stuff works is that each "sample" step is a transition function from an initial state to a final state. Is that vaguely correct? So that means even the inpainting has to start from an initial state. My suggestion is to have this initial state be a colour rather than being blank or transparent.

From my understanding what they do is replace the content under your mask with more or less noise depending on your settings (in advanced settings) not blank or transparent