Inpainting colors desaturated

[ThereforeGames]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

Hi,

It seems that img2img inpainting produces very washed out results, regardless of color correction mode. I'll list what I think are the relevant settings below.

I may update this post with example images later, please let me know if that would be helpful.

Steps to reproduce the problem

1. Set inpainting conditioning mask strength to a value > 0
2. Use either the SD 1.5 inpainting model or one finetuned from it
3. Use high denoising and CFG scale to make the problem more apparent
4. Masked content mode "original"

All other settings--including sampler--seem to be irrelevant here.

What should have happened?

The contrast of the original image should be generally preserved. I reverted to a commit that's ~4 days old (don't know the hash, sorry) and it was working fine then.

Commit where the problem happens

4af3ca5

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Brave

Command Line Arguments

--disable-safe-unpickle --xformers, though I tried other combinations of arguments

Additional information, context and logs

N/A

[aatamakaycee]

My inpainting even doesn't work, try to correct eyes the results is only black color painted over eyes

[mclsugi]

Is it still washed out if you changed the VAE's used? Mine were always have those kind of results if I'm using VAE meant for anime generations (like they put greyish layer on top of it).

[ThereforeGames]

Is it still washed out if you changed the VAE's used? Mine were always have those kind of results if I'm using VAE meant for anime generations (like they put greyish layer on top of it).

Yeah, changing the VAE didn't seem to make a difference. It is definitely like a "greyish layer on top" though.

I'll run some more tests soon, see if I can identify the culprit and post some example pics.

[ThereforeGames]

Oh! I figured it out!

I made a customization in processing.py that I neglected to include in my PR #5644 :

I don't understand the inpainting model well enough to explain why my one-line version works better, but look at the difference it makes:

Current version (washed out)

My version

At least for faceswaps, I don't notice any side effects with my version, but I'm hesitant to submit another PR without understanding the reason for the washed-out look.

Thoughts?

[mclsugi]

Well, subjectively speaking, given the example above, I like your version better. My observation so far concluding that SD treat faces differently than any other parts of a picture, way too much emphasis on human face or faces in general. Perhaps you could give another examples that does not include face if you trying to raise a new PR without proper reasoning behind it, IMHO.

[ThereforeGames]

Perhaps you could give another examples that does not include face if you trying to raise a new PR without proper reasoning behind it, IMHO.

Sure, here's one more.

Original image:

Swap to "tomato" at 0.75 conditioning mask strength

Current version:

My version:

I don't know if the current implementation is bugged, but I have never been able to get it to produce a pleasing result when the conditioning mask strength value is > 0 and like < 0.95.

Not only is it washed out, it also doesn't seem to morph the shape of the image at high values nearly as much as one might expect.

[mclsugi]

I don't really know how the results is meant or supposed to be, TBH. I kind of just adapt to what I have kind of guy.

Using your source image, I can confirm that it is washed out and does not even try to morph to what in prompt if we are using the 'original' masked content. AFAIK, not changing the subject of the masked content is what original supposed to do, but TBH I don't know how it was supposed to. Regarding the washed out, yes, I can confirm that.

My results:

Current version:

Your version:

Side by side comparison:

EDIT: * It is even more DESATURATED in my examples. Different configs somewhere perhaps?

[ilcane87]

@EllangoK
Why close an issue that isn't solved?
Just try to run img2img with conditioning mask < 1, and then run it again in inpainting; the second result is entirely different, way too close to the source image, and with washed out colors.

[dsxailab]

For this guy, if we inpaint his shirt (instead of face) to show his skin, lol, the result will be the alpha blend of skin color and shirt color. Looks extermly bad when he is wearing a black shirt. Hmm, seems the color correction is NOT (updated, was missing) designed for this anyways.

[Enferlain]

Dunno if this is the same issue, but some models (possibly vae related?) will give scuffed colors when you inpaint

Even though in the preview just before generation completes it seems fine

Does anyone know what I could do to fix this? It makes editing really hard

It was actually somehow the default orangemix vae not being loaded in img2img. Forced it on in settings, set --no-half-vae (it wouldn't work otherwise) and way better results now

[catboxanon]

Related/dup issues: #2754, #5557

I do think this is VAE related but it would be appreciated if somebody more knowledgeable can give feedback and/or know how to fix what exactly is happening here before closing this. The VAE of choice, --no-half-vae, and the color correction option can resolve this in some cases, but results are still inconsistent. The answers and advice most seem to give is based on their own experience and workarounds, and not actual solutions or well-informed reasons. VAE of choice is likely the root cause though, as far as I can tell.

It should be mentioned as well that there are only a few known VAEs for SD1/2 models that are not simply merges or out encoder layer fixes, the latter of which is actually a bad idea to even use as it harms hires fix results. saltacc/VAE-BlessUp#1 (comment) (tl;dr: loss of detail and color range)

This is what should be worked off of for resolving this. Any other VAE is likely tampered with or is a merge that is not really representative of it's original training.

• "SD1.0" VAE (part of early SD checkpoints and the leaked NovelAI model; known for producing distorted faces/purple spots/etc.; universally recommended to not use)
• Finetunes of https://github.com/CompVis/latent-diffusion#pretrained-autoencoding-models (kl-f8)
  • https://huggingface.co/stabilityai/sd-vae-ft-ema-original (vae-ft-ema-560000-ema-pruned)
  • https://huggingface.co/stabilityai/sd-vae-ft-mse-original (vae-ft-mse-840000-ema-pruned)
  • https://huggingface.co/hakurei/waifu-diffusion-v1-4/tree/main/vae (kl-f8-anime & kl-f8-anime2)
• Leaked NovelAI VAE (animevae) (might be finetuned off the SD1.0 VAE, which could explain the somewhat desaturated colors)