Automask by layer for img2img

[guzuligo]

This is something I usually need when I draw something with a brush, or when I use a stamp with alpha channel.

I want to apply img2img without adding a black background.
This is still possible by using the mask brush, but it is tedious and very inaccurate.

My suggestion is to add an option besides the visibility of the layer to apply layer mask

[zero01101]

so i'm definitely missing what you're doing here; of course we say that img2img is definitely not intended for outpainting, but i've just tried dumping manual masking around a transparent stamp and trying the different inpaint modes and wow none of them turned out good at all lol; can you post some examples/use case/maybe a small "tutorial" to do what you're doing manually? i'm not AGAINST the img2img tool being used for outpainting but yeah, that's currently the dream tool's domain

[seijihariki]

The problem with generating anything outside the existing image is that steps will be scaled according to denoising.

Even if we fix the more obvious issue by manually introducing noise in an otherwise 'original' masking query, if you use 30 SD steps with 0.7 denoising with a half-empty area, that area we introduced latent noise to would be run through only ~17ish steps.

[guzuligo]

Actually, I guess this is because I am using it as an inpainting and as an art program rather than just an outpainting tool.
I do realize it is kinda out of scope 😅

Let me give some use cases, hopefully you can think of a better way than what I proposed:

• I generate BG, then bring a png of a (character, building, object) with alpha channel and place it in a separate layer. Now I want to only inpaint that object without the background. The current method is to use mask, and even so, it will create stuff in the new layer.
• I want to img2img an object and use it as a stamp
• I want to fix an edge with img2img before I outpaint.

The idea is that I want the mask to cut the extra stuff generated and utilize the alpha channel as a masking tool.

[seijihariki]

Oh, so you mean a 'keeptransparent' option? Like, what was transparent before continues transparent now? If so, that we can probably do that.

But as img2img uses the full visible info, not sure what layer-by-layer would actually do, and would make things very complex.

[seijihariki]

I think the best option regarding masking by layer in this case would be to add two buttons to the mask brush like 'masktransparent' and 'maskcontent' that creates masks based on the currently selected layer.

Adding a mask button to the layers themselves would be heck to implement (what would happen if one tried to erase layer masking, and would there be any visual hints regarding masked layers?, Etc...)

[guzuligo]

What if it was in the img2img context menu? An option to:
-mask by none
-mask by layer
-mask by visible
-mask by mask

So when img2img finishes generating, the extra parts get cut based on selection.