Image2Image fit option blends pixels when scaling down which ruins mask edges

[mavispuford]

Describe your environment

• GPU: RTX 3080 founders edition with i9 9900k
• VRAM: 12
• CPU arch: x86
• OS: Windows
• Python: miniconda
• Branch: main
• Commit: 89da371

Describe the bug
When passing a masked image with "fit" enabled, it scales down the image and blends the pixels in the process. This causes artifacts near the edges in the output. Maybe there should be a "nearest neighbor" scaling option?

To Reproduce
Images:
Source image:

Large masked image (2496x1664):

Small masked image (960x640):

Steps to reproduce the behavior:

1. Enter a prompt ("a frowning woman") using the large masked image above, default settings (50 steps, strength .75) but with width at 960 and height at 640
2. Notice the seams around the masked area
3. Now try it with the small masked image
4. Notice that it is seamless

Expected behavior
There should be no seams/artifacts in the masked area. The resulting image should be seamless

Screenshots
Large image output:

Small image output:

Additional context
Could you add a nearest neighbor/non-filtered scaling option so the pixels don't get blurred? I like the "fit" option because I don't have to resize images on my own, so it would be nice if it handled masked images well.

[lstein]

I see the problem. Would the PIL.Image.Resampling.NEAREST resampling filter be the best choice here?

[mavispuford]

@lstein Seems like that would do it. There's definitely a quality difference when using nearest neighbor vs the other methods, though, so it'd probably be best as a parameter. Either that or I suppose you could switch to nearest neighbor when transparent pixels/inpainting is detected.

[lstein]

I'll give it a try as soon as I get access to my development environment again. (It's an HPC cluster down for maintenance over the weekend). In the meantime, if you'd like to swap the nearest resampling in, just open the file ldm/dream/image_util.py, look for the line resized_image = self.image.resize((rw,rh),resample=Image.Resampling.LANCZOS) and change the resampling method to Image.Resampling.NEAREST

[mavispuford]

Seems to work for me. Here's my output using the large masked image with 960x960 chosen as the size:

[lstein]

This should be fixed now.