[Bug]: Denoising strength not working properly with latent

[Dasor92]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

If you set the denoising strength under 0.6 the result image is blurry and deformed, i think it's caused by the amount of noise added to the image before upscaling. i don't know how it used to work but i think the noise added was lesser when choosing a lower strenght. i suggest restoring the old denoising strength script to add noise.
The issue is not present with a strong upscaler like Esrgan
images for reference:
Latent:

Nearest:

EsrGan:

Steps to reproduce the problem

1. Open WebUI
2. Type any prompt
3. Activate Hires Fix
4. Set Denoising Strength to 0.2
5. watch the first image generated perfectly fine
6. watch the "denoised" image, blurry

What should have happened?

The image before the last commits used to be perfectly fine even with 0.1 denoising strength

Commit where the problem happens

f8d0cf6

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

No response

Additional information, context and logs

No response

[Jonseed]

I've been wondering why low denoising strength has been causing blurriness and distortion when using hires fix. Low denoising should just result in the generated image being closer to the original (upscaled?) image. It's definitely not like it was before the hires fix UI change. Something about adding the different upscalers into the pipeline possibly?

[auraria]

I've noticed the same with the latents, None seems to be the best option so far.

[darcula1993]

same here. low denosing strength just show blurr/artifacts results.

[jimys]

same here. low denosing strength just show blurr/artifacts results.

OMG, 3 month past, still not fixed? this made the hires.fix unusable

[ToxicObsidian]

I've checked the codes and I found the reason + alternative solutions.

1. Hires. Fix is applying an img2img after txt2img process.

When you inspect modules/processing.py you can find the hires. fix process in StableDiffusionProcessingTxt2Img.sample(). The line below is the main function of Hires. fix.

samples = self.sampler.sample_img2img(self, samples, noise, conditioning, unconditional_conditioning, steps=self.hr_second_pass_steps or self.steps, image_conditioning=image_conditioning)

2. The Latent samplers are the same as what you selected in txt2img.

You can find the definition of self.sampler just a few lines above. When you apply print(f'{img2img_sampler_name}'), you can find that it is the same as what you selected in txt2img (not Hires. fix), except PLMS.

The author comments: PLMS does not support img2img so we just silently switch to DDIM
This means you can hijack the txt2img process and replace another sampler of the hires. fix process, technically.

3. Unable to locate the error (because it is not an error).

After inspecting the source code of image generation, I found everything normal. (Seemingly)

The sample_img2img() function is defined like:

def sample_img2_img(self, p, x, noise, ...):
    ...
    return samples

The parameter p is a general process object, containing configs like steps, denoising_strength, and extra_generation_params, etc. (denoising_strength is used rarely, I didn't even find where it exists, and this is the real problem)

The parameter x is the basic latent.

• In Img2Img process (not hires. fix), x is the initial latent.
• In Txt2Img Hires. fix, this is the truncated sample (image).

The parameter noise is just random tensors created by a certain function.

These codes work very well in the Img2Img process, so I think that's not what we need.

4. Troubleshooting the blurry or glitchy latent.

After modifying the modules/processing.py source code, I captured the intermediate image just after the latent was upscaled. I'm using Latent (nearest-exact) at a very low denoising strength (about 0.15), look at these images. The reason caused the problem is there.

modified code in StableDiffusionProcessing.sample():

samples = torch.nn.functional.interpolate(samples, size=(target_height // opt_f, target_width // opt_f), mode=latent_scale_mode["mode"], antialias=latent_scale_mode["antialias"])
for i in range(samples.shape[0]):
    sample_image = sd_samplers.sample_to_image(samples, i, approximation = 0)
    images.save_image(sample_image, self.outpath_samples, "", seeds[i], prompts[i], opts.samples_format, suffix="-hires-just-after-upscale")

Original (before hires. fix):

Just after upscale:

Complete hires fix:

So we can finally conclude that the blurry/glitchy/deformed images are caused by the upscalers.

5. Why?

Image upscaling is bad while using the bilinear/bicubic/nearest/nearest-exact, but we didn't see such bad images in Img2Img, so I inspected the Img2Img process.

The truth is that the Img2Img process doesn't use latent upscaling at all.

Instead of using the torch.nn.functional.interpolate() method, the Img2Img process uses the resize_image() method to resize and upscale images, whose definition is located at modules/images.py.

def resize_image(resize_mode, im, width, height, upscaler_name=None):
    upscaler_name = upscaler_name or opts.upscaler_for_img2img

    def resize(im, w, h):
        if upscaler_name is None or upscaler_name == "None" or im.mode == 'L':
            return im.resize((w, h), resample=LANCZOS)
        ...

In this case, we should only focus on the resize() function. After printing the upscaler_name and a None has been given, I finally found that the Img2Img process is very likely to use LANCZOS as the upscaler. Not the interpolation method.

6. Test for LANCZOS

After changing the upscaler of hires. fix to LANCZOS, I got an image without blurry/glitchy/deformed areas while using a very low denoising strength (about 0.15).

Before hires. fix:

Using the LANCZOS upscaler:

7. Conclusions & Alternative solutions

So, changing the upscaler to LANCZOS is the solution?

No, it's one of the alternative solutions.

To be clear, I'm not an ML expert, even not an ML student, so the explanations may not be reasonable

When we apply the Latent upscaler to hires. fix, it actually brings more potential to the image. So Latent upscalers should require more denoising strength to process such potential into image details. On the other hand, lower denoising strength may cause such potential to remain on the image.

By using the other upscalers like LANCZOS or ESRGAN series, they reinforce the image. Less potential is given, and fewer differences are taken place.

======
So, there's no 100% solution for such image generation. But you can follow these steps to avoid such image generation:

• If you are confident about the model you are using, and want fewer modifications on the image.
  1. Lower your denoising strength to ~0.4.
  2. Switch your hires. fix upscaler to non-Latent upscalers (e.g. LANCZOS, ESRGAN).
  3. Bang! You get some nice images.
• If you want more modifications and more potential.
  1. Apply a higher denoising strength (~0.55 to ~0.7)
  2. Pick a favorite Latent upscaler
  3. You get just what you want.
• For a general solution.
  1. Don't use hires. fix
  2. use extra->upscale to upscale your image (make your image sharper if possible)
  3. send the upscaled image to img2img
  4. if you want more potential, using Inpaint and Inpaint Sketch is better.
  5. process and finetune.
  6. You will get some awesome images. (just wasting too much time compared with other alternative solutions.)
• For a geek solution
  1. Hijack the txt2img process
  2. Apply some geek techniques.
  3. Woolah, you are the best guy!

I'm not very good at English, so please stand my expression faults.
Hope my post can help you guys <3.

---

Update:

This method is also an alternative way. But it also gives less potential.

[catboxanon]

This is not a bug. Upscaling the latent directly is expected to be noisy. The fixed mentioned above is simply substituting in a different upscaler. You can see just how noisy the latent actually is by setting the hires fix parameters to these mentioned on the wiki (or just look at the noisy images themselves for reference): https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#upscalers

The None option uses bilinear upscaling (which, with the denoising process of diffusion, comes fairly close to Lanczos). For SD1/2 it is preferred to use a GAN-based upscaler or similar.

Closing.