StableDiffusionImg2ImgPipeline with low strength provides blurry result

[xxmtg]

Describe the bug

I have encountered a strange issue with the img2img pipeline recently

When I run the StableDiffusionImg2ImgPipeline with a low strength value, it will generate blurry images.

Where the original image is:

I have done some research about this issue and found #1368. Following the answer in that link, I have tried to increase the num_inference_steps by a factor of (1/strength), switch to different schedulers, and switch to different models. However, the results are mostly the same.

Reproduction

I am using a custom model, but "runwayml/stable-diffusion-v1-5" can reproduce the issue.
Below is a portion of my pipeline code:

class Img_diffuser:
    def __init__(self):
        torch.backends.cuda.matmul.allow_tf32 = True
        torch.backends.cudnn.allow_tf32 = True

        default_repo_id = "runwayml/stable-diffusion-v1-5"

        self.sched_euler_a = EulerAncestralDiscreteScheduler.from_pretrained(default_repo_id, subfolder="scheduler")
        self.sched_euler_a_iteration = 25

        self.pipe_t2i = StableDiffusionPipeline.from_pretrained(
                default_repo_id,
                scheduler=self.sched_euler_a,
                torch_dtype=self.torch_dtype, # torch.float16
        )
        self.pipe_i2i = StableDiffusionImg2ImgPipeline(**self.pipe_t2i.components)

        self.pipe_t2i = self.pipe_t2i.to(self.cuda_device)
        self.pipe_i2i = self.pipe_i2i.to(self.cuda_device)

        self.generator = torch.Generator(device=self.cuda_device).manual_seed(1234)

    def i2i_redesign(self, np_img, prompts):
        self.pipe_i2i.scheduler = self.sched_euler_a
        noise_strength = 0.25 # higher noise_strength generates normal result (or it is just harder to tell the bug from the image)
        return self.pipe_i2i(
            prompt=prompts["pos"],
            image=np_img,
            strength=noise_strength,
            num_inference_steps=self.sched_euler_a_iteration * (1.0 / noise_strength),  # increasing num_inference_steps does not help
            generator=self.generator,
            guidance_scale=7.5,
            negative_prompt=prompts["neg"],
            num_images_per_prompt=6, # I have tried to set this to 1 as well, same result
            output_type="np",
        ).images

prompts = {
    "pos" : "A simple house",
    "neg" : "blurry",
}
img_diffuer = Img_diffuser()
images = img_diffuer .i2i_redesign(test_img, prompts) # where test_img is the screenshot house

I compared my pipeline with webui's parameters and they seems match:

Logs

C:\Users\Python311\Lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_img2img.py:545: FutureWarning: You have passed 6 text prompts (`prompt`), but only 1 initial images (`image`). Initial images are now duplicating to match the number of text prompts. Note that this behavior is deprecated and will be removed in a version 1.0.0. Please make sure to update your script to pass as many initial images as text prompts to suppress this warning.
  deprecate("len(prompt) != len(image)", "1.0.0", deprecation_message, standard_warn=False)
100%|█████████████████████████████████████████████████████████████████████████████| 25/25 [00:08<00:00,  3.04it/s]

I have this information in the log

System Info

Below is my diffusers version:

import diffusers
diffusers.version
'0.19.3'

Python 3.11.4

Who can help?

@patrickvonplaten @sayakpaul @will

[xxmtg]

Just to point out, the images I got from StableDiffusionImg2ImgPipeline look like they are still in the latent space.

[xiankgx]

From what I see, it's not blurry image, rather it's like the colors or brightness is off. I had similar experience in this, and this is because you are using an image of type np.ndarray. Change it to type PIL.Image.Image and your problem fixed.

[xiankgx]

It's a case of incorrect image normalisation and/or denormalization.

[pedrogengo]

Try to open your image like that:

from PIL import Image

test_img = Image.open(<path>).convert("RGB")

[xxmtg]

Worked, thanks!
😁