Add ddim noise comparative analysis pipeline

[aengusng8]

Research question: What visual concepts do the diffusion models learn from each noise level during training?

The P2 weighting (CVPR 2022) paper proposed an approach to answer this question, which is their second contribution.
The approach consists of the following steps:

1. The input is an image x0.
2. Perturb it to xt using a diffusion process q(xt|x0).
   • strength is a value between 0.0 and 1.0, that controls the amount of noise that is added to the input image. Values that approach 1.0 allow for lots of variations but will also produce images that are not semantically consistent with the input.
3. Reconstruct the image with the learned denoising process pθ(ˆx0|xt).
4. Compare x0 and ˆx0 among various t to show how each step contributes to the sample.
   The authors used openai/guided-diffusion model to denoise images in FFHQ dataset. This pipeline extends their second contribution by investigating DDIM on any input image.

import torch
from PIL import Image
import numpy as np

image_path = "path/to/your/image" # images from CelebA-HQ might be better
image_pil = Image.open(image_path)
image_name = image_path.split("/")[-1].split(".")[0]

device = torch.device("cpu" if not torch.cuda.is_available() else "cuda")
pipe = DiffusionPipeline.from_pretrained(
    "google/ddpm-ema-celebahq-256",
    custom_pipeline="ddim_noise_comparative_analysis",
)
pipe = pipe.to(device)

for strength in np.linspace(0.1, 1, 25):
    denoised_image, latent_timestep = pipe(
        image_pil, strength=strength, return_dict=False
    )
    denoised_image = denoised_image[0]
    denoised_image.save(
        f"noise_comparative_analysis_{image_name}_{latent_timestep}.png"
    )

Here is the result of this pipeline (which is DDIM) on CelebA-HQ dataset.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Super cool!