Add circular padding for artifact-free StableDiffusionPanoramaPipeline

[EvgenyKashin]

Artifact-Free Panorama Generation with circular padding

This pull request addresses the issue mentioned in #4024. The proposed solution aims to achieve artifact-free panorama generation by seamlessly aligning the content on the left and right sides of the generated image, creating a proper “panorama”, viewable in 360 panoramas viewers.

In the existing MultiDiffusion approach, the image is cropped from left to right with a specified stride. To enable circular padding and support the desired 360-degree scenario, this PR introduces additional crops after the rightmost point of the image. Circular padding allows the model to “see” the transition from the rightmost part of the image to the leftmost part, maintaining consistency in 360-sense.

Furthermore, modifications are made to the decoding of latents in the case of StableDiffusion. Without these modifications, the decoded latents would not match in the RGB space, even if they align well in the latent space. Thus, the decoding of latents also incorporates circular padding to ensure consistent results.

A panorama image generated without circular padding:

A panorama image generated with circular padding:

To maintain backward compatibility and preserve the previous behaviour, the proposed circular padding functionality is implemented as an optional argument “circular_padding”, which is set to False by default.

Usage example:

import torch
from diffusers import StableDiffusionPanoramaPipeline, DDIMScheduler

model_ckpt = "stabilityai/stable-diffusion-2-base"
scheduler = DDIMScheduler.from_pretrained(model_ckpt, subfolder="scheduler")
pipe = StableDiffusionPanoramaPipeline.from_pretrained(model_ckpt, scheduler=scheduler, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "an indoor equirectangular panorama of a living room"

image = pipe(prompt, width=1024, height=512,
    circular_padding=True).images[0]

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[EvgenyKashin]

Visualization of a concatenation of the most right and the most left parts of a generated panorama images: No circular padding (default):

Circular padding, but without decoding padding in the VAE (without decode_latents_with_padding method):

Circular padding full (circular_padding=True):

[EvgenyKashin]

In general, it's an easy fix for the stitching artifact problem, but maybe I've missed an easier way to do it with the current pipeline. Perhaps this is too narrow case to add this to the main.

I've also noticed tests for this pipeline that I break with my changes (even with default circular_padding=False option, that is just the same as current main in my local tests):
18 failed, 5 passed, 5 skipped, 2 warnings

I've tried "cuda" and "cpu", different batch_size and view_batch_size, it works fine. I couldn't understand why it's failing tests yet.

[EvgenyKashin]

@omerbt @sayakpaul @Isotr0py guys, what do you think?

[sayakpaul]

I will defer to @omerbt for this one. But I like what I see :)

[omerbt]

Looks great!

[sayakpaul]

Thanks @omerbt!

@EvgenyKashin, let's make sure the tests on the CI pass. Then I think we can approach merging.

[EvgenyKashin]

Thanks, guys. @sayakpaul I've fixed one place in the code. Basically, I decided to save one line originally, but one more check was required for small images case (like in test cases). I've also added a few simple test cases for circular padding.

[EvgenyKashin]

Sorry, originally tagged the wrong person, @sayakpaul

[sayakpaul]

Looks great to me! Thank you.

Left some minor nits. One thing we could do is add a note about circular_padding to the panorama docs: https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/panorama.mdx.

[EvgenyKashin]

Thanks for the review @sayakpaul, that's improving the PR! I've rewritten the doc-string, added information to the .mdx and removed the slow test.

[EvgenyKashin]

Could we merge it?

[sayakpaul]

Looks amazing @EvgenyKashin! Thanks much for iterating. I just have one more ask here: #4025 (comment) and then we will ship this 🚀.

[EvgenyKashin]

@sayakpaul could you approve two images merging to the documentation-images https://huggingface.co/datasets/huggingface/documentation-images/discussions/132 ?

[sayakpaul]

Will merge after the CI passes.

[EvgenyKashin]

@sayakpaul thanks, ready to ship now 🚀

[sayakpaul]

Thanks very much for your valuable contributions!