Staircase/Scanline artifacts with SDXL 1.0

[n00mkrad]

Describe the bug

SDXL creates colorful scanline-like artifacts, especially on areas with high-frequency detail.

SDXL - Diffusers 0.19.0 - 25 Steps of DPMSolverMultistepScheduler with Karras - 20% refine steps - CFG Scale 7

Left: SDXL 1.0 - Right: SDXL 0.9

Reproduction

Run SDXL 1.0 and compare results with SDXL 0.9

Logs

No response

System Info

Diffusers 0.19.0 - Windows 10 22H2 - RTX 4090 with drivers 536.40 - FP16 model

Who can help?

No response

[n00mkrad]

Also happens without refiner model.

[n00mkrad]

Workaround: Copy the 0.9 VAE to the 1.0 model.

The 1.0 VAE is broken.

LEFT: New VAE - RIGHT: Old VAE

[sayakpaul]

Thanks for investigating. We think it will be beneficial for potentially similar future issues :)

[catboxanon]

Did StabilityAI ever address this other than just silently dropping a "fixed" version (using the 0.9 VAE in the 1.0 checkpoints, as described above) in their respective repos? There's so much confusion and miscommunication around this that even the ControlNet author wasn't aware of this. (see discussion starting from here)

https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors
https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors

[fralumz]

The onnx vae should be reverted to 0.9 as well.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[catboxanon]

Still interested to know whether or not StabilityAI officially addressed this, outside of silently releasing a "fixed" version. Seems awfully fishy to me.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[catboxanon]

Bumping again.

[patrickvonplaten]

As always, we need a reproducible code snippet here :-)

[catboxanon]

You made this change though.

https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/discussions/30
https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/discussions/8

So, technically this issue has already been addressed. However, my question in #4310 (comment) has not been.

[Woisek]

This VAE fixed it for me:
https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/tree/main

[catboxanon]

That VAE works because it also had to be reverted back to the 0.9 VAE.

https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/commit/6d1073461cd0b5a6ea4fda10b812e3d9d58a8330

Further-reduced discrepancies with original SDXL-VAE (0.9) decoder

It still does not answer why this had to be done, and why StabilityAI made no official announcement regarding it. I believe it's incredibly misleading that a VAE labeled a version number higher, as "1.0", would have a scanline-like "watermark".

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[catboxanon]

Bumping again.

[sayakpaul]

It is still missing a reproducible code snippet.

[catboxanon]

from diffusers.pipelines.stable_diffusion_xl.pipeline_stable_diffusion_xl import StableDiffusionXLPipeline
import torch
import gc
from PIL import Image

for url in [
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors",
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors",
]:
    generator = torch.Generator(device="cuda").manual_seed(42)
    pipeline_text2image = StableDiffusionXLPipeline.from_single_file(
        url, torch_dtype=torch.float16, variant="fp16", use_safetensors=True,
    ).to("cuda")
    prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
    image: Image.Image = pipeline_text2image(prompt=prompt, generator=generator).images[0]
    image.save(f'{".".join(url.split('/')[-1].split('.')[:-1])}.png')
    del pipeline_text2image
    gc.collect()

sd_xl_base_1.0.png	sd_xl_base_1.0_0.9vae.png

1600% zoom of each.

sd_xl_base_1.0.png	sd_xl_base_1.0_0.9vae.png

The docs use sd_xl_base_1.0.safetensors for from_single_file() as well. https://huggingface.co/docs/diffusers/using-diffusers/sdxl#load-model-checkpoints

A user should expect something labeled "1.0" to be the model to use and to not contain VAE decoding artifacts. They should not need insider knowledge that something additionally labeled "0.9" is actually the better model to use because VAE decoding artifacts are not present.

[sayakpaul]

@patrickvonplaten WDYT?

[patrickvonplaten]

Hey @catboxanon,

This is really more an issue for the repo here: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/discussions as Stability AI decides how to name the files in their repo. Could you open an issue there?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.