[SDXL-lightning Inference] diffusers inference result is different with webui

[ApolloRay]

Describe the bug

I use diffuser to generate the image with dreamshaper_lightning model, but the result is really strange.
Model url:https://civitai.com/models/112902?modelVersionId=354657

Reproduction

`import torch
from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler, DPMSolverMultistepScheduler
from transformers import CLIPTextModel

base_model_path = "dreamshaperXL_lightningDPMSDE.safetensors"

pipe = StableDiffusionXLPipeline.from_single_file(base_model_path,torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True, algorithm_type="dpmsolver++")
pipe = pipe.to("cuda")

prompt = "a girl"
negative_prompt = "NSFW,"
generator = torch.Generator(torch.device("cuda")).manual_seed(-1)
num_inference_steps = 4
guidance_scale = 2
image = pipe(prompt=prompt,
negative_prompt=negative_prompt,
width=512,
height=512,
num_inference_steps=num_inference_steps,
guidance_scale=guidance_scale).images[0]
image.save("a girl.png")
`

Logs

No response

System Info

• diffusers version: 0.24.0
• Platform: Linux-4.14.0_1-0-0-45-x86_64-with-glibc2.18
• Python version: 3.10.13
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Huggingface_hub version: 0.20.1
• Transformers version: 4.30.2
• Accelerate version: 0.21.0
• xFormers version: 0.0.20

Who can help?

@yiyixuxu

[tolgacangoz]

Hi @ApolloRay!
When I removed the use_karras_sigmas=True, algorithm_type="dpmsolver++" parts the result was promising.

[ApolloRay]

When I removed the use_karras_sigmas=True, algorithm_type="dpmsolver++" parts the result was promising.

OH，that's right. Thank you!

[asomoza]

To be fair, the model owner says to use DPM++ SDE Karras.

So you should use:

pipe.scheduler = DPMSolverSinglestepScheduler.from_config(
    pipe.scheduler.config,
    use_karras_sigmas=True,
)

Don't really know why using DPMSolverMultistepScheduler with karras generates bad results, it happens with ComfyUI too.

[sayakpaul]

Can we close this issue then?

Thanks @StandardAI for your help here.

[ApolloRay]

Thanks