[Feature Request]: Stabilize the sampling of DPM-Solver++ by a stabilizing trick

[LuChengTHU]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Thank you for the support for DPM-Solver++! The examples in #4304 are so amazing!

However, the current implementations for DPM-Solver++ suffer from instability issues for steps <= 10. It can be further improved by a stabilizing trick. Please see this issue for details: crowsonkb/k-diffusion#43

Feature request: Stabilize the sampling of DPM-Solver++ by a stabilizing trick, which can further stabilize the sampling by DPM-Solver++ for steps <= 10

Proposed workflow

Support the stabilizing trick

Additional information

No response

[aliencaocao]

+1
Observed the same issue when using it, kind of pity because the main advantage with DPM solver is the low steps required for a good image.
Just to add (for those that did not know) this guy is the first author of the paper that proposed DPM solver++ so whatever he is saying definitely has some credibility there.

[LuChengTHU]

+1 Observed the same issue when using it, kind of pity because the main advantage with DPM solver is the low steps required for a good image. Just to add (for those that did not know) this guy is the first author of the paper that proposed DPM solver++ so whatever he is saying definitely has some credibility there.

Haha, thank you! Sorry to forget to introduce myself. I'm the first author of both DPM-Solver and DPM-Solver++. Thanks to the community, our proposed solvers are widely used in many applications of diffusion models.

In my recent experience, the "stabilizing" trick I mentioned here is a key to stabilizing DPM-Solver++2M, and I have implemented it in my PR to diffusers: huggingface/diffusers#1132. I believe the proposed DPM-Solver++2M will be the next-generation trend for accelerating the sampling by diffusion models!

[LuChengTHU]

An example of the stabilizing trick:

https://github.com/LuChengTHU/dpm-solver/blob/414c74f62fb189723461aadc91dc6527301e1dbe/dpm_solver_pytorch.py#L1094-L1098

[hentailord85ez]

Looking at k-diffusion's current implementation of 2M and if I understand correctly, it seems like it is already doing this.

I've modified the inner loop of the 2M implementation to print whether it's doing order 1 or order 2:

        if old_denoised is None or sigmas[i + 1] == 0:
            order_1...
            print("O1")
        else:
            order_2...
            print("O2")

With S = 5, it does 1 step of order 1, 3 steps of order 2, and then 1 final step of order 1.

[ClashSAN]

@hentailord85ez so, these improvements are already in?

for

DPM++ 2M
DPM++ 2M Karras
DPM++ 2S
DPM++ 2S a Karras