Add the SDE variant of DPM-Solver and DPM-Solver++ to DPM Single Step

[cmdr2]

Following up on our conversation at #4167 (comment)

Uses the same approach as https://github.com/huggingface/diffusers/pull/3344/files

Note: This PR is not ready to merge yet, since it's not clear whether to use sigma_s0 or sigma_s1 in the second-order update. There is a discrepancy between the implementations of single-step and multi-step for second-order.

The current Single Step implementation uses sigma_s1 for the second-order update:

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_singlestep.py

Line 503 in 45f6d52

(sigma_t / sigma_s1) * sample

While the reference PR for Multi Step uses sigma_s0 for the second-order update: https://github.com/huggingface/diffusers/pull/3344/files#diff-517cce3913a4b16e1d17a0b945a920e400aa5553471df6cd85f71fc8f079b4b4R478

@patrickvonplaten I don't know which is the correct one. Is it a bug in multi-step, or single-step, or should I use sigma_s1 for single-step?

The test doesn't pass right now due to this error, I intentionally left the code in this state.

Thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

Gently pinging the OG DPM author @LuChengTHU

Do you have an idea here by any chance? :-)

[LuChengTHU]

Hi all, just see this cool PR! Thank you for the effort! The single-step sde-dpm++ may be a bit tricky and I need to carefully check this PR. Please give me some more time :)

[cmdr2]

Thanks @LuChengTHU ! :)

The second-order update confuses me - in DPM Single Step it divides by the second-last timestep, while in DPM Multi Step it divides by the last timestep. Not sure if that's intentional.

Single Step:

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_singlestep.py

Line 491 in 45f6d52

t, s0, s1 = prev_timestep, timestep_list[-1], timestep_list[-2]

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_singlestep.py

Line 503 in 45f6d52

(sigma_t / sigma_s1) * sample

Multi Step:

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

Line 508 in 45f6d52

t, s0, s1 = prev_timestep, timestep_list[-1], timestep_list[-2]

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

Line 520 in 45f6d52

(sigma_t / sigma_s0) * sample

Thanks for your help!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[LuChengTHU]

Sorry for the delay. will check it soon

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

This PR is actually an important one - @yiyixuxu can you try taking this over?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[cmdr2]

Not stale

[patrickvonplaten]

Gentle ping here @yiyixuxu

[patrickvonplaten]

Gentle ping again @yiyixuxu

[patrickvonplaten]

Gentle ping @yiyixuxu

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

It has been on my to-do list for a while, but I don't have time for it right now. If anyone in the community wants to take a shot, feel free! This is an important feature to add.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Scorpinaus]

Don't think this is stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[tolgacangoz]

I want to work on continuing this PR if it is OK.

[yiyixuxu]

@standardAI please!!