Implement Deepspeed with the DiffusionPipeline

[vanakema]

Is your feature request related to a problem? Please describe.
Trying to train Stable Diffusion on your local box using the Dreambooth method causes out-of-memory issues on an RTX 3090. I'd like to be able to train such a model without having to have an A100 80GB

Describe the solution you'd like
Implement Deepspeed integration with the DiffusionPipeline class

Describe alternatives you've considered

• Reducing batch size (even BS=1 causes issues)
• Reducing resolution: haven't had much success here, but I could be doing something wrong. I am still learning

[anton-l]

This would be great @vanakema! While the core maintainers of diffusers don't have enough bandwidth to implement support for DeepSpeed yet, anyone who comes across this issue is welcome to open a PR with a DS integration, just let us know! 🤗

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

BTW, I think we should not "natively" implement DeepSpeed to diffusers, but rather make sure diffusers + deepspeed + accelerate works. This means we should rather help accelerate to have an easy deepspeed integration that can be used with diffusers.

[vanakema]

Yeah sorry I guess I completely left that out. I made this ticket after realizing that for the HF accelerate deepspeed wrapper, you can only provide 1 model at a time, and the wrapper model object won't be picked up at all, so there's nothing to call backward on (unfortunately I'm going off memory. I can take a look at my notes later and clarify if this doesn't make sense)

I commented on an issue on the accelerate repo, and they said they ditched the idea of supporting training multiple models at the same time which would preclude training SD using the dreambooth technique (I believe. Correct me if I'm wrong, I am pretty new to this)

I guess it might be possible to train each necessary network sequentially? But it logically seems like training both CLIP and the diffusion network at the same time would be necessary.

Luckily this ticket isn't necessary for me anymore since this repo now provides Dreambooth examples that work on smaller GPUs!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[0xdevalias]

I commented on an issue on the accelerate repo, and they said they ditched the idea of supporting training multiple models at the same time which would preclude training SD using the dreambooth technique (I believe. Correct me if I'm wrong, I am pretty new to this)

While I don't know the deeper specifics, the dreambooth examples in this repo seem to make use of accelerate, so hopefully that means it does work for training multi models?

[vanakema]

I commented on an issue on the accelerate repo, and they said they ditched the idea of supporting training multiple models at the same time which would preclude training SD using the dreambooth technique (I believe. Correct me if I'm wrong, I am pretty new to this)

While I don't know the deeper specifics, the dreambooth examples in this repo seem to make use of accelerate, so hopefully that means it does work for training multi models?

Accelerate does overall I believe, but the DeepSpeed wrapper implemented does not

huggingface/accelerate#253 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[liuxz-cs]

I commented on an issue on the accelerate repo, and they said they ditched the idea of supporting training multiple models at the same time which would preclude training SD using the dreambooth technique (I believe. Correct me if I'm wrong, I am pretty new to this)

While I don't know the deeper specifics, the dreambooth examples in this repo seem to make use of accelerate, so hopefully that means it does work for training multi models?

Accelerate does overall I believe, but the DeepSpeed wrapper implemented does not

huggingface/accelerate#253 (comment)

So how do you solve your problem? It seems that till now the deepspeed still cannot support more than 1 models with using accelerate.