[Feature Request]: colossalai integration

[jeanhadrien]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Integrate ColossalAI improvements : https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion

Proposed workflow

User is able to toggle improvements by colossal AI

Additional information

No response

[Ratinod]

I also want dreambooth with 8 Gb VRAM...
Probably worth duplicating the Feature Request here:
Dreambooth Extension for Stable-Diffusion-WebUI

[sosawolfo]

Integration with Colossal-AI would be amazing. The speed and VRAM improvements seem to be well worthwhile.

[0xdevalias]

• https://github.com/hpcaitech/ColossalAI

  • Colossal-AI: A Unified Deep Learning System for Big Model Era

• https://medium.com/@yangyou_berkeley/diffusion-pretraining-and-hardware-fine-tuning-can-be-almost-7x-cheaper-85e970fe207b

---

Probably worth duplicating the Feature Request here:
Dreambooth Extension for Stable-Diffusion-WebUI

Done:

• Community Integration: Making AIGC cheaper, faster, and more efficient. huggingface/diffusers#1212
• Integrate with ColossalAI for memory/speed efficiency and improvements d8ahazard/sd_dreambooth_extension#174
• [FEATURE]: Integrate StableDiffusion Dreambooth with ColossalAI hpcaitech/ColossalAI#1935
• Integrate StableDiffusion Dreambooth with ColossalAI TheLastBen/fast-stable-diffusion#493
• Integrate StableDiffusion Dreambooth with ColossalAI ShivamShrirao/diffusers#129

---

Also, here are a few other alternative speed/efficiency/etc increase potential issues i'm aware of, each using different techniques/etc:

• Stable Diffusion Meta AITemplate with >= 200% performance increase #1625
• [Feature Request]: Explore potential StableDiffusion speed benefits from implementing kernl (Up to 12X faster GPU inference) #4096
• [Feature Request]: Explore NVIDIA/TransformerEngine for speed/efficiency #4721

[0xdevalias]

It looks like the dev team of ColossalAI is looking to work directly with huggingface diffusers to simplify integrations in this issue:

• Community Integration: Making AIGC cheaper, faster, and more efficient. huggingface/diffusers#1212

This is very cool @binmakeswell ! Would be very happy to integrate diffusers in colossalAI examples!

Originally posted by @patil-suraj in huggingface/diffusers#1212 (comment)

Thanks for your enthusiastic response and we look forward to working together!

How do you expect it to be carried out?
Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.

Originally posted by @binmakeswell in huggingface/diffusers#1212 (comment)

And there seems to be at least some initial/cursory interest on the issue I opened on the ColossalAI repo RE: integrating with dreambooth:

• [FEATURE]: Integrate StableDiffusion Dreambooth with ColossalAI hpcaitech/ColossalAI#1935

Hi, thanks for reaching out! I think it’s pretty awesome. Can I check it with our team and get back to you ASAP?

Originally posted by @Sze-qq in hpcaitech/ColossalAI#1935 (comment)

So hopefully things rapidly move towards the 'easier to integrate'/'better documentation' direction of things!

Originally posted by @0xdevalias in ShivamShrirao/diffusers#129 (comment)

[Kuri-su]

any progress?

[binmakeswell]

Sorry for the late reply. The Colossal-AI team is busy with open source ChatGPT recently and received a lot of cooperation interest.

We are expected to have manpower to assist in this work in March. We also look forward to and welcome the participation of the open source community. Thanks.