Leverage deepspeed for significantly faster inference

[LifeIsStrange]

https://github.com/microsoft/DeepSpeed
Deepspeed is a state of the art library that enable out of the box many optimizations for inference.
While especially good for clusters, I believe it can bring optimizations to single GPUs too.

[goldfish1974]

I believe DeepSpeed is supported by HuggingFace Accelerate. (https://huggingface.co/docs/accelerate/usage_guides/fsdp and https://huggingface.co/docs/accelerate/usage_guides/deepspeed)

The same docs also talks about how you can shard the training and inference so that you can use a) less GPU resources and/or b) Multiple GPUs.

This would open up a lot more people to be able to perform training which appears to be a limiting factor as the lowest GPU VRAM I've seen training on is around 10GB (Down from 24GB).

I feel that a lot of value would be added by reducing the VRAM requirements (offset against training speed) out in the Stable Diffusion public community.

Please consider this issue as it seems the implementation should be fairly simple as it can apparently be implemented with only a few lines of code changes and a matching configuration file.