DiffEngine

📘 Documentation | 🤔 Reporting Issues

📄 Table of Contents

• DiffEngine
  • 📄 Table of Contents
  • 📖 Introduction 🔝
  • 🛠️ Installation 🔝
  • 👨‍🏫 Get Started 🔝
  • 🖋 Example Notebook 🔝
  • 📘 Documentation 🔝
  • 📊 Model Zoo 🔝
  • 🙌 Contributing 🔝
  • 🎫 License 🔝
  • 🖊️ Citation 🔝
  • 💻 Sponsors
  • 🤝 Acknowledgement 🔝

📖 Introduction 🔝

DiffEngine is the open-source toolbox for training state-of-the-art Diffusion Models. Packed with advanced features including diffusers and MMEngine, DiffEngine empowers both seasoned experts and newcomers in the field to efficiently create and enhance diffusion models. Stay at the forefront of innovation with our cutting-edge platform, accelerating your journey in Diffusion Models training.

1. Training state-of-the-art Diffusion Models: Empower your projects with state-of-the-art Diffusion Models. We can use Stable Diffusion, Stable Diffusion XL, DreamBooth, LoRA etc.
2. Unified Config System and Module Designs: Thanks to MMEngine, our platform boasts a unified configuration system and modular designs that streamline your workflow. Easily customize hyperparameters, loss functions, and other crucial settings while maintaining a structured and organized project environment.
3. Inference with diffusers.pipeline: Seamlessly transition from training to real-world application using the diffusers.pipeline module. Effortlessly deploy your trained Diffusion Models for inference tasks, enabling quick and efficient decision-making based on the insights derived from your models.

🛠️ Installation 🔝

Before installing DiffEngine, please ensure that PyTorch >= v2.0 has been successfully installed following the official guide.

Install DiffEngine

pip install git+https://github.com/okotaku/diffengine.git

👨‍🏫 Get Started 🔝

DiffEngine makes training easy through its pre-defined configs. These configs provide a streamlined way to start your training process. Here's how you can get started using one of the pre-defined configs:

1. Choose a config: You can find various pre-defined configs in the configs directory of the DiffEngine repository. For example, if you wish to train a DreamBooth model using the Stable Diffusion algorithm, you can use the configs/stable_diffusion_dreambooth/stable_diffusion_v15_dreambooth_lora_dog.py.

2. Start Training: Open a terminal and run the following command to start training with the selected config:

diffengine train stable_diffusion_v15_dreambooth_lora_dog

3. Monitor Progress and get results: The training process will begin, and you can track its progress. The outputs of the training will be located in the work_dirs/stable_diffusion_v15_dreambooth_lora_dog directory, specifically when using the stable_diffusion_v15_dreambooth_lora_dog config.

work_dirs/stable_diffusion_v15_dreambooth_lora_dog
├── 20230802_033741
|   ├── 20230802_033741.log  # log file
|   └── vis_data
|         ├── 20230802_033741.json  # log json file
|         ├── config.py  # config file for each experiment
|         └── vis_image  # visualized image from each step
├── step999/unet
|   ├── adapter_config.json  # adapter conrfig file
|   └── adapter_model.bin  # weight for inferencing with diffusers.pipeline
├── iter_1000.pth  # checkpoint from each step
├── last_checkpoint  # last checkpoint, it can be used for resuming
└── stable_diffusion_v15_dreambooth_lora_dog.py  # latest config file

An illustrative output example is provided below:

4. Inference with diffusers.pipeline: Once you have trained a model, simply specify the path to the saved model and inference by the diffusers.pipeline module.

from pathlib import Path

import torch
from diffusers import DiffusionPipeline
from peft import PeftModel

checkpoint = Path('work_dirs/stable_diffusion_v15_dreambooth_lora_dog/step999')
prompt = 'A photo of sks dog in a bucket'

pipe = DiffusionPipeline.from_pretrained(
    'runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16)
pipe.to('cuda')
pipe.unet = PeftModel.from_pretrained(pipe.unet, checkpoint / "unet", adapter_name="default")
if (checkpoint / "text_encoder").exists():
    pipe.text_encoder = PeftModel.from_pretrained(
        pipe.text_encoder, checkpoint / "text_encoder", adapter_name="default"
    )

image = pipe(
    prompt,
    num_inference_steps=50
).images[0]
image.save('demo.png')

🖋 Example Notebook 🔝

For a more hands-on introduction to DiffEngine, you can run the Example Notebook on Colaboratory. This notebook demonstrates the process of training using SDV1.5 and SDV2.1 DreamBooth configurations.

📘 Documentation 🔝

For detailed user guides and advanced guides, please refer to our Documentation:

• Get Started for get started.

Run Guides

• Run Stable Diffusion
• Run Stable Diffusion XL
• Run Stable Diffusion DreamBooth
• Run Stable Diffusion XL DreamBooth
• Run Stable Diffusion LoRA
• Run Stable Diffusion XL LoRA
• Run Stable Diffusion ControlNet
• Run Stable Diffusion XL ControlNet
• Run IP Adapter
• Run T2I Adapter
• Run InstructPix2Pix
• Run Wuerstchen
• Run Wuerstchen LoRA
• Run LCM XL
• Run LCM XL LoRA
• Run PixArt-α
• Run PixArt-α LoRA
• Run PixArt-α DreamBooth
• • Run Kandinsky 2.2
• Inference

User Guides

• Learn About Config
• Prepare Dataset

Blog Posts

• Introduction to DiffEngine
• Train ControlNet with DiffEngine
• On Architectural Compression of Text-to-Image Diffusion Models
• SSD-1B: A Leap in Efficient T2I Generation

📊 Model Zoo 🔝

Supported algorithms

Stable Diffusions	Stable Diffusion XLs	DeepFloyd IFs	Others
Stable Diffusion (2022) ControlNet (ICCV'2023) DreamBooth (CVPR'2023) LoRA (ICLR'2022) Distill SD DreamBooth (2023) Inpaint	Stable Diffusion XL (2023) ControlNet (ICCV'2023) DreamBooth (CVPR'2023) LoRA (ICLR'2022) ControlNet Small (2023) T2I-Adapter (2023) IP-Adapter (2023) Erasing Concepts from Diffusion Models (2023) SSD-1B (2023) InstructPix2Pix (2022) LoHa (ICLR'2022) LoKr (2022) OFT (NeurIPS'2023) ControlNet-XS (2023) Inpaint	DeepFloyd IF (2023) DreamBooth (CVPR'2023)	Min-SNR Loss (ICCV'2023) DeBias Estimation Loss (2023) Offset Noise (2023) Pyramid Noise (2023) Input Perturbation (2023) Time Steps Bias (2023) V Prediction (ICLR'2022) Diffusion DPO (2023)
Wuerstchen	Latent Consistency Models	PixArt-α	Kandinsky
Wuerstchen (2023) LoRA (ICLR'2022)	Latent Consistency Models (2023) LoRA (ICLR'2022)	PixArt-α (2023) LoRA (ICLR'2022) DreamBooth (CVPR'2023)	Kandinsky 2.2 (2023) Kandinsky 3 (2023)
aMUSEd
aMUSEd (2024)

🙌 Contributing 🔝

We appreciate all contributions to improve clshub. Please refer to CONTRIBUTING.md for the contributing guideline.

🎫 License 🔝

This project is released under the Apache 2.0 license.

🖊️ Citation 🔝

If DiffEngine is helpful to your research, please cite it as below.

@misc{diffengine2023,
    title = {{DiffEngine}: diffusers training toolbox with mmengine},
    author = {{DiffEngine Contributors}},
    howpublished = {\url{https://github.com/okotaku/diffengine}},
    year = {2023}
}

💻 Sponsors

takuoko is a member of Z by HP Data Science Global Ambassadors. Special Thanks to Z by HP for sponsoring me a Z8G4 Workstation with dual A6000 GPU and a ZBook with RTX5000 GPU.

🤝 Acknowledgement 🔝

This repo borrows the architecture design and part of the code from mmengine, mmagic and diffusers.

Also, please check the following openmmlab and huggingface projects and the corresponding Documentation.

• OpenMMLab
• HuggingFace

@article{mmengine2022,
  title   = {{MMEngine}: OpenMMLab Foundational Library for Training Deep Learning Models},
  author  = {MMEngine Contributors},
  howpublished = {\url{https://github.com/open-mmlab/mmengine}},
  year={2022}
}

@misc{mmagic2023,
    title = {{MMagic}: {OpenMMLab} Multimodal Advanced, Generative, and Intelligent Creation Toolbox},
    author = {{MMagic Contributors}},
    howpublished = {\url{https://github.com/open-mmlab/mmagic}},
    year = {2023}
}

@misc{von-platen-etal-2022-diffusers,
  author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},
  title = {Diffusers: State-of-the-art diffusion models},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/diffusers}}
}

@Misc{peft,
  title =        {PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods},
  author =       {Sourab Mangrulkar and Sylvain Gugger and Lysandre Debut and Younes Belkada and Sayak Paul and Benjamin Bossan},
  howpublished = {\url{https://github.com/huggingface/peft}},
  year =         {2022}
}