Non-confusing Generation of Customized Concepts in Diffusion Models

This repository contains the implementation of the paper:

Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 24)

   

Wang Lin^1,*, Jingyuan Chen¹, Jiaxin Shi^4,*, Yichen Zhu¹, Chen Liang⁵,
Junzhong Miao⁶, Tao Jin¹, Zhou Zhao¹ Fei Wu¹ Shuicheng Yan² Hanwang Zhang^2,3 (* Equal Contribution)

¹ Zhejiang University     ² Skywork AI，Singapore     ³ Nanyang Technological University     ⁴ Huawei Cloud Computing     ⁵ Tsinghua University     ⁶ Harbin Institute of Technology

Dependencies and Installation

conda create -n clif python=3.9
pip install diffusers==0.23.1
conda activate clif

Training

Step 1:

We first fine-tuning the customized concepts with contrastive learning.

bash run_train_clif.sh

Step 2:

We then perform text inversion on customized concepts to encode visual details into token embeddings

bash run_train_ti.sh

Step 3:

Finally we train lora and token embeddings together

bash run_train_lora.sh

Evaluation

The evaluation of our method are based on two metrics: text-alignment and image-alignment following Custom Diffusion.

The prompts used in our quantitative evaluations can be found in dataset.

Acknowledgements

This code is builds on the code from the diffusers library

BibTex

@InProceedings{pmlr-v235-lin24d,
  title = 	 {Non-confusing Generation of Customized Concepts in Diffusion Models},
  author =       {Lin, Wang and Chen, Jingyuan and Shi, Jiaxin and Zhu, Yichen and Liang, Chen and Miao, Junzhong and Jin, Tao and Zhao, Zhou and Wu, Fei and Yan, Shuicheng and Zhang, Hanwang},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  pages = 	 {29935--29948},
  year = 	 {2024},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {21--27 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://raw.githubusercontent.com/mlresearch/v235/main/assets/lin24d/lin24d.pdf},
  url = 	 {https://proceedings.mlr.press/v235/lin24d.html},
}