Skip to content

sail-sg/finetune-fair-diffusion

Repository files navigation

Finetuning Text-to-Image Diffusion Models for Fairness

Code and checkpoints for the paper Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024, oral presentation).

Images generated by the original SD Images generated by the debiased SD

Prompt: "A photo of the face of a electrical and electronics repairer, a person". Left images are generated by the original stable-diffusion-v1-5. Right images are generated by the finetuned stable diffusion, debiased for gender and race jointly. For every image, the first color-coded bar denotes the predicted gender: male (blue) or female (red). The second denotes race: WMELH (green), Asian (orange), Black (black), or Indian (brown). WMELH is the broad race class of White, Middle Eastern, and Latino Hispanic. Bar height represents prediction confidence. Bounding boxes denote detected faces. Images with the same number label are generated using the same noise.

Prerequisite:

This project has been tested under the following environment:

  • OS: Red Hat Enterprise Linux 8.4
  • GPU: NVIDIA A100-SXM4-40GB ($\times$ 2, or more)
  • CUDA Version: 12.2
  1. We manage environments with Conda. To set up the environment, follow these steps:
conda env create -f environment.yml
conda activate fair-diffusion
  1. To set up wandb for logging, run:
wandb login
  1. Download data.zip from this link to the working directory, unzip it by:
unzip data.zip

This will create a data folder in the current working directory.

Experiments:

This repository includes the code and checkpoints necessary to replicate the five experiments detailed in our paper. To find more information on a specific experiment, please click on the experiment's name, which will direct you to the corresponding folder.

Experiment Name Prompts Description
exp-1-debias-gender Occupations Finetune LoRA on text encoder and/or U-Net to debias binary gender to a perfectly balanced distribtion.
exp-2-debias-gender-token Occupations Finetune prompt prefix to debias binary gender to a perfectly balanced distribtion.
exp-3-debias-gender-race Occupations Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion.
exp-4-debias-gender-race-age Occupations Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. Simultaneouly, control age to 75% yound and 25% old distribution.
exp-5-debias-gender-race-multi-concepts (1) Occupations, (2) occupations with style and context, (3) personal descriptors, and (4) sports Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion.
exp-6-debias-race Occupations Finetune LoRA on text encoder and/or U-Net to debias four classes of race, to a perfectly balanced distribtion.

Fair diffusion model adaptors

If you're primarily interested in utilizing and evaluating our debiased Stable Diffusion v1-5, please follow below instructions.

Experiment Name Checkpoints Instructions on how to use
exp-1-debias-gender link unzip to exp-1-debias-gender/outputs, follow instructions starting from this point onwards.
exp-2-debias-gender-token link unzip to exp-2-debias-gender-token/outputs, follow instructions starting from this point onwards.
exp-3-debias-gender-race link unzip to exp-3-debias-gender-race/outputs, follow instructions starting from this point onwards.
exp-4-debias-gender-race-age link unzip to exp-4-debias-gender-race-age/outputs, follow instructions starting from this point onwards.
exp-5-debias-gender-race-multi-concepts link unzip to exp-5-debias-gender-race-multi-concepts/outputs, follow instructions starting from this point onwards.
exp-6-debias-race link unzip to exp-6-debias-race/outputs, follow instructions starting from this point onwards.

Get in touch

We sincerely appreciate your interest in our work. Should you have any questions, please don't hesitate to reach out to Xudong at xudong.shen@u.nus.edu.

Citation

@inproceedings{shen2024finetuning,
      title={Finetuning Text-to-Image Diffusion Models for Fairness}, 
      author={Xudong Shen and Chao Du and Tianyu Pang and Min Lin and Yongkang Wong and Mohan Kankanhalli},
      booktitle={The Twelfth International Conference on Learning Representations},
      year={2024},
      url={https://openreview.net/forum?id=hnrB5YHoYu}
}

Releases

No releases published

Packages

No packages published