FairerCLIP: Debiasing CLIP’s Zero-Shot Predictions using Functions in RKHSs

Welcome to the official repository for the paper, FairerCLIP: Debiasing CLIP’s Zero-Shot Predictions using Functions in RKHSs, published in the Twelfth International Conference on Learning Representations (ICLR'24).

Authors: Sepehr Dehdashtian*, Lan Wang* and Vishnu Naresh Boddeti

Abstract

Large pre-trained vision-language models such as CLIP provide compact and general-purpose representations of text and images that are demonstrably effective across multiple downstream zero-shot prediction tasks. However, owing to the nature of their training process, these models have the potential to 1) propagate or amplify societal biases in the training data and 2) learn to rely on spurious features. This paper proposes FairerCLIP, a general approach for making zero-shot predictions of CLIP more fair and robust to spurious correlations.

We formulate the problem of jointly debiasing CLIP’s image and text representations in reproducing kernel Hilbert spaces (RKHSs), which affords multiple benefits:

1. Flexibility: Unlike existing approaches, which are specialized to either learn with or without ground-truth labels, FairerCLIP is adaptable to learning in both scenarios.
2. Ease of Optimization: FairerCLIP lends itself to an iterative optimization involving closed-form solvers, which leads to 4×-10× faster training than the existing methods.
3. Sample Efficiency: Under sample-limited conditions, FairerCLIP significantly outperforms baselines when they fail entirely.
4. Performance: Empirically, FairerCLIP achieves appreciable accuracy gains on benchmark fairness and spurious correlation datasets over their respective baselines.

How to use

git clone https://github.com/human-analysis/FairerCLIP.git
cd FairerCLIP

Requirements

A requirements.txt file has been provided.

1. Download the pre-extracted features

Create a folder named features and download the features and labels Google Drive.

mkdir features

2. Mitigating spurious correlation (w/ labels)

# Waterbirds Dataset + CLIP ViT-L/14
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_waterbird.yaml

# Waterbirds Dataset + CLIP ResNet-50
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_waterbird_RN50.yaml

# CelebA Dataset + CLIP ViT-L/14
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_celeba.yaml

# CelebA Dataset + CLIP ResNet-50
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_celeba_RN50.yaml

3. Mitigating spurious correlation (w/o labels)

# Waterbirds Dataset + CLIP ViT-L/14
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_waterbird.yaml --opt nolabel True

# Waterbirds Dataset + CLIP ResNet-50
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_waterbird_RN50.yaml --opt nolabel True

# CelebA Dataset + CLIP ViT-L/14
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_celeba.yaml --opt nolabel True

# CelebA Dataset + CLIP ResNet-50
CUDA_VISIBLE_DEVICES=0 python train.py --config-file configs/debias_celeba_RN50.yaml --opt nolabel True

4. Mitigating Intrinsic Dependency

CUDA_VISIBLE_DEVICES=0 python train_eo.py --config-file configs/debias_celeba_highcheek.yaml

Citation

If you find this paper and repository useful for your research, please consider citing the paper:

@inproceedings{dehdashtian2024fairerclip,
  title={FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs},
  author={Dehdashtian, Sepehr and Wang, Lan and Boddeti, Vishnu Naresh},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2024}
}

Acknowledgements

The code of model evaluation is primarily inspired by the supplement of Zhang and Ré.