Face Embedding Association Test (FEAT)

This repository contains a data and code for the paper Measuring Embedded Human-like Biases in Face Recognition Models [paper], published at AAAI2022 Workshop on Artificial Intelligence with Biased or Scarce Data. We introduced Face Embedding Association Test (FEAT), which is an extension of Word Embedding Association Test for face recognition models.

Data

Our dataset is in data folder. We employed dataset from this repo and additionally collected dataset to test a wide range of social biases by using google search. We only uploaded the dataset that we newly crawled. You can use crawl.py to crawl additional data. To get google image dataset, we use google_images_download package. Please install the required package.

pip install google_images_download

Below is the data directory structure.

/data/attributes  
 ┬  
 ├ [DIR] race 
     └ [DIR] 8 asian attributes
 ├ [DIR] age  
     └ [DIR] 8 attributes
         ├ young
         └ old
 ├ [DIR] intersectional 
     ├ [DIR] competent 
         ├ asian_female
         ├ black_female
         └ white_female
     └ [DIR] incompetent 
         ├ asian_female
         ├ black_female
         └ white_female

Experimental Setup

• Python == 3.6
• TensorFlow == 2.4.1
• Keras == 2.4.0
• dlib
• cv2
• pillow
• skin

Download pre-trained models

Download the pretrained models for:

• VGGFace

For VGGFace, you can use keras-vggface package. Please install the required package.

pip install keras-vggface

You can update the weights of the models by using links below:

• FaceNet
• Deepface
• DeepID
• Openface
• Arcface

Experiment 1: FEAT on social biases (race, gender, age, intersectional)

You can run run_test.py function to run FEAT test on openface, arcface, vggface, deepface, facenet, and deepid models.

python test/run_test.py

Experiment 2: Race sensitivity analysis

You can run change_total(image_dir, output_dir, transformation_level) function in race_transformation/black_to_white.py to convert race black to white. And the same function in race_transformation/white_to_black.py to convert race white to black.

Reference

@inproceedings{lee2022measuring,
  title={Measuring Embedded Human-Like Biases in Face Recognition Models},
  author={Lee, SangEun and Oh, Soyoung and Kim, Minji and Park, Eunil},
  booktitle={Computer Sciences and Mathematics Forum},
  volume={3},
  number={1},
  pages={2},
  year={2022},
  organization={MDPI}
}