trustworthy-asv-fairness

Improving fairness of speaker representations using adversarial and multi-task learning methods

Paper (Published in Elsevier Computer Speech and Language): https://www.sciencedirect.com/science/article/abs/pii/S0885230822001048

Adversarial training ideas were adopted and modified from Jaiswal, A., Wu, Y., AbdAlmageed, W., & Natarajan, P. (2019). Unified adversarial invariance. arXiv preprint arXiv:1905.03629.

Highlights

• Systematic evaluation of biases with respect to gender in speaker verification systems at multiple operating points.
• Amount of fairness achieved through training data balancing depends on operating region of speaker verification system.
• Adversarial and multi-task training based embedding transformation techniques improve the fairness of existing speaker verification systems.
• Utility is an important consideration in choosing appropriate bias mitigation strategies. Multi-task technique improves fairness and retains utility, adversarial technique improves fairness at the cost of reduced utility.

Shared data, models

1. Pre-trained models for each of the methods described in paper: https://drive.google.com/drive/folders/1m_mv_klf3ZFAREuv0gct1SPV8KftxyTm?usp=sharing
2. Baseline embeddings (from FairVoice [2]) for training and evaluation: https://drive.google.com/drive/folders/1Yb6GEultj4ig1kVb3h19Z7Mzz_UHU6Po?usp=sharing
3. Transformed embeddings (extracted using above pre-trained models) on eval-dev, eval-test and voxceleb1_h datasets: https://drive.google.com/drive/folders/1HIB0Z7fEMFjOBXg_DrcXAZqUvuoDTyJl?usp=sharing
4. Verification trials based on Mozilla CommonVoice (MCV) and Voxceleb1-H audio: https://drive.google.com/drive/folders/1DJaGfuG6DaAFaQyICfE22rTR9aIhuif4?usp=sharing

Conda environment

Will create a conda environment named uai_36 with required packages. Needed for subsequent steps

conda env create -f conda_env.yml

Steps to run fairness evaluations

1. Download the transformed embeddings and trials from the above shared links, and place them in local directory of choice (<data_dir>)
2. Evaluate models with fairness metrics (Fairness Discrepancy Rate: [1])

bash run_compute_auFDR.sh <data_dir> <test_split> <method>

<data_dir> is the chosen directory to save downloaded embeddings and trials files

<test_split> is one of "eval-dev", "eval-test" or "voxceleb1_h".

<method> is one of "AT", "MTL", "NLDR", "UAI", "UAI-AT" or "UAI-MTL"

For example, to run evaluations for "eval-dev" set and for "UAI-MTL" method

bash run_compute_auFDR.sh <data_dir> eval-dev UAI-MTL

Steps to transform the baseline embeddings (Pre-transformed embeddings are already provided in the above links)

1. Download the baseline embeddings (FairVoice) and saved models from the above shared links, and place them in local directory of choice (<data_dir>)
2. Transform the baseline embeddings

bash run_transform.sh <data_dir> <test_split> <method>

<data_dir> is the chosen directory to save downloaded embeddings and trials files

<test_split> is one of "eval-dev", "eval-test" or "voxceleb1_h".

<method> is one of "AT", "MTL", "NLDR", "UAI", "UAI-AT" or "UAI-MTL"

For example, to transform embeddings for "eval-dev" set and for "UAI-MTL" method

bash run_transform.sh <data_dir> eval-dev UAI-MTL

Methods

UAI

UAI-AT AND UAI-MTL

Results

References

[1] T. de Freitas Pereira, S. Marcel, Fairness in biometrics: a figure of merit to assess biometric verification systems

[2] G. Fenu, H. Lafhouli, M. Marras, Exploring algorithmic fairness in deep speaker verification, in: International Conference on Computational Science and Its Applications, Springer, 2020, pp. 77–93.