SBFT: Search-Based Fairness Testing

SBFT is a fairness testing tool for regression-based AI software. It estimates the fairness degree (D) of the system under test. Fariness degree is defined as follows.

Given an AI software system, the fairness degree is measured by the maximum difference in the predicted values by the AI software for all pairs of instances (x_i,x_j) that are identical apart from the sensitive attribute, i.e., x_i^s ≠ x_j^s.

D = max_∀i, j|y_i−y_j|; x_i^s ≠ x_j^s & x_i^k = x_j^k; ∀k ≠ s

where y_i and y_j are the outputs for inputs x_i and x_i.

How to use SBFT?

Parameter Settings

All the parameters are defined in configs.yml

Parameter	Description	Default Value
time_budget	Execution time (seconds)	7200
proportion_test_insertion	Rate of random test insertion in every iteration	0.1
p_crossover	Crossover probability	0.75
p_mutation	Mutation probability	0.8
population_size	Population size	100
p_cache	Probability of using cache in fitness evaluation	0.5
max_generations	Maximum number of generations	1000
crossover_type	Crossover type	uniform
mutation_type	Mutation type	uniform
parent_selection_type	Parent selection type	roulette_wheel
protected_features	Sensitive attributes, i.e., protected features	-
model_filepath	Filepath to the model	-
variable_boundaries_filepath	Filepath to the variable boundaries file	-
categorical_variables_dir	Path to the directory containing categorical variables	-
primary_logger_type	Primary logger type	console

Steps to Run SBFT

Currently, SBFT can be run by executing the source code.

1. Clone SBFT:

   • git clone https://github.com/search-based-fairness-testing/sbft.git

2. Configure the parameters listed above and locate the configs.yml file in the workspace directory.

3. Run SBFT

   • cd sbft
   • python3 -u ft/sbft.py <path to workspace dir>

Publications

If you use the SBFT tool, please cite:

@article{perera2022search,
  title={Search-based fairness testing for regression-based machine learning systems},
  author={Perera, Anjana and Aleti, Aldeida and Tantithamthavorn, Chakkrit and Jiarpakdee, Jirayus and Turhan, Burak and Kuhn, Lisa and Walker, Katie},
  journal={Empirical Software Engineering},
  volume={27},
  number={3},
  pages={1--36},
  year={2022},
  publisher={Springer}
}

More Information

• Contact Anjana Perera