Code and experiments for "A confidence-based approach for balancing fairness and accuracy"
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data first commit Jan 19, 2016
plots more style Feb 25, 2018
results add empty files so that dirs are present in github repo Feb 25, 2018
weaklearners first commit Jan 19, 2016
.gitignore add venv to gitignore Feb 25, 2018
README.md add example to README Feb 25, 2018
boosting.py more style Feb 25, 2018
errorfunctions.py more style Feb 25, 2018
experiment-FWL.py more style Feb 25, 2018
experiment-RM.py more style Feb 25, 2018
experiment-RR.py more style Feb 25, 2018
experiment-SDB.py start cleaning Feb 25, 2018
experiment-baselines.py start cleaning Feb 25, 2018
lr.py fix bad code Feb 25, 2018
margin.py more style Feb 25, 2018
massaging.py more style Feb 25, 2018
plot-all.py more style Feb 25, 2018
relabeling.py more style Feb 25, 2018
run-all.sh start cleaning Feb 25, 2018
scatter_plots.py more style Feb 25, 2018
svm.py style Feb 25, 2018
tox.ini more style Feb 25, 2018
utils.py style Feb 25, 2018

README.md

Code and experiments for "A confidence-based approach for balancing fairness and accuracy"

All experiments used in this paper were implemented in Python 3 with the following dependencies

numpy
matplotlib
scikitlearn

One-click rerun of all experiments

To re-run all experiments used in the paper, run the following from the command line

./run-all.sh

This will re-run all the experiments and output the data to plaintext files in the results/ subdirectory.

To generate all plots used in the paper, run the following from the command line:

python plot-all.py

Datasets

The datasets are given the following names

adult
german 
singles 

Loading into Python

For each dataset there is a data loader module and a baseline (see the Baselines section below). We will use adult as the prototype, and unless otherwise stated all datasets operate the same way with adult replaced by the dataset name. The raw data files are adult.train and adult.test. If preprocessing occurred to split a dataset into training and testing subsets, then the unprocessed data files are in the preprocessing/ subdirectory along with python scripts to perform the (randomized) preprocessing. Additional preprocessing is performed to turn categorical features into (possibly many) binary features.

To load a dataset, you can run the following commands from the base directory of the project.

$ python
Python 3.3.3 (default, Dec 30 2013, 23:51:18) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
>>> from data import adult
>>> trainingData, testData = adult.load()
>>> adult.protectedIndex
1
>>> len(trainingData)
32561
>>> trainingData[0]
((39, 1, 0, 0, 0, 0, 0, 1, 0, 0, 13, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2174, 0, 40, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0), -1)

An example experiment

An example experiment, testing the linear regression learner on the German dataset.

Python 3.6.3 (default, Oct  4 2017, 06:09:15) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
>>> from data import german
>>> train, test = german.load()
>>> from margin import *
>>> def lrLearner(train, protectedIndex, protectedValue):
...    marginAnalyzer = lrSKLMarginAnalyzer(train, protectedIndex, protectedValue)
...    shift = marginAnalyzer.optimalShift()
...    print('best shift is: %r' % (shift,))
...    return marginAnalyzer.conditionalShiftClassifier(shift)
... 
>>> h = lrLearner(train, german.protectedIndex, german.protectedValue)
best shift is: -0.19250157835095894
>>> from errorfunctions import signedStatisticalParity, labelError, individualFairness
>>> labelError(test, h)
0.25825825825825827 

Copy-pastable:

from data import german
from errorfunctions import signedStatisticalParity, labelError, individualFairness
from margin import *

train, test = german.load()
def lrLearner(train, protectedIndex, protectedValue):
    marginAnalyzer = lrSKLMarginAnalyzer(train, protectedIndex, protectedValue)
    shift = marginAnalyzer.optimalShift()
    print('best shift is: %r' % (shift,))
    return marginAnalyzer.conditionalShiftClassifier(shift)
 
h = lrLearner(train, german.protectedIndex, german.protectedValue)
labelError(test, h)

Experiments

The experiments are organized by method, using the acronyms from the paper. So random relabeling (RR) is in the experiment-RR.py file. Each experiment has a runAll() function that runs all of the experiments for every dataset and learner (SVM, logistic regression, and AdaBoost). Note that boosting and SVM take ~5-30 minutes per run on large datasets, and each experiment averages over 10 runs.

Plots

The main plots in the paper are produced by the MarginAnalyzer class in margin.py. See the MarginAnalyzer.plotMarginHistogram and MarginAnalyzer.plotTradeoff functions for details.