Permalink
Browse files

improving constraints for FPR and FNR

  • Loading branch information...
Muhammad Bilal Zafar Muhammad Bilal Zafar
Muhammad Bilal Zafar authored and Muhammad Bilal Zafar committed Jan 30, 2018
1 parent b5fe95a commit fc11b119b210dc1ad3c999680ad5b4aa44b6f960
@@ -12,7 +12,7 @@
* [2. Using the code](#2-using-the-code)
* [2.1. Training a(n) (un)fair classifier](#21-training-an-unfair-classifier)
* [2.2. Making predictions](#22-making-predictions)

* [3. Code update](#3-code-update)

## 1. Fair classification demo

@@ -86,10 +86,10 @@ The results for the fair classifier look like this:
```
=== Constraints on FPR ===
Accuracy: 0.778
Accuracy: 0.774
|| s || FPR. || FNR. ||
|| 0 || 0.18 || 0.33 ||
|| 1 || 0.19 || 0.19 ||
|| 0 || 0.16 || 0.37 ||
|| 1 || 0.16 || 0.21 ||
```
So, the classifier sacrificed around 5% of (overall) accuracy to remove disparity in false positive rates for the two groups. Specifically, it decreased the FPR (as compared to the original classifier) for group-0 and increased the FPR for group-1. The code will also show how the classifier shifts its boundary to achieve fairness, while making sure that a minimal loss in overall accuracy is incurred.
@@ -114,10 +114,10 @@ The results and the decision boundary for this experiment are:
```
=== Constraints on FNR ===
Accuracy: 0.767
Accuracy: 0.768
|| s || FPR. || FNR. ||
|| 0 || 0.57 || 0.17 ||
|| 1 || 0.04 || 0.15 ||
|| 0 || 0.58 || 0.16 ||
|| 1 || 0.04 || 0.14 ||
```

<img src="synthetic_data_demo/img/syn_cons_dtype_1_cons_type_2.png" width="500px" style="float: right;">
@@ -140,10 +140,10 @@ The output looks like:
```
=== Constraints on both FPR and FNR ===
Accuracy: 0.606
Accuracy: 0.637
|| s || FPR. || FNR. ||
|| 0 || 0.75 || 0.02 ||
|| 1 || 0.82 || 0.00 ||
|| 0 || 0.68 || 0.02 ||
|| 1 || 0.76 || 0.00 ||
```

<img src="synthetic_data_demo/img/syn_cons_dtype_1_cons_type_4.png" width="500px" style="float: right;">
@@ -191,10 +191,10 @@ Accuracy: 0.678
-----------------------------------------------------------------------------------
== Constraints on FPR ==
Accuracy: 0.665
Accuracy: 0.659
|| s || FPR. || FNR. ||
|| 0 || 0.27 || 0.37 ||
|| 1 || 0.29 || 0.45 ||
|| 0 || 0.24 || 0.43 ||
|| 1 || 0.25 || 0.50 ||
```
You will notice that in this dataset, controlling for unfairness w.r.t. false positive rates also helps control unfairness on false negative rates (rather than making it worse, or not affecting it at all). For more discussion, please see Section 5 of our <a href="http://arxiv.org/abs/1610.08452" target="_blank">paper</a>.
@@ -226,3 +226,8 @@ The function will return the weight vector learned by the classifier. Given an _
distance_boundary = numpy.dot(w, X) # will give the distance from the decision boundary
predicted_labels = np.sign(distance_boundary) # sign of the distance from boundary is the class label
```


## 3. Code update

We have made changes to constraints for false positive and false negative rates. In the <a href="http://arxiv.org/abs/1610.08452" target="_blank">paper</a>(in Section 4), we compute the misclassification covariance over the whole dataset for _all_ constraint types. However, if the base rates (fraction in positive class in the training dataset) are different for various groups, computing the covariance over the whole dataset could result in under- or over-estimating the false positive / negative rates (the constraints for overall misclassification rates do not get affected by different base rates). To overcome this issue, we have changed the constraints in the following way: for false positive rate constraints, the covariance is computed over the ground truth negative dataset, and for false negative rate constraints, the covariance is computed over the ground truth positive dataset. This way, the constraints do not under- or over-estimate FPR or FNR.
@@ -50,7 +50,7 @@ def load_compas_data():

# load the data and get some stats
df = pd.read_csv(COMPAS_INPUT_FILE)

df = df.dropna(subset=["days_b_screening_arrest"]) # dropping missing vals

# convert to np array
data = df.to_dict('list')
@@ -120,7 +120,7 @@ def train_model_disp_mist(x, y, x_control, loss_function, EPS, cons_params=None)

# check that the fairness constraint is satisfied
for f_c in constraints:
assert(f_c.value == True)
assert(f_c.value == True) # can comment this out if the solver fails too often, but make sure that the constraints are satisfied empirically. alternatively, consider increasing tau parameter
pass


@@ -215,16 +215,19 @@ def get_constraint_list_cov(x_train, y_train, x_control_train, sensitive_attrs_t

if index_dict is None: # binary attribute, in this case, the attr_arr_transformed is the same as the attr_arr

s_val_to_total = {}
s_val_to_avg = {}
cons_sum_dict = {0:{}, 1:{}, 2:{}} # sum of entities (females and males) in constraints are stored here
s_val_to_total = {ct:{} for ct in [0,1,2]} # constrain type -> sens_attr_val -> total number
s_val_to_avg = {ct:{} for ct in [0,1,2]}
cons_sum_dict = {ct:{} for ct in [0,1,2]} # sum of entities (females and males) in constraints are stored here

for v in set(attr_arr):
s_val_to_total[v] = sum(x_control_train[attr] == v)
s_val_to_total[0][v] = sum(x_control_train[attr] == v)
s_val_to_total[1][v] = sum(np.logical_and(x_control_train[attr] == v, y_train == -1)) # FPR constraint so we only consider the ground truth negative dataset for computing the covariance
s_val_to_total[2][v] = sum(np.logical_and(x_control_train[attr] == v, y_train == +1))


s_val_to_avg[0] = s_val_to_total[1] / float(s_val_to_total[0] + s_val_to_total[1]) # A_0 in our formulation
s_val_to_avg[1] = 1.0 - (s_val_to_total[1] / float(s_val_to_total[0] + s_val_to_total[1])) # A_1 in our formulation

for ct in [0,1,2]:
s_val_to_avg[ct][0] = s_val_to_total[ct][1] / float(s_val_to_total[ct][0] + s_val_to_total[ct][1]) # N1/N in our formulation, differs from one constraint type to another
s_val_to_avg[ct][1] = 1.0 - s_val_to_avg[ct][0] # N0/N


for v in set(attr_arr):
@@ -236,9 +239,9 @@ def get_constraint_list_cov(x_train, y_train, x_control_train, sensitive_attrs_t
# #DCCP constraints
dist_bound_prod = mul_elemwise(y_train[idx], x_train[idx] * w) # y.f(x)

cons_sum_dict[0][v] = sum_entries( min_elemwise(0, dist_bound_prod) ) * (s_val_to_avg[v] / len(x_train))
cons_sum_dict[1][v] = sum_entries( min_elemwise(0, mul_elemwise( (1 - y_train[idx])/2.0, dist_bound_prod) ) ) * (s_val_to_avg[v] / len(x_train))
cons_sum_dict[2][v] = sum_entries( min_elemwise(0, mul_elemwise( (1 + y_train[idx])/2.0, dist_bound_prod) ) ) * (s_val_to_avg[v] / len(x_train))
cons_sum_dict[0][v] = sum_entries( min_elemwise(0, dist_bound_prod) ) * (s_val_to_avg[0][v] / len(x_train)) # avg misclassification distance from boundary
cons_sum_dict[1][v] = sum_entries( min_elemwise(0, mul_elemwise( (1 - y_train[idx])/2.0, dist_bound_prod) ) ) * (s_val_to_avg[1][v] / sum(y_train == -1)) # avg false positive distance from boundary (only operates on the ground truth neg dataset)
cons_sum_dict[2][v] = sum_entries( min_elemwise(0, mul_elemwise( (1 + y_train[idx])/2.0, dist_bound_prod) ) ) * (s_val_to_avg[2][v] / sum(y_train == +1)) # avg false negative distance from boundary
#################################################################


@@ -360,24 +363,28 @@ def get_sensitive_attr_constraint_fpr_fnr_cov(model, x_arr, y_arr_true, y_arr_di
arr = np.dot(model, x_arr.T) * y_arr_true # the product with the weight vector -- the sign of this is the output label
arr = np.array(arr)

s_val_to_total = {}
s_val_to_avg = {}
cons_sum_dict = {0:{}, 1:{}, 2:{}} # sum of entities (females and males) in constraints are stored here
s_val_to_total = {ct:{} for ct in [0,1,2]}
s_val_to_avg = {ct:{} for ct in [0,1,2]}
cons_sum_dict = {ct:{} for ct in [0,1,2]} # sum of entities (females and males) in constraints are stored here

for v in set(x_control_arr):
s_val_to_total[v] = sum(x_control_arr == v)
s_val_to_total[0][v] = sum(x_control_arr == v)
s_val_to_total[1][v] = sum(np.logical_and(x_control_arr == v, y_arr_true == -1))
s_val_to_total[2][v] = sum(np.logical_and(x_control_arr == v, y_arr_true == +1))


s_val_to_avg[0] = s_val_to_total[1] / float(s_val_to_total[0] + s_val_to_total[1]) # A_0 in our formulation
s_val_to_avg[1] = 1.0 - ( s_val_to_total[1] / float(s_val_to_total[0] + s_val_to_total[1]) ) # A_1 in our formulation
for ct in [0,1,2]:
s_val_to_avg[ct][0] = s_val_to_total[ct][1] / float(s_val_to_total[ct][0] + s_val_to_total[ct][1]) # N1 / N
s_val_to_avg[ct][1] = 1.0 - s_val_to_avg[ct][0] # N0 / N


for v in set(x_control_arr):
idx = x_control_arr == v
dist_bound_prod = arr[idx]

cons_sum_dict[0][v] = sum( np.minimum(0, dist_bound_prod) ) * (s_val_to_avg[v] / len(x_arr))
cons_sum_dict[1][v] = sum( np.minimum(0, ( (1 - y_arr_true[idx]) / 2 ) * dist_bound_prod) ) * (s_val_to_avg[v] / len(x_arr))
cons_sum_dict[2][v] = sum( np.minimum(0, ( (1 + y_arr_true[idx]) / 2 ) * dist_bound_prod) ) * (s_val_to_avg[v] / len(x_arr))
cons_sum_dict[0][v] = sum( np.minimum(0, dist_bound_prod) ) * (s_val_to_avg[0][v] / len(x_arr))
cons_sum_dict[1][v] = sum( np.minimum(0, ( (1 - y_arr_true[idx]) / 2 ) * dist_bound_prod) ) * (s_val_to_avg[1][v] / sum(y_arr_true == -1))
cons_sum_dict[2][v] = sum( np.minimum(0, ( (1 + y_arr_true[idx]) / 2 ) * dist_bound_prod) ) * (s_val_to_avg[2][v] / sum(y_arr_true == +1))


cons_type_to_name = {0:"ALL", 1:"FPR", 2:"FNR"}

0 comments on commit fc11b11

Please sign in to comment.