week-5/week-5-1-fairness-tradeoffs-homework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework 5-2: Fair prediction\n",
    "\n",
    "In this homework you will experiment with modifying the logistic regression classifier we built on the COMPAS data, tuning it to get equal false positive rates between black and white defendants."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 0. Loading the data and building the feature matrix.\n",
    "Free code, copied from our class notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import tree\n",
    "from sklearn import metrics\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select between data on overall arrests and arrests for violent crimes\n",
    "# This allows quick comparisons of the difference between these two data sets\n",
    "violent = False\n",
    "\n",
    "if violent:\n",
    "    fname ='compas-scores-two-years-violent.csv'\n",
    "    decile_col = 'v_decile_score'\n",
    "    score_col = 'v_score_text'\n",
    "else:\n",
    "    fname ='compas-scores-two-years.csv'\n",
    "    decile_col = 'decile_score'\n",
    "    score_col = 'score_text'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cv = pd.read_csv(fname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5278, 53)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data cleaning ala ProPublica\n",
    "cv = cv[\n",
    "    (cv.days_b_screening_arrest <= 30) &  \n",
    "    (cv.days_b_screening_arrest >= -30) &  \n",
    "    (cv.is_recid != -1) &\n",
    "    (cv.c_charge_degree != 'O') &\n",
    "    (cv[score_col] != 'N/A')\n",
    "]\n",
    "\n",
    "# Keep only black and white races for this analysis\n",
    "cv = cv[(cv.race == 'African-American') | (cv.race=='Caucasian')]\n",
    "         \n",
    "# renumber the rows from 0 again\n",
    "cv.reset_index(inplace=True, drop=True) \n",
    "cv.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build up dummy variables for age, race, gender\n",
    "features = pd.concat(\n",
    "    [pd.get_dummies(cv.age_cat, prefix='age'),\n",
    "     pd.get_dummies(cv.sex, prefix='sex'),\n",
    "     pd.get_dummies(cv.c_charge_degree, prefix='degree'), # felony or misdemeanor charge ('f' or 'm')\n",
    "     cv.priors_count],\n",
    "    axis=1)\n",
    "\n",
    "# We should have one less dummy variable than the number of categories, to avoid the \"dummy variable trap\"\n",
    "# See https://www.quora.com/When-do-I-fall-in-the-dummy-variable-trap\n",
    "features.drop(['age_25 - 45', 'sex_Female', 'degree_M'], axis=1, inplace=True)\n",
    "\n",
    "# Try to predict whether someone is re-arrested\n",
    "target = cv.two_year_recid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1. Your basic logistic regression\n",
    "\n",
    "Fit a logistic regression to this data. Print out the accuracy, PPV, and FPV overall, and for just black vs. white defendants. \n",
    "\n",
    "Most of the code you need can be found in the class notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit a logistic regression\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Predict the result on the training data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Free code for you!\n",
    "\n",
    "# cm is a confusion matrix. The rows are guessed, the columns are actual \n",
    "def print_ppv_fpv(cm):\n",
    "    # the indices here are [col][row] or [actual][guessed]\n",
    "    TN = cm[False][False]   \n",
    "    TP = cm[True][True]\n",
    "    FN = cm[True][False]\n",
    "    FP = cm[False][True]\n",
    "    print('Accuracy: ', (TN+TP)/(TN+TP+FN+FP))\n",
    "    print('PPV: ', TP / (TP + FP))\n",
    "    print('FPR: ', FP / (FP + TN))\n",
    "    print('FNR: ', FN / (FN + TP))\n",
    "    print()\n",
    "\n",
    "def print_metrics(guessed, actual):\n",
    "    cm = pd.crosstab(guessed, actual, rownames=['guessed'], colnames=['actual'])\n",
    "    print(cm)\n",
    "    print()\n",
    "    print_ppv_fpv(cm)    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the accuracy, PPV, FPV, FNV for\n",
    "#  - everyone \n",
    "#  - just white defendants\n",
    "#  - just black defendants\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2. Equalizing false positive rates\n",
    "Now you'll build your own classifier that equalizes the false positive rates between white and non-white defendants. There are many ways to do this. We're going to use race explicitly to set a different threshold for white and black defendants. \n",
    "\n",
    "To begin with, we are going to write our own prediction function, starting with this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This takes a trained LogisticRegression, a set of features, and a threshold\n",
    "# Predicts true wherever the regression gives a probability > threshold\n",
    "# Note: returns a numpy array, not a dataframe\n",
    "def predict_threshold(classifier, features, threshold):\n",
    "    # predict_proba returns two columns: probability of true, and probability of false\n",
    "    # [:,1] selects the second column\n",
    "    return classifier.predict_proba(features)[:,1] > threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is the same as lr.predict(x) when we use a threshold of 0.5\n",
    "guessed2 = predict_threshold(lr, x, 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now adapt this function so it takes two thresholds `a_threshold` and `b_threshold`, and a column of values `use_b` which means use the `b_threshold` for any row where it's true. The idea is to allow us to adjust the thresholds independently on two different groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Write a function which takes the following arguments\n",
    "def predict_threshold_groups(classifier, feautes, a_threshold, b_threshold, use_b):\n",
    "    # calculate probabilities from our classifier\n",
    "    \n",
    "    # Create one Series which is True where the probabilities are bigger than a_threshold, \n",
    "    # and another for b_threshold\n",
    "    # Then combine them, selecting values from either Series according to use_b\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use this function with different thresholds for black and white defendants. Print out the confusion martrix, accuracy, FPV, and PPV for the results -- again, overall and for each race."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict recidivism with different thresholds for black and white\n",
    "# Print out metrics for everyone, black, and white\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tune the thresholds so the False Positive Rate is the same for white and black defendants.\n",
    "- What did you change to achive this?\n",
    "- What effect does this have on the overall accuracy, FPR, FNR, and PPV?\n",
    "- What effect does this have on the PPV for white and black?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case I raised the threshold for black defendants from 0.5 to 0.585, which equalizes the FPR at about 17%. The overall accuracy fell only slightly from 66% to 65%, and the accuracy for black defendants fell from 67% to 64%. But the PPV for black defendants -- the probability that someone who is categorized as high risk will actually be re-arrested within two years -- increased from 69% to 75%, because the higher threshold removes some of the people who were not particularly risky from the high risk group. The cost is a higher false negative rate for black defendents, which has gone up from 33% to 52%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus: Predicting race and the impossibility of blinding\n",
    "So far we've excluded race as a predictive variable, hoping that this would make the results unbiased. But is race encoded in the other data points? To find out, alter the regression above to try to predict race from the other demographic and criminal history variables.\n",
    "\n",
    "How accurately can you predict race just on these factors alone?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use cross validation and the classifier of your choice to see how well you can predict race\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare this accuracy to just guessing one race all the time. Which race is more common in this data and what would the accuracy be if we just always guessed that race."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the most common race in our arrest data?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the accuracy if we always guess the most common race?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this, how much information about race \"leaks\" into our original recidivism predictor, even if we don't give it the race variable as a feature?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(your answer here)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}