Fair classifier with adversarial networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fair classifiers with adversarial networks\n",
    "\n",
    "Gilles Louppe, 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We illustrate how one can use adversarial networks for building a classifier whose output is forced to be independent of some chosen attribute. We follow the adversarial networks setup described in \"Learning to Pivot with Adversarial Networks\" (Louppe, Kagan and Cranmer, 2016, [arXiv:1611.01046](https://arxiv.org/abs/1611.01046)).\n",
    "\n",
    "In this notebook, we will show more specifically how one can build a fair classifier whose decision is made independent of gender."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "@article{louppe2016pivot,\n",
    "           author = {{Louppe}, G. and {Kagan}, M. and {Cranmer}, K.},\n",
    "            title = \"{Learning to Pivot with Adversarial Networks}\",\n",
    "          journal = {ArXiv e-prints},\n",
    "    archivePrefix = \"arXiv\",\n",
    "           eprint = {1611.01046},\n",
    "     primaryClass = \"stat.ML\",\n",
    "             year = 2016,\n",
    "            month = nov,\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython import display\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using the [adult](https://archive.ics.uci.edu/ml/datasets/Adult) UCI dataset, where the prediction task is to predict whether someone makes over 50,000$ a year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>Education</th>\n",
       "      <th>Education-Num</th>\n",
       "      <th>Martial Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital Gain</th>\n",
       "      <th>Capital Loss</th>\n",
       "      <th>Hours per week</th>\n",
       "      <th>Country</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age         Workclass  fnlwgt  Education  Education-Num  \\\n",
       "0   39         State-gov   77516  Bachelors             13   \n",
       "1   50  Self-emp-not-inc   83311  Bachelors             13   \n",
       "2   38           Private  215646    HS-grad              9   \n",
       "3   53           Private  234721       11th              7   \n",
       "4   28           Private  338409  Bachelors             13   \n",
       "\n",
       "       Martial Status         Occupation   Relationship   Race     Sex  \\\n",
       "0       Never-married       Adm-clerical  Not-in-family  White    Male   \n",
       "1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n",
       "2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n",
       "3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n",
       "4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n",
       "\n",
       "   Capital Gain  Capital Loss  Hours per week        Country Target  \n",
       "0          2174             0              40  United-States  <=50K  \n",
       "1             0             0              13  United-States  <=50K  \n",
       "2             0             0              40  United-States  <=50K  \n",
       "3             0             0              40  United-States  <=50K  \n",
       "4             0             0              40           Cuba  <=50K  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_data = pd.read_csv(\n",
    "    \"adult.data.txt\", \n",
    "    names=[\"Age\", \"Workclass\", \"fnlwgt\", \"Education\", \"Education-Num\", \n",
    "           \"Martial Status\", \"Occupation\", \"Relationship\", \"Race\", \"Sex\", \n",
    "           \"Capital Gain\", \"Capital Loss\", \"Hours per week\", \"Country\", \"Target\"],\n",
    "    sep=r'\\s*,\\s*', engine='python', na_values=\"?\")\n",
    "original_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.get_dummies(original_data)\n",
    "target = data[\"Target_>50K\"].values\n",
    "gender = data[\"Sex_Male\"].values\n",
    "del data[\"Target_<=50K\"]\n",
    "del data[\"Target_>50K\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "X_train, X_test, y_train, y_test, gender_train, gender_test = train_test_split(data, target, gender, train_size=0.5)\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Standard classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first train a standard neural network on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import keras.backend as K\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "inputs = Input(shape=(X_train.shape[1],))\n",
    "Dx = Dense(32, activation=\"relu\")(inputs)\n",
    "Dx = Dense(32, activation=\"relu\")(Dx)\n",
    "Dx = Dense(32, activation=\"relu\")(Dx)\n",
    "Dx = Dense(1, activation=\"sigmoid\")(Dx)\n",
    "D = Model(input=[inputs], output=[Dx])\n",
    "D.compile(loss=\"binary_crossentropy\", optimizer=\"adam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.3781     \n",
      "Epoch 2/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3205     \n",
      "Epoch 3/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3073     \n",
      "Epoch 4/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3013     \n",
      "Epoch 5/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.2958     \n",
      "Epoch 6/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.2916     \n",
      "Epoch 7/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.2876     \n",
      "Epoch 8/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.2843     \n",
      "Epoch 9/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.2801     \n",
      "Epoch 10/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.2776     \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb09003a0b8>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D.fit(X_train, y_train, nb_epoch=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.89805904237036704"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "y_pred = D.predict(X_test)\n",
    "roc_auc_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performance is good, but as the plot below illustrates, the distribution of the classifier output is different depending on gender. In particular, the classifier models that women are less likely to make more than 50,000$ a year than men."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEqRJREFUeJzt3X9sXfV5x/HPk+DILjZ2SYgb2QRnP0oXkhHqrKkWVNmZ\nxiBQqo0KlW6tWiZZ07SB0JAI+2PJNE31/lk61nYroqir2sRMLdA2rJ2QuB5rOtYlXUgAD8SolTkD\nBZzhxm1MSPzsD9+oKfeee8+5Pj/u9973S7qKfe7xPc+TOJ9z7vd8z7nm7gIAhGNF0QUAAJIhuAEg\nMAQ3AASG4AaAwBDcABAYghsAAnNJnJXMbFrSaUnnJZ1z961ZFgUAiBYruMtG3f2NzCoBAMTCUAkA\nBMbiXDlpZj+SNKeloZIvuvuDVdYZkzQmSV1dXcNXXnllQwUtnn9bK1Z2VD7x9k+ljnc19JrNbnFx\nUStWtNc+tN16brd+JXpO6qWXXnrD3a+ItbK7131IGij/uVbSs5I+VGv94eFhb1Rp397qT+y+rOHX\nbHalUqnoEnLXbj23W7/u9JyUpEMeI4/dPd5QibufKP95UtJjkj6QfH8CAEhD3eA2s0vNrOfC15Ju\nkPRc1oUBAKqLM6ukX9JjZnZh/X3u/t1MqwIARKob3O7+iqRrc6gFABry9ttva2ZmRgsLC4XW0dvb\nq6mpqZrrdHZ2anBwUB0dVSZhxJRkHjcANKWZmRn19PRoaGhI5dGBQpw+fVo9PT2Rz7u7ZmdnNTMz\now0bNjS8nfaaqwOgJS0sLGj16tWFhnYcZqbVq1cv+50BwQ2gJTR7aF+QRp0ENwAEhjFuAC1n+/hT\nOvHmmdReb6CvSwd37ai5jpnp9ttv1yOPPCJJOnfunNatW6dt27bpwIEDqdUiEdwAWtCJN89oevzm\n1F5vaNcTdde59NJLNTU1pTNnzqirq0tPPvmkBgYGUqvhYgyVAEBKbrjhBj3xxFLI79+/X3fccUcm\n2yG4ASAlt912myYmJrSwsKCjR49q27ZtmWyH4AaAlGzatEnT09Pav3+/du7cmdl2GOMGgBTdeuut\nuvfeezU5OanZ2dlMtkFwA0CK7rzzTvX19Wnz5s2anJzMZBsEN4CWM9DXFWsmSJLXi2twcFB33XVX\natuuhuAG0HLqzbnOwvz8vE6fPv1zy0ZGRjQyMpL6tjg5CQCBIbgBIDAENwAEhuAGgMAQ3AAQGIIb\nAALDdEAArWfvZmnueHqv17teuudYzVVWrlypa665RitWLB0PP/744xoaGkqvhosQ3ABaz9xxac9c\neq+3p7fuKl1dXTp48GDNz5xMC0MlABAYjrgBIAVnzpzR9u3btWLFCm3YsEGPPfZYZtsiuAEgBQyV\nAAAiEdwAEBiGSgC0nt71sWaCJHq9JkJwA2g9deZcZ6HabV2zwlAJAASG4AaAwBDcAFqCuxddQixp\n1ElwAwheZ2enZmdnmz683V2zs7Pq7Oxc1utwchJA8AYHBzUzM6PXX3+90DoWFhbqhnJnZ6cGBweX\ntR2CG0DwOjo6tGHDhqLL0OTkpK677rrMt8NQCQAEJnZwm9lKM/tPMzuQZUEAgNqSHHHfLWkqq0IA\nAPHECm4zG5R0s6SHsi0HAFCPxZk+Y2Zfl/QZST2S7nX3W6qsMyZpTJL6+/uHJyYmGipo/tRJdV++\ntvKJV49I67Y09JrNbn5+Xt3d3UWXkat267nd+pXoOanR0dHD7r411sruXvMh6RZJXyh/PSLpQL2f\nGR4e9kaV9u2t/sTuyxp+zWZXKpWKLiF37dZzu/XrTs9JSTrkdbL1wiPOUMl2Sbea2bSkCUk7zOyr\nyfcnAIA01A1ud7/f3QfdfUjSxyQ95e6/l3llAICqgrkAZ8bXaLDa/XV71xdyC0cAKEqi4Hb3SUmT\nmVRSx/VvPaDp8Zsrn0jzZukAEACunASAwBDcABAYghsAAkNwA0BgCG4ACAzBDQCBIbgBIDAENwAE\nhuAGgMAQ3AAQGIIbAAJDcANAYAhuAAgMwQ0AgSG4ASAwBDcABIbgBoDAENwAEBiCGwACQ3ADQGAI\nbgAIDMENAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAkNwA0BgCG4ACAzBDQCBIbgBIDAENwAEhuAG\ngMAQ3AAQmLrBbWadZvYDM3vWzJ43sz/PozAAQHWXxFjnLUk73H3ezDokfc/MvuPuz2RcGwCgirrB\n7e4uab78bUf54VkWBQCIZku5XGcls5WSDkv6JUmfd/f7qqwzJmlMkvr7+4cnJiYaKmj+1El1X762\nYvmxE3PaPNBb+QOvHpHWbWloW81ifn5e3d3dRZeRq3brud36leg5qdHR0cPuvjXWyu4e+yGpT1JJ\n0qZa6w0PD3ujSvv2Vl1+1X0Hqv/A7ssa3lazKJVKRZeQu3brud36dafnpCQd8phZnGhWibu/WQ7u\nG5PtSwAAaYkzq+QKM+srf90l6Tcl/VfWhQEAqoszq2SdpH8oj3OvkPSP7n4g27IAAFHizCo5Kum6\nHGoBAMTAlZMAEBiCGwACQ3ADQGAIbgAIDMENAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAkNwA0Bg\nCG4ACAzBDQCBIbgBIDAENwAEhuAGgMAQ3AAQmDgfXdYUBvq6NLTriYrl050FFAMABQomuA/u2lH9\niT25lgEAhWOoBAACQ3ADQGAIbgAIDMENAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAkNwA0BgCG4A\nCAzBDQCBIbgBIDAENwAEhuAGgMAQ3AAQmLrBbWZXmlnJzF4ws+fN7O48CgMAVBfnE3DOSfoTd/+h\nmfVIOmxmT7r7CxnXBgCoou4Rt7u/6u4/LH99WtKUpIGsCwMAVGfuHn9lsyFJT0va5O4/fsdzY5LG\nJKm/v394YmKioYLmT51U9+Vr4//Aq0ekdVsa2lazmJ+fV3d3d9Fl5Krdem63fiV6Tmp0dPSwu2+N\ntbK7x3pI6pZ0WNLv1Ft3eHjYG1XatzfZD+y+rOFtNYtSqVR0Cblrt57brV93ek5K0iGPmcexZpWY\nWYekb0j6mrs/2tDuBACQijizSkzSlyRNuftfZ18SAKCWOEfc2yV9QtIOMztSfuzMuC4AQIS60wHd\n/XuSLIdaGjLjazS4p7fyid710j3H8i8IADIWZx53U7v+rQc0PX5z5RPVwhwAWkDwwT3Q16WhXU9U\nLJ/uLKAYAMhB8MF9cNeO6k/sybUMAMgNN5kCgMAQ3AAQGIIbAAJDcANAYAhuAAgMwQ0AgSG4ASAw\nwc/jrqXahTkDfV3Rc78BIAAtHdzVLoWvFuYAEBKGSgAgMAQ3AASG4AaAwBDcABAYghsAAkNwA0Bg\nCG4ACAzBDQCBaekLcKqJ+qgzrqgEEIq2C+6ocOaKSgChYKgEAAJDcANAYAhuAAgMwQ0AgSG4ASAw\nbTerJArTBAGEguAuY5oggFAQ3ACQ0Pbxp3TizTMVy+/fsqiRHLZPcANAQifePFP1oxH/9mvfzGX7\nnJwEgMAQ3AAQGIIbAAJTd4zbzB6WdIukk+6+KfuSUtK7XtrTW335PceW/fJRJyeYPggga3FOTn5Z\n0uckfSXbUlIWFc7VwrwBUScn0pw+yM4BQDV1g9vdnzazoexLwTvlsXMAEB5z9/orLQX3gVpDJWY2\nJmlMkvr7+4cnJiYaKmj+1El1X762oZ+N5eQL0vmzlctXrpLWbqxY/OJrp3X2/GLF8lUrV+jq9/TE\nXr/Wz8zPz6u7u7ti+bETc9o8UPkOIWp5SKJ6blXt1q/U2j1H/R88eWpOay9v7P/m6OjoYXffGmtl\nd6/7kDQk6bk467q7hoeHvVGlfXsb/tll2X1Z5pu46r4DVZeXSqVE60ctD0lUz62q3fp1b+2eo/4P\nPvDVxxt+TUmHPGbGMqsEAALDlZNtjJOfQJjiTAfcL2lE0hozm5G0292/lHVhiJbWnQw5+QmEKc6s\nkjvyKKSdvfjaaX0qIoiraYU7GdbqmaN9oDaGSnIUdaR8/xZVPfJtZWfPL2p6/MMVy0Pa+QBFIbhz\nFHUkOTk5mW8hAILGrBIACAxH3C2Ej18D2gPB3UJa4aQlgPoIblRohSN35qijlRHcF2R8G9iQFHnk\nzhx1oD6C+4KMbwOLeKLCefv4U8G/CwDSQnAjtqij4UbcvyXZhCbG74GfIbjbQK3hhyTSPLJNa+56\nWr0BISG420ArDyW0cm9AFIK7UXs3S3PHK5e34clMAPkiuBs1d1zaM1e5nJOZADJGcNdTa5ogsExR\n880lZswgGsFdD8MebaFWgFaTVqhGzTeX0psxw8VIrYfgRlupNQslya11Q5pXzsVIrYfgRltJK1Tz\nmFfeCrceQDYI7rRFjYlfeI6hl5YW/WEZixpJ+FpcRVpf1sNAtV6/SAR32moFMzNOWl5UWHxx4tup\nXSjEVaQ/k/UwUK1zEEUiuIEcXP2eHk2Pj2S6jdCvIk16glgKp7e0EdzN4OQL0p6PVC5naAUJpDVM\nUtQslDSPbjM/PxBxAd773vuXy3/tGAjuZnD+LBfzoGmkNfzw4mun9amCxuKTDiclHsuOuACvY/9n\n4xe5DAR3nqJOXG78TP615CHqtgBSueeRPKvBOyQdWkl6FHv2/KKmxz9csbzWydWspTUdtGgEd56i\nhj1a9VPeo24LIEk5HZkgWtKj3qSzXKJu3VvkzJdWmXVDcANYlqgwTOvWvaiU7G72AIDCccTdzJJ+\nDia3mgXaAsHdzJJ+DmYr3Gq2qJ0POz0EhOAOUdJbzYb0CfZRO5+9m7PtoRV2elHYKTWu1t9dgQju\nECX9z5b0yL3WNL4k0vzljuohKtCjtGNYtfJOKWu1ZkYViOBGpSb9Za0qaQgnDataNw2LWj+UHUNa\n78SidvRpztXnXcPPIbhRjJWrivlkoaTDTFnvGGq9u8k6lJK+E4sStaNPOle/3t9FEUNoTYrgbmdF\nfizb2o3S7QUc1Rf1n7nW/Wii3t2ktRMI5WP2Gnmnl9bOJzAEdztr4SOSwtS6rUHSUEo6lJHWEFcj\nw0PV1HpXVdTvXtLzN0260yO4gTSleVuDpCdk0wqZtEI16l1V1vVfeK2obYRy/qaGWMFtZjdK+htJ\nKyU95O7jmVYFIFro75TyqD/0v6M66l7ybmYrJX1e0k2SNkq6w8w2Zl0YAKC6OPcq+YCkl939FXc/\nK2lCUpWzLACAPMQZKhmQ9D8XfT8jads7VzKzMUlj5W/nzezFBmtao4/f80aDPxuqNZLoubW1W79S\nu/bceH5dFXfF1E5OuvuDkh5c7uuY2SF335pCScGg59bXbv1K9JylOEMlJyRdedH3g+VlAIACxAnu\n/5D0y2a2wcxWSfqYpG9lWxYAIErdoRJ3P2dmfyTpn7U0HfBhd38+w5qWPdwSIHpufe3Wr0TPmTF3\nz2M7AICU8NFlABAYghsAAlNIcJvZjWb2opm9bGa7qjxvZvZA+fmjZvb+IupMU4yef7fc6zEz+76Z\nXVtEnWmq1/NF6/2amZ0zs4/mWV8W4vRsZiNmdsTMnjezf8m7xrTF+N3uNbNvm9mz5Z4/XUSdaTGz\nh83spJk9F/F89vnl7rk+tHSC878l/YKkVZKelbTxHevslPQdSSbpg5L+Pe86C+j51yW9u/z1Te3Q\n80XrPSXpnyR9tOi6c/h37pP0gqT15e/XFl13Dj3/qaS/Kn99haRTklYVXfsyev6QpPdLei7i+czz\nq4gj7jiX0H9E0ld8yTOS+sxsXd6Fpqhuz+7+fXf/v/K3z2hpvnzI4t4q4Y8lfUPSyTyLy0icnj8u\n6VF3Py5J7h5633F6dkk9ZmaSurUU3OfyLTM97v60lnqIknl+FRHc1S6hH2hgnZAk7ef3tbTHDlnd\nns1sQNJvS/q7HOvKUpx/5/dKereZTZrZYTP7ZG7VZSNOz5+T9CuS/lfSMUl3u/tiPuUVIvP84n7c\nTcbMRrUU3NcXXUsOPivpPndfXDoYawuXSBqW9BuSuiT9m5k94+4vFVtWpn5L0hFJOyT9oqQnzexf\n3f3HxZYVriKCO84l9K12mX2sfszsVyU9JOkmd5/NqbasxOl5q6SJcmivkbTTzM65++P5lJi6OD3P\nSJp1959I+omZPS3pWkmhBnecnj8tadyXBoBfNrMfSXqfpB/kU2LuMs+vIoZK4lxC/y1Jnyyfnf2g\npDl3fzXvQlNUt2czWy/pUUmfaJGjr7o9u/sGdx9y9yFJX5f0hwGHthTvd/ubkq43s0vM7F1autPm\nVM51pilOz8e19A5DZtYv6WpJr+RaZb4yz6/cj7g94hJ6M/uD8vN/r6UZBjslvSzpp1raYwcrZs9/\nJmm1pC+Uj0DPecB3VovZc0uJ07O7T5nZdyUdlbSopU+UqjqtLAQx/53/QtKXzeyYlmZa3Ofuwd7u\n1cz2SxqRtMbMZiTtltQh5ZdfXPIOAIHhykkACAzBDQCBIbgBIDAENwAEhuAGgMAQ3AAQGIIbAALz\n/zfp6+Z52U/JAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb073d7b9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(y_pred[gender_test == 1], bins=50, histtype=\"step\",  normed=1, label=\"M\")\n",
    "plt.hist(y_pred[gender_test == 0], bins=50, histtype=\"step\", normed=1, label=\"F\")\n",
    "plt.ylim(0, 5)\n",
    "plt.legend()\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pearson correlation coefficient between gender and the classifier output also clearly highlights this dependency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.29867010783449688, 0.0)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import pearsonr\n",
    "pearsonr(gender_test, D.predict(X_test).ravel())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training with adversarial networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now jointly train our classifier with an adversarial network. The goal of this second network is to predict gender from the classifier output. If this network is doing well, then it clearly indicates that the classifier output is correlated with the attribute. Accordingly, one can force the classifier to distort its decision to make the adversarial network performs worse. This is the strategy we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def make_trainable(network, flag):\n",
    "    network.trainable = flag\n",
    "    for l in network.layers:\n",
    "        l.trainable = flag\n",
    "\n",
    "inputs = Input(shape=(X_train.shape[1],))\n",
    "\n",
    "Dx = Dense(32, activation=\"relu\")(inputs)\n",
    "Dx = Dense(32, activation=\"relu\")(Dx)\n",
    "Dx = Dense(32, activation=\"relu\")(Dx)\n",
    "Dx = Dense(1, activation=\"sigmoid\")(Dx)\n",
    "D = Model(input=[inputs], output=[Dx])\n",
    "\n",
    "Rx = Dx\n",
    "Rx = Dense(32, activation=\"relu\")(Rx)\n",
    "Rx = Dense(32, activation=\"relu\")(Rx)\n",
    "Rx = Dense(32, activation=\"relu\")(Rx)\n",
    "Rx = Dense(1, activation=\"sigmoid\")(Rx)\n",
    "R = Model(input=[inputs], output=[Rx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lam = 10.0  # control the trade-off between classification performance and independence\n",
    "\n",
    "def make_loss_D(c):\n",
    "    def loss_D(y_true, y_pred):\n",
    "        return c * K.binary_crossentropy(y_pred, y_true)\n",
    "    return loss_D\n",
    "\n",
    "def make_loss_R(c):\n",
    "    def loss_R(z_true, z_pred):\n",
    "        return c * K.binary_crossentropy(z_pred, z_true)\n",
    "    return loss_R\n",
    "\n",
    "opt_D = SGD()\n",
    "D.compile(loss=[make_loss_D(c=1.0)], optimizer=opt_D)\n",
    "\n",
    "opt_DRf = SGD(momentum=0.0)\n",
    "DRf = Model(input=[inputs], output=[D(inputs), R(inputs)])\n",
    "make_trainable(R, False)\n",
    "make_trainable(D, True)\n",
    "DRf.compile(loss=[make_loss_D(c=1.0), make_loss_R(c=-lam)], optimizer=opt_DRf)\n",
    "\n",
    "opt_DfR = SGD(momentum=0.0)\n",
    "DfR = Model(input=[inputs], output=[R(inputs)])\n",
    "make_trainable(R, True)\n",
    "make_trainable(D, False)\n",
    "DfR.compile(loss=[make_loss_R(c=1.0)], optimizer=opt_DfR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.5117     \n",
      "Epoch 2/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3982     \n",
      "Epoch 3/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3617     \n",
      "Epoch 4/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3459     \n",
      "Epoch 5/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3365     \n",
      "Epoch 6/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.3299     \n",
      "Epoch 7/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3249     \n",
      "Epoch 8/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3209     \n",
      "Epoch 9/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3175     \n",
      "Epoch 10/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.3141     \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb073cb23c8>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pretraining of D\n",
    "make_trainable(R, False)\n",
    "make_trainable(D, True)\n",
    "D.fit(X_train, y_train, nb_epoch=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.6385     \n",
      "Epoch 2/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.6056     \n",
      "Epoch 3/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5878     \n",
      "Epoch 4/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5755     \n",
      "Epoch 5/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5689     \n",
      "Epoch 6/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5661     \n",
      "Epoch 7/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5647     \n",
      "Epoch 8/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5636     \n",
      "Epoch 9/10\n",
      "16280/16280 [==============================] - 1s - loss: 0.5629     \n",
      "Epoch 10/10\n",
      "16280/16280 [==============================] - 0s - loss: 0.5624     \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb073abe278>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pretraining of R\n",
    "make_trainable(R, True)\n",
    "make_trainable(D, False)\n",
    "DfR.fit(X_train, gender_train, nb_epoch=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_losses(i, losses):\n",
    "    display.clear_output(wait=True)\n",
    "    display.display(plt.gcf())\n",
    "\n",
    "    ax1 = plt.subplot(311)   \n",
    "    values = np.array(losses[\"L_f\"])\n",
    "    plt.plot(range(len(values)), values, label=r\"$L_f$\", color=\"blue\")\n",
    "    plt.legend(loc=\"upper right\")\n",
    "    plt.grid()\n",
    "    \n",
    "    ax2 = plt.subplot(312, sharex=ax1) \n",
    "    values = np.array(losses[\"L_r\"]) / lam\n",
    "    plt.plot(range(len(values)), values, label=r\"$L_r$\", color=\"green\")\n",
    "    plt.legend(loc=\"upper right\")\n",
    "    plt.grid()\n",
    "    \n",
    "    ax3 = plt.subplot(313, sharex=ax1)\n",
    "    values = np.array(losses[\"L_f - L_r\"])\n",
    "    plt.plot(range(len(values)), values, label=r\"$L_f - \\lambda L_r$\", color=\"red\")  \n",
    "    plt.legend(loc=\"upper right\")\n",
    "    plt.grid()\n",
    "    \n",
    "    plt.show()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "losses = {\"L_f\": [], \"L_r\": [], \"L_f - L_r\": []}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb07302e710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfWd//HXJzskIQhIBCImyOICAoKiLVKpOuJScati\nW+1UHXVGrFTrlE77s3bs1KpddMaFOkLHVku0ihYtxZVUu6gsRlaRXQKJQBBIAiHb5/fH9wRuwr3J\n3c+N+Twfj/vIuWd933Nvzud+zzn3HFFVjDHGmDS/AxhjjEkNVhCMMcYAVhCMMcZ4rCAYY4wBrCAY\nY4zxWEEwxhgDWEEwxhjjsYJgjDEGsIJgjDHGk+F3gEj069dPi4uLo5q2rq6O3Nzc+AaKA8sVGcsV\nGcsVmVTNBbFlW7p06S5VPbrTEVW1yzzGjRun0Vq0aFHU0yaS5YqM5YqM5YpMquZSjS0bsETD2MZ2\nqRaCAVUoK4Ndu+Doo2HNGqithV693GPkSBg16vC4e/dC796+RjbGdBFWELqQNWvghhvgH//oeLyT\nT4b8fNi0CT79FMaOhTPOgO3bXZHIzYVjjoGBA2HAgLaPggIQSc7rMcakFisIKeTgQXjuOfjoI2hs\nhPHj4YQT3EZ62TK4/nrIyoJZs2DCBNi5E0aMgD59YN8+2LMH3noLXn7ZbfinTIEhQ+DFF2HuXCgq\ngowMN25VFezff2SGnBxXGI45ZiSPPuqKCbhWyEsvwYoVcMUVLldDA/Tr54bv2gXLl8Pu3ZCW5lol\nxcXukWanLhjTJVhB8FlLC/z3f8OSJbBoEVRUQHq6ezQ0tB13xAhYuNBtZNvLy3Pf+E86CaZPbzvs\n7ruPHF/VFYbKyuCPBQt6MW4cjB7tisTSpa5IicADDxyez6BB7jVUVgZ/fb16wSmnuN1b+/fD4MGu\n9VJVBX37wrHHunmoutd87LHuMXCgK17G+KGxsZGKigrq6+v9jnJIQUEBa9as6XCcnJwcioqKyMzM\njGoZ9i/nsyefHMLcuW4jePLJMHs2nHsuNDe7b9wbN7rjAEVFMHGi2/DHg4hreRQUuG/77b3yyvss\nXjyRJUtc4ZgxAy6+2G3cX3gBPvvMzWPZMrfhPuUU9zjmGFcgdu+GDRugvBw++ADWroWePV3h278f\nCgvdOPv2hc7Yq5crcKNGud1co0dDfX0+mza5ZQwaBF/8oiswxsRTRUUF+fn5FBcXIymyD7Wmpob8\nDj7sqkp1dTUVFRWUlJREtYyYCoKITAEeBtKBJ1X1Z+2GTwXuBVqAJmCGqv7VG/Yd4EZAgRXAt1Q1\ndcpxgjU0wD33wNy5g7n5Znj88bb77tPSYNw49/BDXl4TP/5x8GE33BDePCZP7nycffvcsY20NNcC\n2brVPbZvd7uhysvhj390u6zcLq62KyQjA44/Ho46yu0yy8tzLakRI1x3TY1bRk0N1NW5ItO3ryuE\n1dVumf37uwLVo4crvnv3woEDkJkJZ5/tds+F2ibU1bUdpupaP2vXwscfw5Ytrrjn5bkCNmiQ28WX\nmekeWVluuk8+gaYmV5yLimw3m9/q6+tTqhiEQ0To27cvO3fujHoeURcEEUkHHgXOAyqAxSIyX1VX\nB4z2JjBfVVVETgGeA04QkUHAt4GTVPWAiDwHTAP+L9o8XcmmTXDppa4FMGVKJY88MqDbHshtPTuq\n1cknBx9P1R2/mDdvFddcczI9esD69fDGG7BunWuxFBW5jfk778AzzxyetmdP14ro2dMN/+wzN7/M\nTFdQDhzoOOOwYXDOOW7X2a5drpDU1cGOHW7DrwrZ2WfRv78rSjU1h6cNtfuvIz17QkmJOz4zfjyc\nfrpreY0c6YqJn6qr4dVX3YkN27e7402DBrl1X1zs8qWluV2IWVmuZbdli1snFRXuWFZzM/TvfxJ3\n3unGHTMGLrvMrePsbH9fH7jWZ2Mj7N4tNDe7jNnZ7rWkpXGoXyr+z8ZawGJpIZwOrFfVjV6QUmAq\ncKggqGptwPi5uNZA4LJ7iEgj0BPYHkOWLmPZMrjgAveBe/llyMtbS0bGAL9jpTwRt0tq9+6djBjh\n+g0eDF/+cvDx9+93B+nz8488FtHc7Foc+fluvrW17mys+np3MLx3b9daqKmB55+HefMOF5i+fd1G\nLz/ftUKmTXMbi2XLttOjx7EUFMDw4W7Y8OFuV2Bampv39u2wbZsrSo2Nrkg0Nro8RUWucKxd684m\n27LFZXrkEfc6Wh1zjFveOefAt751eIOcne02uNnZ7nhN63ahujqLBQvchjw93R2bGTzYLa+1dVJb\nC6+84lpm/fq56RsaXJaqKld4P/jAvYa9e90GMzfXvbZFi1yBDdcXvuBaZx98kM+oUa6YPv88PPmk\nG56X59Zvnz4uy5gxbl1mZ7vW2rHHuvEaGtz63L/f7ZqsqnJfLAoKXLH+29/c3x493OOMM+Cii9xn\noaHBfSlbv96dmLF9uyty69fDnj0T2b8f/vznI4t4Wtrh6Xv0OPw56Nnz89OiE43ynsoiciUwRVVv\n9J5fC0xQ1entxrsMuA/oD1ykqv/w+t8O/BdwAHhNVb8eYjk3ATcBFBYWjistLY0qb21tLXnx2gEf\npW3bcpg+/VSys1t44IHlDB68PyVyBWO5IpOoXAcPprF1aw92785i/fp8tm3rwf796bz7bl/q69MB\nEFFycpo5cMBVvpycZgYMOMC+fZlUV4f+yt2nz0EyMpTdu7Noagq+RevZs4nCwnqGD6+lZ88mevVq\nYsKEaoYPryHdLZ76+jR27symqiqHmpoMWlqEfv0aaGwUDhxIp7Cwnry8JnJyWujb121lA9dXY6Pw\n3nt92LQpj337MqipyaSmJoPq6iw2bsyjsTHt0OscPNidGrd9e49D/YPJzm6mZ89mGhrSqK9Po7k5\njZ49m8jKamHfvkxaWtp+kx46tIZhw2rJzDxA795p3HzzPoYPP570dKWlBRoa0qiry6C5WcjKaqGu\nLuOI9Z+R4f726NGMCGRltcRUKFRd8QUhLU1paWkmvXWld2D9+vXs3bu3Tb/JkycvVdXxnU2b8IIQ\nMP4k4G5VPVdEjgJeAK4G9gB/AJ5X1ac7Wub48eN1yZIlUeUtKyvj7LPPjmraeNi71zX9d+1yze3h\nw1MjVyiWKzLJzrV7N/z1r+7vli3u74knuhbHxo3u0bs39Oq1jquuGsbAgYe/VX/yyeFHc7NrNVxw\ngTtov3u3+9acluY+o4k6YB/u+jp40LWU9u1zJzOUl7vWz7BhruXQs6fbVTVwoGvp7N3rso8de7gF\n1NQEf/oTvP66e71HH+2mHzrUtbh693bHoAJzrVmzhhNPPLHDbI2NrhXZenyrsbFta07E7WZsaXEZ\nRNyy8vNd/+xsN6y+3j0OHnSPxsbDjxde+DXr1i3ne997lEGD9jNgQM9O11mw7CISVkGIZZfRNuDY\ngOdFXr+gVPVtERkiIv2AycAmVd3phZ0HfAHosCB0ZdOnu6btW28dLgbGRKtPH7jkks7HKyvbxlln\nDTv0vJNtHL16BT+t2S/Z2W4XF7hjKNHIyICpU90jnjIzD+/eanXwoDsm1dLiisSBA25XXUaGKwq7\nd7svhcG0HqvIzHQFIzMTqqpWcNppoxg40LU4Ei2WgrAYGCYiJbhCMA34WuAIIjIU2OAdVD4VyAaq\ngU+AM0SkJ26X0TlAdF/9u4C5c+Hpp91ZRZMm+Z3GGJMo2dmHD4wHOwGgpeVwS6C+3hWBnJzDBaD9\nMeH165dzww3XMHAg1NREtzcnElEXBFVtEpHpwKu4007nqOoqEbnFGz4LuAK4zjtwfAC42rvQ0nsi\n8jywDHc66gfAE7G9lNS0ciX8y7+48+V/8AO/0xhj/JSWdvhAdzhWrlzJyGibRlGI6XcIqroAWNCu\n36yA7vuB+0NM+yPgR7EsP9Xt2+dOL+3VC/7wB/vlrTFd0YwZ7thFPI0ZAw891PE4W7duJT8/n4KC\ngjb977vvPqqrq7nxxhs5IdivSmPwOTlZKjX9+7+709uee86dHmiMMeFasWIFo1ovXex57733mDt3\nLsXFxXEvBmCXrkiYt96CX/8a7rzTXXLCGNM1dfZNPlGWL19+REEYMWIEX/rSl5je/oJlcWIthASo\nrXWXdxg2DO691+80xpiuaMWKFTzxxBMUFxdTXFzMOeecQ3l5OaNHj07YMq2FkADf/747N/ztt8M/\neGSMMYGeCbz+Cu7idrNnz2ZiAnc5WEGIs7ffdpcb+Pa3bVeRMSa+ZsyYkdD52y6jONq/393EZsgQ\n+OlP/U5jjDGRsRZCHP3wh+7XyIsWuYt/GWNMV2IthDj5+9/d2Qj/+q/uqozGGNPVWEGIgwMH3K6i\nwYPh/qA/wzPGmNRnu4xipAq33uquHf/663Y7R2M+L1S1S90xDVzmWFgLIUazZ8NvfuOOH5x7rt9p\njDHxkJOTQ3V1dcwb2GRqvadyTk5O1POwFkIMtm+H73zH3b3qnnv8TmOMiZeioiIqKipiuj9xvNXX\n13e6sc/JyaGoqCjqZVhBiMFdd7mbjsyaBWHcyMgY00VkZmZSUlLid4w2ysrKGDt2bEKXYbuMovT3\nv8Pvf+8uYDd0qN9pjDEmdlYQonTvve4m4DNn+p3EGGPiI6aCICJTRGStiKwXkSM2jSIyVUSWi0i5\niCwRkYkBw3qLyPMi8pGIrBGRM2PJkkxLl8LChe74gf0AzRjzeRH1MQQRSQceBc4DKoDFIjJfVVcH\njPYmMN+7heYpwHNA60W8HwYWquqVIpIFdH736BTx059CQYE73dQYYz4vYmkhnA6sV9WNqtoAlAJt\nbmOtqrV6+LytXEABRKQAmATM9sZrUNU9MWRJmtWrYd48uO02VxSMMebzQqI9z1ZErgSmqOqN3vNr\ngQmqOr3deJcB9wH9gYtU9R8iMgZ3D+XVwGhgKXC7qtYFWc5NwE0AhYWF40pLS6PKW1tbS15eXlTT\nBvrpT0/gnXeOprT0XQoKGmOeX7xyxZvliozliozlilws2SZPnrxUVcd3OqKqRvUArgSeDHh+LfBI\nB+NPAt7wuscDTbgCAm730b2dLXPcuHEarUWLFkU9basNG1TT01XvvDPmWR0Sj1yJYLkiY7kiY7ki\nF0s2YImGsV2PZZfRNuDYgOdFXr9QhedtYIiI9MMdc6hQ1fe8wc8Dp8aQJSkefRRE4I47/E5ijDHx\nF0tBWAwME5ES76DwNGB+4AgiMlS8i4GIyKlANlCtqlXAVhEZ4Y16Dm73Ucravx/mzIErroCBA/1O\nY4wx8Rf1WUaq2iQi04FXgXRgjqquEpFbvOGzgCuA60SkETgAXO01XwBuA57xislG4FsxvI6EmzsX\n9uyxM4uMMZ9fMV26QlUXAAva9ZsV0H0/EPSC0KpajjuWkPKam+Hhh2HkSLstpjHm88uuZRSG3/0O\nVqxwl6roYlfDNcaYsNmlKzpRVwc/+AFMmADTpvmdxhhjEsdaCJ14+GF3mevnnrPWgTHm881aCB3Y\nswcefBC+8hX44hf9TmOMMYllBaEDv/qVKwr/+Z9+JzHGmMSzghBCdbUrCFdcAWPG+J3GGGMSzwpC\nCA8+CLW18OMf+53EGGOSwwpCEJ9+Cv/zP3DNNXDyyX6nMcaY5LCCEMT990N9PfzoR34nMcaY5LGC\n0M62bfDYY3DddTB8uN9pjDEmeawgtPPTn7pLVdx9t99JjDEmuawgBHjjDZg1C268EUpK/E5jjDHJ\nZQXBs3kzXH01nHiiO8PIGGO6GysIgCrccgs0NsJLL0GK3kHPGGMSyq5lhCsCr77qfog2dKjfaYwx\nxh8xtRBEZIqIrBWR9SIyM8jwqSKyXETKRWSJiExsNzxdRD4QkVdiyRGLv/3N3fRm1CiYPt2vFMYY\n47+oC4KIpAOPAhcAJwHXiMhJ7UZ7ExitqmOA64En2w2/HVgTbYZYzZ0LkyZBTo6750GGtZeMMd1Y\nLC2E04H1qrpRVRuAUmBq4AiqWhtwy8xcoLUbESkCLuLIIpEU+/fDnXfC+PHw4YcwerQfKYwxJnXI\n4e11hBOKXAlMUdUbvefXAhNUdXq78S4D7gP6Axep6j+8/s97/fOB76rqxSGWcxNwE0BhYeG40tLS\nqPLW1taSF3C0eO7cY3niieP57//+gFGj9kY1z3honytVWK7IWK7IWK7IxZJt8uTJS1W181sWq2pU\nD+BK4MmA59cCj3Qw/iTgDa/7YuAxr/ts4JVwljlu3DiN1qJFiw51/+Uvqvn5qhdfHPXs4iYwVyqx\nXJGxXJGxXJGLJRuwRMPYxsay13wbcGzA8yKvX6jC87aIDBGRfsAXgUtE5EIgB+glIk+r6jdiyBPS\nc8/BvHlDWLPG3Rv5ySdhyBB45JFELM0YY7qmWI4hLAaGiUiJiGQB04D5gSOIyFARd+NJETkVyAaq\nVfX7qlqkqsXedG8lqhgALFkC8+YV8W//Br/9rbvHwT/+Accdl6glGmNM1xN1C0FVm0RkOvAqkA7M\nUdVVInKLN3wWcAVwnYg0AgeAq73mS1I98ACcf/7bDBt2NgMGQGZmshMYY0zqi+lES1VdACxo129W\nQPf9wP2dzKMMKIslRzjS02Hw4EQvxRhjuq6ozzLyg4jsBLZEOXk/YFcc48SL5YqM5YqM5YpMquaC\n2LIdp6pHdzZSlyoIsRCRJRrOaVdJZrkiY7kiY7kik6q5IDnZ7OJ2xhhjACsIxhhjPN2pIDzhd4AQ\nLFdkLFdkLFdkUjUXJCFbtzmGYIwxpmPdqYVgjDGmA1YQjDHGAN2kIHR2I58k5jhWRBaJyGoRWSUi\nt3v97xGRbd6NhMq9azwlO9tmEVnRejMjr18fEXldRNZ5f49KcqYRAeukXET2icgMP9aXiMwRkR0i\nsjKgX8j1IyLf9z5va0Xk/CTnelBEPvJuTvWiiPT2+heLyIGA9TYr9JwTkivk++bz+no2INNmESn3\n+idzfYXaNiT3MxbOFfC68gN3WY0NwBAgC/gQOMmnLAOAU73ufOBj3M2F7sFdAtzP9bQZ6Neu3wPA\nTK97JnC/z+9jFXCcH+sLd7XeU4GVna0f7z39EHftrhLv85eexFz/BGR43fcH5CoOHM+H9RX0ffN7\nfbUb/gvgbh/WV6htQ1I/Y92hhdDpjXySRVUrVXWZ112Du1vcID+yhGkq8JTX/RRwqY9ZzgE2qGq0\nv1SPiaq+Dexu1zvU+pkKlKrqQVXdBKzHfQ6TkktVX1PVJu/pu7grESdViPUViq/rq5V3Ic6rgLmJ\nWHZHOtg2JPUz1h0KwiBga8DzClJgIywixcBY4D2v121eE39OsnfNeBR4Q0SWirspEUChqlZ63VVA\noQ+5Wk2j7T+q3+sLQq+fVPrMXQ/8OeB5ibf74y8icpYPeYK9b6myvs4CPlXVdQH9kr6+2m0bkvoZ\n6w4FIeWISB7wAjBDVfcBj+N2aY0BKnHN1mSbqO7e1xcAt4rIpMCB6tqpvpyjLO7y6pcAf/B6pcL6\nasPP9ROKiPwAaAKe8XpVAoO99/kO4Pci0iuJkVLufWvnGtp+6Uj6+gqybTgkGZ+xLvU7hH79+mlx\ncXFU09bV1ZGbmxvfQHFguSJjuSJjuSKTqrkgtmxLly7dpWFc3C7hB0vi+YjXLTRTieWKjOWKjOWK\nTKrmUk39W2gan6gqe+r3sPvAbrLSs2hsaWTfwX00tTTRoi00tzRzsPkg9U31hx61DbXsrd/Lnvo9\nHGg6QEZaBumSTpqkISIIcugvwJYtW/j7O38nXdJJT0unuaWZZm2muaXZLcPrbtbmQ8sEUA6dNXFE\nd2v2wO5wp2lVWVXJ3BrXqm+fu6O/wZYdTXeonJWVlTy196mwpq9vqqeusY6C7AJys3IPrXOgbbcc\n7u5oWGD/9sMqt1fybO2zQYe1ZmrRlrbdAf0CnwfrF/jeBMvU2q/1eet8du7aSe/tvduMH+772Trf\n1u5gn6fA5+GOA7Br1y76VfUjHCJy6H8oPc37K23/pknaoff9iM9CQP/2n6tg05ybcy5nc3ZY2aJl\nBcFnqsr2mu3srNtJTUMNNQdrqG2oPdQd+HfDZxtYu2stlbWVNDQ3RLW8NEkjJyOH5pbmQwWk9UN3\nhM0dz6u1WAQWFqDNRrh9N3DEP3a406gqBxsOklWTdcQ/TWd/A+cbKke43cFyNhxsIKc+p9PXIiJk\np2eTm5XLlj1b2N+4/9D6DHwf2m+oQg1r/961H9bY0EjmvsygwwRp84WgtTtN0o54Hqxf4PoMli/Y\nRrj1s7L/4H4O1h489L6G+162zqe1X6jC0/55uOPU1tdSu6eWcLQWt8AvRcGed1TQwh0GcGbRmWHl\nioUVhCSrqq3itQ2v8damt9j42UZWVq7ks7c/63CarPQs8rPyGVwwmImDJzIofxDH5B1Dnx59aGxp\nJCMtg4LsAjLSMg798+Zk5LR55Gbl0junN3lZeaRJ8HMJAv/hFpUtYuJZEw+1BFo3/K1/2397TZay\nsjLOPvtsX5bdEcsVGcsVubKysoQvwwpCEuyo28HTy5/m6eVP80HVBwAc3fNoTjz6RE7vczoXjb2I\nQb0GkZeVR35WPvnZ+Yf+5mXlkZWelZScgd+Y0iWd7IzspCzXmFTT2NhIRUUF9fX1fkc5pKCggDVr\n1nQ4Tk5ODkVFRWRGeeN4KwgJVF5Vzn/+5T95+eOXaWpp4vRBp3PfOfdx/vHnM/qY0aRJmvtGMuFs\nv6MaYwJUVFSQn59PcXGxb63h9mpqasjPzw85XFWprq6moqKCkpKSqJYRVkEQkSnAw7jLBzypqj8L\nMs7ZwENAJrBLVb8kIscCv8X9mEKBJ1T1YW/8e4B/AXZ6s/gPVV0Q1atIMZ/s/YT/t+j/8bsPf0fv\nnN7MmDCDb439FicdfZLf0YwxYaivr0+pYhAOEaFv377s3Lmz85FD6LQgiEg68ChwHu7XcItFZL6q\nrg4YpzfwGDBFVT8Rkf7eoCbgTlVdJiL5wFIReT1g2l+p6s+jTp9i9tTv4b537uPh9x4G4K4v3MXM\niTM5qodfP6Q1xkSrKxWDVrFmDqeFcOhaQN4CW68FtDpgnK8B81T1EwBV3eH9rcT92g9VrRGR1utz\nBE7b5R1sOsjjSx7n3rfv5bMDn3Ht6Gu5d/K9DC4Y7Hc0Y4wJWziXrgjnmhnDgaNEpMy7Fs517WcS\n5No9kBrXo4nJKx+/womPnsh3Xv0O4waMY9nNy3jq0qesGBhjupxOL10hIlfidgXd6D2/FpigqtMD\nxnkEGI+7ImUP4B/ARar6sTc8D/gL8F+qOs/rVwjswh1buBcYoKrXB1n+TcBNAIWFheNKS0ujeqG1\ntbXk5eVFNW0wqkrp1lKe2PQEQ3KHcMuQWzitz2m+54oXyxUZyxWZVM9VUFDA0KFD/Y7DnDlzWLly\nJb/85S9pbm4mPT2902nWr1/P3r172/SbPHnyUlUd3+nEnf2UGTgTeDXg+feB77cbZybw44Dns4Gv\net2ZwKvAHR0so5gwrjueSpeu+OGbP1TuQac9P00PNB6Iej6p+lN5yxUZyxWZVM+1evVqf4N4br31\nVn388cdVVXXfvn1hTRMsO2FeuiKcXUaLgWEiUuJddXIaML/dOH8EJopIhoj0BCYAa8Qd4ZgNrFHV\nXwZOICIDAp5eBqyki/jF33/BT975CTeOvZFnLn+GnIwcvyMZYz6Hli9fzqhRo5K2vE4PKqtqk4hM\nx33LTwfmqOoqEbnFGz5LVdeIyEJgOdCCOzV1pYhMBK4FVoh3WzoOn176gIiMwe0y2gzcHO8XlwgL\n1y/krtfv4qqTr2LWxbNC/urXGGNitXLlSkaOHJm05YX1OwRvA76gXb9Z7Z4/CDzYrt9fgaDnQanq\ntRElTQGf7P2Er73wNUYVjuI3U39Delrn+/OMMV3bjIUzKK8q73zECIw5ZgwPTXmow3G2bt1Kfn4+\nBQUFQYeratxPjbWvt2FSVW5+5WYamhuYd9U8emb29DuSMeZzbMWKFUfsLqqqquLMM8/k/vvvp6qq\nKu7LtEtXhKl0ZSkL1y/k4SkPc3yf4/2OY4xJks6+ySdKsOMH5eXlXHPNNXz7299OyDKtIIRh1/5d\n3L7wdk4beBq3nnar33GMMd3AihUrWLhwIXPnuvt/9O/fn8svv5ypU6cmbJlWEMJw+8Lb2VO/h9mX\nzLbjBsaYpHjmmWfaPK+pqWHGjBmMGDEiYcu0YwidmL92Pr9f8Xt+OOmHjCpM3ulfxhjT3uzZs0lL\nS9xm2wpCB/bU7+GWV27hlMJTmDlxpt9xjDEmoWyXUQe++9p32VG3g5eveTlpN6kxxhi/WAshhFU7\nVjHngzl854zvMG7gOL/jGGNMwllBCOHet+8lNyvXdhUZY7oNKwhBrNqxiudWPcdtp99G3559/Y5j\njPGBdnIl6FQUa2YrCEG0tg7uOPMOv6MYY3yQk5NDdXV1lyoK6t1TOScn+ottJvSeyh1NKyJ9gGdx\nl77eDFylqp9F/UripLV1MHPiTPr17Od3HGOMD4qKiqioqIjp/sTxVl9f3+nGPicnh6KioqiXkdB7\nKncy7UzgTVX9mYjM9J5/L+pXEifWOjDGZGZmUlJS4neMNsrKyhg7dmxClxHOLqND91RW1Qag9Z7K\ngYLeU7mTaacCT3ndTwGXRv8y4qOyppI/rP4Dt4y7xVoHxphuJ9H3VO5o2kJVrfS6q4DCiJInwNPL\nn6ZFW7jx1Bv9jmKMMUkXrx+mZQDjCLinsoi8G+7EqqoiEvToTbt7KlNWVhZVwNra2g6nVVUeW/IY\nJ+WfROXKSiqpDDluPHWWyy+WKzKWKzKWK3JJydbZPTaJ4Z7KHU0LrAUGeN0DgLWdZUnkPZWXbFui\n3IM+vvjxqJcRjVS/t2yqsVyRsVyRSdVcqrFlIxXuqdzJtPOBb3rd3/Tm4Zv/ef9/6JHRg6tPvtrP\nGMYY45uE3lMZINi03qx/BjwnIjcAW4Cr4vzawrbxs408vfxpbjv9No7qcZRfMYwxxlcJvadyqGm9\n/tW4Yw6+u++d+8hIy+CuL97ldxRjjPFNt/+l8t76vfx2+W/51phvMTB/oN9xjDHGN92+ICxYt4CG\n5gauHX3obAJ8AAAXK0lEQVSt31GMMcZX3b4gvLT2JQpzCzmj6Ay/oxhjjK+6dUE42HSQBesWMHXE\nVNKkW68KY4zp3gXhzU1vUttQy6Un+H7VDGOM8V23LggvffQS+Vn5fLnky35HMcYY33XbgtDc0swf\n1/6RC4ddSHZGtt9xjDHGd922ILxb8S476nbY7iJjjPF024Lw4kcvkpmWyYXDLvQ7ijHGpIRuWRBU\nlZc+eolzhpxDr+xefscxxpiU0C0LwtLKpWz4bAOXn3C531GMMSZlhFUQRGSKiKwVkfXe7S7bDz9b\nRPaKSLn3uNvrPyKgX7mI7BORGd6we0RkW8CwpO27+d2HvyM7PZuvnvzVZC3SGGNSXlzuqex5R1Uv\nDuyhqmuBMQHz2Qa8GDDKr1T15zHkj1hjcyNzV87lKyO+Qu+c3slctDHGpLR43VM5HOcAG1R1SxTT\nxs1rG15j5/6dXHuKXbvIGGMCxeueygBfEJHlIvJnETk5yPBpwNx2/W7zppkjIkm5EcHza57nqJyj\nmDJ0SjIWZ4wxXYa4u6t1MILIlcAUVb3Re34tMEFVpweM0wtoUdVa71jAw6o6LGB4FrAdOFlVP/X6\nFQK7AAXuxd1O8/ogyw+8p/K40tLSqF5obW0tubm5XPnulYwuGM3dJ90d1Xzirba2lry8PL9jHMFy\nRcZyRcZyRS6WbJMnT16qquM7HbGze2wSxj2Vg0yzGegX8Hwq8FoH4xcDKzvLEus9lcsry5V70N98\n8Juo5xNvqXoPV8sVGcsVGcsVuS5zT2UROUZExOs+HbcrqjpglGtot7tIRAYEPL0MWBlGlpgsXL8Q\ngPOPPz/RizLGmC4nLvdUBq4E/lVEmoADwDSvKiEiubgzlG5uN+sHRGQMbpfR5iDD427hhoWMLhzN\ngPwBnY9sjDHdTFzuqayqjwCPhJi2DugbpH9ST/PZ37Sfv37yV+48885kLtYYY7qMbvNL5WV7ltHU\n0mRnFxljTAjdpiAs3r2YvKw8vnDsF/yOYowxKalbFARV5f3P3ufLJV8mKz3L7zjGGJOSukVB+Lj6\nY6rqq5hyvO0uMsaYULpFQTh0uulQO93UGGNC6RYFoaq2ipLcEoYcNcTvKMYYk7LCOu20q7vv3Ps4\nN/1cv2MYY0xK6xYtBIB0Sfc7gjHGpLROL26XSkRkJxDt5bP74S6ml2osV2QsV2QsV2RSNRfElu04\nVT26s5G6VEGIhYgs0XCu9pdklisylisylisyqZoLkpOt2+wyMsYY0zErCMYYY4DuVRCe8DtACJYr\nMpYrMpYrMqmaC5KQrdscQzDGGNOx7tRCMMYY04FuURBEZIqIrBWR9SIy08ccx4rIIhFZLSKrROR2\nr/89IrJNRMq9x4U+ZNssIiu85S/x+vURkddFZJ3396gkZxoRsE7KRWSfiMzwY32JyBwR2SEiKwP6\nhVw/IvJ97/O2VkQSds2UELkeFJGPRGS5iLwoIr29/sUiciBgvc0KPeeE5Ar5vvm8vp4NyLRZRMq9\n/slcX6G2Dcn9jIVzn82u/MDd5W0DMATIAj4ETvIpywDgVK87H/gYOAm4B/iuz+tpMwH3wfb6PQDM\n9LpnAvf7/D5WAcf5sb6AScCpBNz7O9T68d7TD4FsoMT7/KUnMdc/ARle9/0BuYoJ497lCcwV9H3z\ne321G/4L4G4f1leobUNSP2PdoYVwOrBeVTeqagNQCkz1I4iqVqrqMq+7BlgDDPIjS5imAk953U8B\nl/qY5Rxgg6pG+8PEmKjq28Dudr1DrZ+pQKmqHlTVTcB63OcwKblU9TVVbfKevgsUJWLZkebqgK/r\nq5V3X/iraHf/92ToYNuQ1M9YdygIg4CtAc8rSIGNsIgUA2OB97xet3lN/DnJ3jXjUeANEVkqIjd5\n/QpVtdLrrgIKfcjVahpt/1H9Xl8Qev2k0mfueuDPAc9LvN0ffxGRs3zIE+x9S5X1dRbwqaquC+iX\n9PXVbtuQ1M9YdygIKUdE8oAXgBmqug94HLdLawxQiWu2JttEVR0DXADcKiKTAgeqa6f6ckqaiGQB\nlwB/8Hqlwvpqw8/1E4qI/ABoAp7xelUCg733+Q7g9yLSK4mRUu59a+ca2n7pSPr6CrJtOCQZn7Hu\nUBC2AccGPC/y+vlCRDJxb/gzqjoPQFU/VdVmVW0B/pcENZc7oqrbvL87gBe9DJ+KyAAv9wBgR7Jz\neS4Alqnqp15G39eXJ9T68f0zJyL/DFwMfN3bkODtXqj2upfi9jsPT1amDt63VFhfGcDlwLOt/ZK9\nvoJtG0jyZ6xL/Q6hX79+WlxcHNW0dXV15ObmxjdQHFiuyFiuyFiuyKRqLogt29KlS3dpGBe3S/jR\n83g+xo0bp9F6+5VXop42kRYtWuR3hKAsV2QsV2QsV+RiyQYsUTvLyHPrrZx2ww3QhVpDxhiTbN2j\nIJx8MjmffgobN/qdxBhjUlb3KAjnerfPfOMNf3MYY0wK6xb3VGbYMOr79yfnjTfg5pv9TmOMiUJj\nYyMVFRXU19cnbBkFBQWsWbMmYfOPRTjZcnJyKCoqIjMzM6pldI+CIMJnp57KgLfeguZmSLf7KxvT\n1VRUVJCfn09xcTHuR8XxV1NTQ35+fkLmHavOsqkq1dXVVFRUUFJSEtUyuscuI+CzceNg924oL/c7\nijEmCvX19fTt2zdhxaCrExH69u0bUwuq+xSEU091Ha+/7m8QY0zUrBh0LNb1020KQmOfPjBqFLz5\npt9RjDEmJXWbggC4s43eeQcOHPA7iTHGpJzuVxAOHoS//93vJMaYLurXv/41t956a9zmt2LFCo47\n7jgef/zxhC4nHN2rIEyaBBkZ9nsEY0zUVqxYwahRo+I2v1GjRlFaWspvf/vbhC4nHN2rIOTlwRln\n2HEEY0zUli9fHvcNdf/+/Vm1alXCl9OZ7vE7hEDnnw933w3btsEg3++TY4yJxowZ8T+FfMwYuPfe\nTkdbuXIlI0eOjOuiZ86cycGDB9myZQvHHXdcwpbTme7VQgC4+mp3kbtnn+18XGOMCbB161by8/Mp\nKCho0/++++7ju9/9Lh999NGhfueeey4jR4484vHHP/6xzbR//vOfqaur46KLLjrUSgi1nECagIt1\ndr8WwrBhMG4c/P73cMcdfqcxxkTjoYcSM9+amg4HB9uv/9577zF37lxuuukmTjjhhEP93wjjWGV9\nfT3f+973mD9/Pr/5zW9YuXIlF154YcjjB1VVVVx22WVceumlXHfddQwYMCDMFxae7tdCAPja12Dp\nUli3rvNxjTHGE2y//ogRI/jSl77E9OnTI57fT37yE6677jqKi4sZNWoUK1euDLkcgPLycq655hq+\n973vxb0YQAIKgojcIyLbvBtTl4vIhSHG2ywiK7xxlsQ7R4euvhpEbLeRMSYiK1as4IknnqC4uJji\n4mLOPPNMysvLGT16dMTzWrt2La+//jozZswAaFMQgi0HXEE477zz4veC2knULqNfqerPwxhvsqru\nSlCG0AYNggkTYP58+OEPk754Y0zX9MwzzxzR76GHHmLixIkRz2vEiBG89957bZ4vW7Ys5HJqampY\nt24dI0aMiHhZ4eqeu4wALrkEFi+G7dv9TmKM6cJmzJjB+PHjk7Ks2bNnk5aWuM22xPtItYjcA3wL\n2AssAe5U1c+CjLfJG6cZ+LWqPhFifjcBNwEUFhaOKy0tjSpXbW0teXl5h57nbtrEaddfz9o77qDy\nK1+Jap7x0D5XqrBckbFckYkmV0FBAUOHDk1QIqe5uZn0FL08frjZ1q9fz969e9v0mzx58lJV7bxq\nhXPj5fYP4A1gZZDHVKAQSMe1Pv4LmBNiHoO8v/2BD4FJnS133LhxUd9k+ogbVLe0qJaUqF50UdTz\njIdUvam35YqM5YpMNLlWr14d/yDt7Nu3L+HLiFa42YKtJ2CJhrFtj+oYgqqeG854IvK/wCsh5rHN\n+7tDRF4ETgfejiZPVETcbqNZs6CuDnJzk7ZoY4xJRYk4yyjwXKjLcC2H9uPkikh+azfwT8HGS7hL\nLnEXu7NrGxljTEIOKj/gnU66HJgMfAdARAaKyAJvnELgryLyIfA+8CdVXZiALB076ywoKHBnGxlj\nUp4m4Ne5nyexrp+4n3aqqteG6L8duNDr3ghEfuJuvGVmwoUXwssv272WjUlxOTk5VFdX2200Q1Dv\nnso5OTlRz6P7XbqivUsugblz4f33wfvxhzEm9RQVFVFRUcHOnTsTtoz6+vqYNqiJFE62nJwcioqK\nol6GFYQpU9w9EubOtYJgTArLzMykpKQkocsoKytj7NixCV1GtJKRrfv+MK1V797w9a/Dk0/Cjh1+\npzHGGN9YQQD4j/9wZxv94hd+JzHGGN9YQQAYPhymTYNHH4V9+/xOY4wxvrCC0Oq229wP1ObN8zuJ\nMcb4wgpCqwkTYOhQ+N3v/E5ijDG+sILQSgS+8Q1YtAgqKvxOY4wxSWcFIdA3vuHut2ytBGNMN2QF\nIdDxx8O557r7tdbV+Z3GGGOSygpCe/fc436P8PjjficxxpiksoLQ3he/COedBw88YK0EY0y3YgUh\nmB//GHbudL9LMMaYbsIKQjBnngnnnw8PPgi1tX6nMcaYpLCCEMqPfwy7drkDzMYY0w1YQQhlwgS4\n4gr4yU9gzRq/0xhjTMJZQejIo49CXh788z9DU5PfaYwxJqGsIHSksNAVhffftyuhGmM+96wgdOaq\nq+DKK+Huu2HVKr/TGGNMwlhB6IwIPPYYFBTYriNjzOeaFYRwHH2023W0ZAn8/Od+pzHGmISwghCu\nr37V7Tr60Y9cYTDGmM8ZKwiReOwxOOYYmDoVtm/3O40xxsSVFYRIHH00zJ8Pe/fCBRe4y1sYY8zn\nhBWESI0eDS++CB9/DF/+Mmzd6nciY4yJCysI0TjvPFiwALZsgXHjXLeq36mMMSYmVhCiNXmy+8Fa\n375w0UUwcSK8+qoVBmNMl2UFIRYnnAAffOBOSf3kE5gyxV0p1VoMxpguyApCrHJy4N/+Ddavh1//\nGqqqXIvh9NPdsYbmZr8TGmNMWKwgxEt2Ntx0kzvY/OSTUF0Nl18OQ4bAXXfBX/9qxcEYk9KsIMRb\nVhbccIMrDC+8ACedBA8/DGed5X7DcMUV7sY777wDn3xC2oEDtnvJGJMSMvwO8LmVkeFaCJdfDvv2\nwcKF8Kc/wd/+BvPmHRptErgi0qePO0Ddp8/hR+DzYN09e0JDA6Snu+UZY0wMbCuSDL16uaumXnWV\ne75jByxeDJWVbFi8mOOPOsrtYtq92z02bnSXx9i9Gw4cCD3ftDRoaXHdeXlul1R6OuTmukfPnu4Y\nR04O9OgRvLv1kZ7u5iMCIhy3aZMrXiJt+rfpFnEZWgtS6zxUXS6Rw/1bH2lpbR/t+3UyTq9Vq9zr\naj9OWpp7/c3N7gKEgbvnAvMGPjoa1tnwdsNytm+HTZvCmzYnx71fe/bA/v2d54jhkdbQAAcPdv46\n462zVq9q23ESlcNEJCEFQURuA24FmoE/qeq/BxlnCvAwkA48qao/S0SWlNS/vzvwDGwdOpTjzz47\n9LgHDhwuFIFFo7ratTxyc6Gx0f16Oj3dbQz374e6Ove3vt49amvdL6tbn9fXu3nX17sNRjslCXrp\nsTrV7wAhnOF3gBAm+R0ghLM7GhhYUFv/huoOt1/Pnu7R1OT+X9pftdgb9wuNja7F3j5LCnQX3HYb\ndLStiIO4FwQRmQxMBUar6kER6R9knHTgUeA8oAJYLCLzVXV1vPN0eT16wKBB7pEord/WAr61/aWs\njC9NmtR2ePvulhb3aP+tPC3NfYhV2w5rHb91msDnYfZb/sEHnDJyZPBxWlshra2S1gzBHsFed7jD\ngwxbs3o1J55wQnjzrq+Hmhro3dsV9M7Gj+GxccMGhpSUdD5uJN/Qwx23g/E2bd5MSXHx4dceuA46\n6hft8P373SMjAzIz3d/WfAEtlZ3btjFo4MAj+qdCd1N+PomWiBbCvwI/U9WDAKq6I8g4pwPrVXUj\ngIiU4oqIFQQ/BNl1oBkZbb8ppYjdmZkJ/5YUjU/LyjgxBXN9UlbGkBTMtaWsjJIUzLWurIxBKZgL\noK6sLOHLEI3zGS4iUg78EZgC1APfVdXF7ca5Epiiqjd6z68FJqjq9CDzuwm4CaCwsHBcaWlpVLlq\na2vJy8uLatpEslyRsVyRsVyRSdVcEFu2yZMnL1XV8Z2NF1ULQUTeAI4JMugH3jz74HarngY8JyJD\nNMrKo6pPAE94y905efLkLdHMB+gH7Ipy2kSyXJGxXJGxXJFJ1VwQW7bjwhkpqoKgqueGGiYi/wrM\n8wrA+yLSgnshgdeK3gYcG/C8yOvX2XKPjiavl2tJOBUy2SxXZCxXZCxXZFI1FyQnWyJ+mPYSMBlA\nRIYDWRxZ1RYDw0SkRESygGnA/ARkMcYYE6ZEFIQ5wBARWQmUAt9UVRWRgSKyAEBVm4DpwKvAGuA5\nVV2VgCzGGGPCFPezjFS1AfhGkP7bgQsDni8AFsR7+R14IonLioTliozliozlikyq5oIkZIv7WUbG\nGGO6Jru4nTHGGKCbFAQRmSIia0VkvYjM9DHHsSKySERWi8gqEbnd63+PiGwTkXLvcWFn80pAts0i\nssJb/hKvXx8ReV1E1nl/j0pyphEB66RcRPaJyAw/1peIzBGRHd6xsdZ+IdePiHzf+7ytFZHzk5zr\nQRH5SESWi8iLItLb618sIgcC1tusJOcK+b75vL6eDci02fstVbLXV6htQ3I/Y6r6uX7grpW0ARiC\nO+PpQ+Akn7IMAE71uvOBj4GTgHtwP+Dzcz1tBvq16/cAMNPrngnc7/P7WIU7nzrp6wt3WaBTgZWd\nrR/vPf0QyMZdFmoDkJ7EXP8EZHjd9wfkKg4cz4f1FfR983t9tRv+C+BuH9ZXqG1DUj9j3aGFcOgy\nGeoOeLdeJiPpVLVSVZd53TW4M6wSeJGimE0FnvK6nwIu9THLOcAGVY32h4kxUdW3gd3teodaP1OB\nUlU9qKqbgPW4z2FScqnqa+rO5AN4F/c7n6QKsb5C8XV9tRIRAa4C5iZi2R3pYNuQ1M9YdygIg4Ct\nAc8rSIGNsIgUA2OB97xet3lN/DnJ3jXjUeANEVnqXS4EoFBVK73uKqDQh1ytptH2H9Xv9QWh108q\nfeauB/4c8LzE2/3xFxE5y4c8wd63VFlfZwGfquq6gH5JX1/ttg1J/Yx1h4KQckQkD3gBmKGq+4DH\ncbu0xgCVuGZrsk1U1THABcCtItLmysnq2qm+nJIm7seLlwB/8Hqlwvpqw8/1E4qI/ABoAp7xelUC\ng733+Q7g9yLSK4mRUu59a+ca2n7pSPr6CrJtOCQZn7HuUBCiukxGoohIJu4Nf0ZV5wGo6qeq2qyq\nLcD/kqDmckdUdZv3dwfwopfhUxEZ4OUeAAS7cm0yXAAsU9VPvYy+ry9PqPXj+2dORP4ZuBj4urch\nwdu9UO11L8Xtdx6erEwdvG+psL4ygMuBZ1v7JXt9Bds2kOTPWHcoCClzmQxvH+VsYI2q/jKg/4CA\n0S4DVrafNsG5ckUkv7Ubd1ByJW49fdMb7Zu4q9j6oc03N7/XV4BQ62c+ME1EskWkBBgGvJ+sUOJu\nPvXvwCWquj+g/9Hi7kWCiAzxcm1MYq5Q75uv68tzLvCRqla09kjm+gq1bSDZn7FkHEH3+4H7hfTH\nuAr/Ax9zTMQ1+ZYD5d7jQuB3wAqv/3xgQJJzDcGdsfAhsKp1HQF9gTeBdcAbQB8f1lkuUA0UBPRL\n+vrCFaRKoBG3v/aGjtYP7sq/G4C1wAVJzrUet3+59TM2yxv3Cu/9LQeWAV9Jcq6Q75uf68vr/3/A\nLe3GTeb6CrVtSOpnzH6pbIwxBugeu4yMMcaEwQqCMcYYwAqCMcYYjxUEY4wxgBUEY4wxHisIxhhj\nACsIxhhjPFYQjDHGAPD/AZEzsPNibEcKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb07302e710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "16280/16280 [==============================] - 0s - loss: 0.6343     \n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "\n",
    "for i in range(201):\n",
    "    l = DRf.evaluate(X_test, [y_test, gender_test], verbose=0)    \n",
    "    losses[\"L_f - L_r\"].append(l[0][None][0])\n",
    "    losses[\"L_f\"].append(l[1][None][0])\n",
    "    losses[\"L_r\"].append(-l[2][None][0])\n",
    "    print(losses[\"L_r\"][-1] / lam)\n",
    "    \n",
    "    if i % 5 == 0:\n",
    "        plot_losses(i, losses)\n",
    "\n",
    "    # Fit D\n",
    "    make_trainable(R, False)\n",
    "    make_trainable(D, True)\n",
    "    indices = np.random.permutation(len(X_train))[:batch_size]\n",
    "    DRf.train_on_batch(X_train[indices], [y_train[indices], gender_train[indices]])\n",
    "        \n",
    "    # Fit R\n",
    "    make_trainable(R, True)\n",
    "    make_trainable(D, False)\n",
    "    DfR.fit(X_train, gender_train, batch_size=batch_size, nb_epoch=1, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86212246538727966"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = D.predict(X_test)\n",
    "roc_auc_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performance is slightly worse, but as the plot and the pearson correlation coefficient show below, the distribution of the classifier output is now almost independent of gender. The classifier is now fair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE/lJREFUeJzt3X9sXWd9x/HP12lamzq1aZway7euMzbKSrO2OCNsiZCd\nqV1JWdEGAtoNNNgWTRNrhVapYX8sRtOE988C3YABBbFpJO4EbSnJYKpUe11NO2hY27QNrQpY2c0y\npXFXE0Ncmua7P3wzQu85vudcnx9+7n2/pKva5xzf833q6HOPn/M8zzF3FwAgHB1lFwAASIfgBoDA\nENwAEBiCGwACQ3ADQGAIbgAIzHlJDjKzWUknJb0i6bS7b86zKABAvETBXTPm7idyqwQAkAhdJQAQ\nGEsyc9LMfihpXktdJZ91989FHLNT0k5J6urqGrn00kubKujMKy+rY83a+h0v/0Ra+5qm3nO1O3Pm\njDo62usztN3a3G7tlWhzWs8+++wJd9+Q6GB3b/iSNFj77yWSHpf0tuWOHxkZ8WZN7d0TvWP3RU2/\n52o3NTVVdgmFa7c2t1t73WlzWpIe9QR57O7Jukrc/Wjtv8cl3SPpLek/TwAAWWgY3GZ2oZmtO/u1\npOskPZl3YQCAaElGlfRLusfMzh6/192/mWtVAIBYDYPb3X8g6aoCagGAprz88suqVqtaXFwstY6e\nnh4dPnx42WM6OztVqVS0dm3EIIyE0ozjBoBVqVqtat26dRoeHlatd6AUJ0+e1Lp162L3u7vm5uZU\nrVa1cePGps/TXmN1ALSkxcVFrV+/vtTQTsLMtH79+hX/ZUBwA2gJqz20z8qiToIbAAJDHzeAlrN1\n4gEdffFUZu832NulmV3blz3GzPSe97xHd911lyTp9OnTGhgY0JYtW7R///7MapEIbgAt6OiLpzQ7\ncUNm7ze860DDYy688EIdPnxYp06dUldXl+6//34NDg5mVsO56CoBgIxcd911OnBgKeT37dunm266\nKZfzENwAkJF3vetdmpyc1OLiop544glt2bIll/MQ3ACQkSuvvFKzs7Pat2+fduzYkdt56OMGgAzd\neOONuu222zQ9Pa25ublczkFwA0CGPvShD6m3t1ebNm3S9PR0LucguAG0nMHerkQjQdK8X1KVSkW3\n3HJLZueOQnADaDmNxlznYWFhQSdPnvy5baOjoxodHc38XNycBIDAENwAEJhgukqq3qfKeE/9jp4h\n6SOHii8IAEoSTHBve+mO6CmsUWEOAC2MrhIACAzBDQCBCaarBAAS27NJmj+S3fsluJe2Zs0avelN\nb1JHx9L18L333qvh4eHsajgHwQ2g9cwfkcbns3u/BPfSurq6NDMzs+wzJ7NCVwkABIYrbgDIwKlT\np7R161Z1dHRo48aNuueee3I7F8ENABmgqwQAEIvgBoDA0FUCoPX0DGU7q7pnKLv3ygDBDaD1lLB+\nUdSyrnmhqwQAAkNwA0BgCG4ALcHdyy4hkSzqJLgBBK+zs1Nzc3OrPrzdXXNzc+rs7FzR+3BzEkDw\nKpWKqtWqnn/++VLrWFxcbBjKnZ2dqlQqKzoPwQ0geGvXrtXGjRvLLkPT09O65pprcj8PXSUAEJjE\nwW1ma8zsP81sf54FAQCWl+aK+1ZJh/MqBACQTKLgNrOKpBsk3ZlvOQCARizJ8Bkz+4qkj0taJ+k2\nd39HxDE7Je2UpP7+/pHJycmmClp44bi6L76kbvuho/PaNBix9sCxx6SBq5s612qxsLCg7u7ussso\nVLu1ud3aK9HmtMbGxg66++ZEB7v7si9J75D06drXo5L2N/qZkZERb9bU3j2R2y+7fX/0D+y+qOlz\nrRZTU1Nll1C4dmtzu7XXnTanJelRb5CtZ19Jukq2SrrRzGYlTUrabmb/lP7zBACQhYbB7e4fdfeK\nuw9Lep+kB9z993KvDAAQiXHcABCYVDMn3X1a0nQulQAAEuGKGwACQ3ADQGAIbgAIDMENAIEJZlnX\nwd4uDe86ULd9dmXrkQNAcIIJ7pld26N3jBdaBgCUjq4SAAgMwQ0AgSG4ASAwBDcABIbgBoDAENwA\nEBiCGwACQ3ADQGAIbgAIDMENAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAhPMetxxqt6nynhP/Y6e\nIekjh4ovCAByFnxwb3vpDs1O3FC/IyrMAaAF0FUCAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAkNw\nA0BgCG4ACAzBDQCBIbgBIDAENwAEpmFwm1mnmX3bzB43s6fM7GNFFAYAiJZkkamXJG139wUzWyvp\nITP7hrs/knNtAIAIDYPb3V3SQu3btbWX51kUACCeLeVyg4PM1kg6KOkXJX3K3W+POGanpJ2S1N/f\nPzI5OdlUQQsvHFf3xZckPv7Q0XltGoxYwvXYY9LA1U3VULSFhQV1d3eXXUah2q3N7dZeiTanNTY2\ndtDdNyc62N0TvyT1SpqSdOVyx42MjHizpvbuSXX8Zbfvj96x+6Kmayja1NRU2SUUrt3a3G7tdafN\naUl61BNmcapRJe7+Yi24r0/3WQIAyEqSUSUbzKy39nWXpGslfS/vwgAA0ZKMKhmQ9A+1fu4OSf/s\n7vvzLQsAECfJqJInJF1TQC0AgASYOQkAgSG4ASAwBDcABIbgBoDAENwAEBiCGwACQ3ADQGAIbgAI\nTJKZk6vaYG+XhncdqNs+21lCMQBQgOCDe2bX9ugd44WWAQCFoasEAAJDcANAYAhuAAgMwQ0AgSG4\nASAwBDcABIbgBoDABD+OO07V+1QZ76nf0TMkfeRQ8QUBQEZaNri3vXSHZiduqN8RFeYAEBC6SgAg\nMAQ3AASG4AaAwBDcABAYghsAAkNwA0BgCG4ACAzBDQCBIbgBIDAENwAEhuAGgMC07FolsXqGotcr\nYfEpAIFo2eAe7O3S8K4DEds/qZnxiCfDs/gUgEC0bHDP7IoIZykyzAEgJPRxA0BgCG4ACEzD4Daz\nS81sysyeNrOnzOzWIgoDAERL0sd9WtKfuft3zWydpINmdr+7P51zbQCACA2vuN39mLt/t/b1SUmH\nJQ3mXRgAIJq5e/KDzYYlPSjpSnf/0av27ZS0U5L6+/tHJicnmypo4YXj6r74kqZ+Noln/uekfvrK\nmbrtmzp+KA1cndt5l7OwsKDu7u5Szl2Wdmtzu7VXos1pjY2NHXT3zYkOdvdEL0ndkg5K+p1Gx46M\njHizpvbuafpnV2T3ReWc192npqZKO3dZ2q3N7dZed9qclqRHPWEeJxrHbWZrJX1V0pfd/e6mPk5W\nuar3qcKMSgABaBjcZmaSviDpsLv/Tf4llWPbS3doduKG+h3MqASwyiQZx71V0vslbTezx2qvHTnX\nBQCI0fCK290fkmQF1AIASICZkwAQGIIbAALTsqsDphW3DOxsZwnFAMAyCO6auGVgNV5oGQDQEF0l\nABAYrrgTiH6STlf8VToA5IjgTiBqYg5P0gFQFrpKACAwBDcABIaukkZ6hiLXK3nogj5JEWubAEDO\nCO5GYlYGjFxJEAAKQFcJAASG4AaAwNBVsgKM7wZQBoJ7BRjfDaAMBHezGG0CoCQEd7MYbQKgJNyc\nBIDAENwAEBiCGwACQ3ADQGAIbgAIDKNKirRnkzR/pH77FR+XNFp0NQACRXAXaf6IND5fv33fJ4qv\nBUCwCO4cxM2e5InxALJAcOcgaiq8JJ4YDyAT3JwEgMBwxV2wqG6UL11VQiEAgkVwZy1m8amz+2bH\n67tRprk5CSAFgjtrMYtPLedlPy867HuGmno/AK2N4F4FvueX6tqoYYKsNAggAsHdDuIm/nBFDwSJ\n4F4Fzl/TEXnTMrNx33ETf7iiB4JEcK8Cl79unWYnRut3jBddCYAQNBzHbWZfNLPjZvZkEQUBAJaX\nZALOlyRdn3MdAICEGnaVuPuDZjacfymIEtX3PdjbpZld20uoBsBqYO7e+KCl4N7v7lcuc8xOSTsl\nqb+/f2RycrKpghZeOK7uiy9p6mdDtbCwoO7u7vodxx6TBq6u23zo6Lw2Daa4sRjzPrHbCxDb5hbV\nbu2VaHNaY2NjB919c6KD3b3hS9KwpCeTHOvuGhkZ8WZN7d3T9M+GampqKnrH7osiN192+/50J4h5\nn9jtBYhtc4tqt/a60+a0JD3qCTOWUSUBGuztogsFaGME92oWs+7JTM+QNFE/cSZuHXAAraVhcJvZ\nPi09V6vPzKqSdrv7F/IuDIqf1RgzceahC26Rxm+u39EzlGFRAMqWZFTJTUUUgpWr2AkNL+6t2z7Y\n2aWZEuoBkA+6SlpM1NN3tk48kO+UegCFIrhDFLfmd0yXSNwNy+ruPlVYThYIDsEdooxC9b1dn9fR\nF0/VbZ9VRD85gFWD4G5jsUMHx6M3b514IDLoGYYIFIvgRmJ3nfojVTpP1G2vnuqT9P3iCwLaFMGN\nxCp2InJd78h+cgC5IbiRCWZyAsUhuBEp7fDBqGGIzOQE8kFwo17PUPTIkrgZmDHDEx+6oE9SfaAD\nWBmCG/XSDjeMOZ6+byAfSZ6AAwBYRbjiRq7i+rk/evUZjRZbCtAyCG7kKuqmpST97Ze/lup9mPwD\n/AzBjVKcv6Yj1RDCoy+eYuQKUENwoxSXv26dZidG67anDeKsngbEFT1CQnBjVVkuiKPEhWrcUrZc\n0aMVENzIT9zys5J0+cek8XfWbZ6RpKiJPhcMSUo+TDEu0OOCOO7pQaWORd+zSZo/Ur+dZXfbHsGN\n/CwXLtPT0k31657EynlM+HLrsKS5co/rcmlqFM38kciasvp/kXv30PGnIz+cj2mDfm3xk/mdtw0Q\n3AjDcg+PyPnqM00XSlyXy2cnv95UX3yeTy5K2z0UF/QPd96qAT1f/wNXfDzyg+eV3a/XbGf9Xzes\nMpkcwY0wxIXznk2lBHravvhmb8ZGDqccT1Lhzyx3ZR1lubZF13NzZEA/M/l1/X7k+3w+8sOKmbbJ\nEdwIW1w4ZxUCMVf6WfXFx/WtS1LV+1SJ2H5MGzSQon13eZ8qExFXsns2SeP1fegzPUPSREQbYo6P\nq/Onr5zR7MRvJa6z6jxKLymCG60pJnAf7tyg4V31h8d2P6QNjJQfGBU7oeHFvZH7Bnu7lj4gXmVg\n/LnI4+OurKO6JSTF96Ev91dMxPHbdh3QbPQZUtn20h0xV/Rcib8awY3WFBO4A+M9mXQ/pBZzo049\nQ5odz2bUStpH0cVahVe3jLP/eQQ32styNznzfP+YG3WtIK5P/KNXp1vDLu59zv41xDj7nyG40V7y\nvpqMe//p6XzPW6K4K97plG1e7i+GuH2x9whavF+c4AZaXd5/ZeRtuYlc6osd657n4/QyHa/fBIIb\naHWhX3kuU/97Jx7Q0ZjulTy7VuLGwKdd9bJZBDeAYKW9IdsqXSsEN4C2Ebe0QWhDDgluAK0npl88\nbrJQaAhuAK0nptsjq8lCZeNhwQAQGK64AbSNuEk+D3fGrP+ySm9aEtwA2kb8AzbiVmKMvmkZNzrl\njW/4qxXVlxTBDQApxY1OWbvvE4WcP1Fwm9n1kj4paY2kO919IteqAKBAabtQyh6d0jC4zWyNpE9J\nulZSVdJ3zOw+d3867+IAoAhpu1DKHp2S5Ir7LZKec/cfSJKZTUp6pySCG0BLS/uko6KYuy9/gNm7\nJV3v7n9Y+/79kra4+4dfddxOSTtr314u6Zkma+qTdKLJnw0VbW597dZeiTandZm7b0hyYGY3J939\nc5I+t9L3MbNH3X1zBiUFgza3vnZrr0Sb85RkAs5RSZee832ltg0AUIIkwf0dSb9kZhvN7HxJ75N0\nX75lAQDiNOwqcffTZvZhSf+qpeGAX3T3p3KsacXdLQGiza2v3dor0ebcNLw5CQBYXVhkCgACQ3AD\nQGBKCW4zu97MnjGz58xsV8R+M7M7avufMLM3l1FnlhK0+XdrbT1kZt8ys6vKqDNLjdp8znG/aman\na3MGgpakzWY2amaPmdlTZvZvRdeYtQT/tnvM7Otm9nitzR8so86smNkXzey4mT0Zsz///HL3Ql9a\nusH5fUm/IOl8SY9LuuJVx+yQ9A1JJumtkv6j6DpLaPOvS3pt7eu3t0ObzznuAUn/IundZdddwO+5\nV0uzjodq319Sdt0FtPnPJf117esNkl6QdH7Zta+gzW+T9GZJT8bszz2/yrji/v8p9O7+U0lnp9Cf\n652S/tGXPCKp18wGii40Qw3b7O7fcvf/rX37iBT8E5aS/J4l6U8lfVXS8SKLy0mSNt8s6W53PyJJ\n7h56u5O02SWtMzOT1K2l4D5dbJnZcfcHtdSGOLnnVxnBPSjpv875vlrblvaYkKRtzx9o6RM7ZA3b\nbGaDkn5b0mcKrCtPSX7Pb5D0WjObNrODZvaBwqrLR5I2/52kX5b035IOSbrV3c8UU14pcs8v1uNe\nZcxsTEvBva3sWgrwCUm3u/uZpYuxtnCepBFJvyGpS9LDZvaIuz9bblm5+k1Jj0naLun1ku43s393\n9x+VW1a4ygjuJFPoW22afaL2mNmvSLpT0tvdfa6g2vKSpM2bJU3WQrtP0g4zO+3u9xZTYuaStLkq\nac7dfyzpx2b2oKSrJIUa3Ena/EFJE77UAfycmf1Q0hslfbuYEguXe36V0VWSZAr9fZI+ULs7+1ZJ\n8+5+rOhCM9SwzWY2JOluSe9vkauvhm12943uPuzuw5K+IulPAg5tKdm/7a9J2mZm55nZayRtkXS4\n4DqzlKTNR7T0F4bMrF9Lq4f+oNAqi5V7fhV+xe0xU+jN7I9r+/9eSyMMdkh6TtJPtPSJHayEbf4L\nSeslfbp2BXraA15ZLWGbW0qSNrv7YTP7pqQnJJ3R0hOlIoeVhSDh7/kvJX3JzA5paaTF7e4e7HKv\nZrZP0qikPjOrStotaa1UXH4x5R0AAsPMSQAIDMENAIEhuAEgMAQ3AASG4AaAwBDcABAYghsAAvN/\n09XxvDHaq7EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb07306c198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(y_pred[gender_test == 1], bins=50, histtype=\"step\",  normed=1, label=\"M\")\n",
    "plt.hist(y_pred[gender_test == 0], bins=50, histtype=\"step\", normed=1, label=\"F\")\n",
    "plt.ylim(0, 5)\n",
    "plt.legend()\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.018571555562095266, 0.017802639756038973)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import pearsonr\n",
    "pearsonr(gender_test, D.predict(X_test).ravel())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}