logistic_regression_model_trainer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From Notebook to ModelOp Center:\n",
    "## Training, Evaluating, and Conforming a Model for Deployment\n",
    "\n",
    "\n",
    "In this notebook, we demonstrate the process of \n",
    "1. training a model, \n",
    "2. evaluating its performance, \n",
    "3. saving it for later use,\n",
    "4. and conforming it to MOC standards.\n",
    "\n",
    "More specifically, we will train a logistic regression classifier on the German Credit Data dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**I - Model Training**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by loading relevant libraries. We will need `sklearn` for model training, and `aequitas` for bias detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import json\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from aequitas.bias import Bias\n",
    "from aequitas.group import Group\n",
    "from aequitas.preprocessing import preprocess_input_df\n",
    "\n",
    "from sklearn import set_config\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, RepeatedStratifiedKFold\n",
    "from sklearn.metrics import make_scorer, accuracy_score, precision_score, recall_score, \\\n",
    "                            f1_score, fbeta_score, confusion_matrix\n",
    "\n",
    "# set_config(display='diagram')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **German Credit Data** dataset can be found here: https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data). Download it and load it from a *CSV* file. For our purposes, the dataset has been modified slightly to include an `id` column, and a `gender` column (engineered from `status_sex`, used to demonstarte bias). The target variable is under `label`. We have mapped the labels `[1,2]` to `[0,1]`, where `1` indicates the positive class (loan default)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"german_credit_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['id', 'duration_months', 'credit_amount', 'installment_rate',\n",
       "       'present_residence_since', 'age_years', 'number_existing_credits',\n",
       "       'checking_status', 'credit_history', 'purpose', 'savings_account',\n",
       "       'present_employment_since', 'debtors_guarantors', 'property',\n",
       "       'installment_plans', 'housing', 'job', 'number_people_liable',\n",
       "       'telephone', 'foreign_worker', 'gender', 'label'], dtype=object)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>duration_months</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>installment_rate</th>\n",
       "      <th>present_residence_since</th>\n",
       "      <th>age_years</th>\n",
       "      <th>number_existing_credits</th>\n",
       "      <th>checking_status</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>...</th>\n",
       "      <th>debtors_guarantors</th>\n",
       "      <th>property</th>\n",
       "      <th>installment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>job</th>\n",
       "      <th>number_people_liable</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "      <th>gender</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>1169</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>67</td>\n",
       "      <td>2</td>\n",
       "      <td>A11</td>\n",
       "      <td>A34</td>\n",
       "      <td>A43</td>\n",
       "      <td>...</td>\n",
       "      <td>A101</td>\n",
       "      <td>A121</td>\n",
       "      <td>A143</td>\n",
       "      <td>A152</td>\n",
       "      <td>A173</td>\n",
       "      <td>1</td>\n",
       "      <td>A192</td>\n",
       "      <td>A201</td>\n",
       "      <td>male</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>5951</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>A12</td>\n",
       "      <td>A32</td>\n",
       "      <td>A43</td>\n",
       "      <td>...</td>\n",
       "      <td>A101</td>\n",
       "      <td>A121</td>\n",
       "      <td>A143</td>\n",
       "      <td>A152</td>\n",
       "      <td>A173</td>\n",
       "      <td>1</td>\n",
       "      <td>A191</td>\n",
       "      <td>A201</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>2096</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "      <td>A14</td>\n",
       "      <td>A34</td>\n",
       "      <td>A46</td>\n",
       "      <td>...</td>\n",
       "      <td>A101</td>\n",
       "      <td>A121</td>\n",
       "      <td>A143</td>\n",
       "      <td>A152</td>\n",
       "      <td>A172</td>\n",
       "      <td>2</td>\n",
       "      <td>A191</td>\n",
       "      <td>A201</td>\n",
       "      <td>male</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>42</td>\n",
       "      <td>7882</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>A11</td>\n",
       "      <td>A32</td>\n",
       "      <td>A42</td>\n",
       "      <td>...</td>\n",
       "      <td>A103</td>\n",
       "      <td>A122</td>\n",
       "      <td>A143</td>\n",
       "      <td>A153</td>\n",
       "      <td>A173</td>\n",
       "      <td>2</td>\n",
       "      <td>A191</td>\n",
       "      <td>A201</td>\n",
       "      <td>male</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>4870</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>53</td>\n",
       "      <td>2</td>\n",
       "      <td>A11</td>\n",
       "      <td>A33</td>\n",
       "      <td>A40</td>\n",
       "      <td>...</td>\n",
       "      <td>A101</td>\n",
       "      <td>A124</td>\n",
       "      <td>A143</td>\n",
       "      <td>A153</td>\n",
       "      <td>A173</td>\n",
       "      <td>2</td>\n",
       "      <td>A191</td>\n",
       "      <td>A201</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  duration_months  credit_amount  installment_rate  \\\n",
       "0   0                6           1169                 4   \n",
       "1   1               48           5951                 2   \n",
       "2   2               12           2096                 2   \n",
       "3   3               42           7882                 2   \n",
       "4   4               24           4870                 3   \n",
       "\n",
       "   present_residence_since  age_years  number_existing_credits  \\\n",
       "0                        4         67                        2   \n",
       "1                        2         22                        1   \n",
       "2                        3         49                        1   \n",
       "3                        4         45                        1   \n",
       "4                        4         53                        2   \n",
       "\n",
       "  checking_status credit_history purpose  ... debtors_guarantors property  \\\n",
       "0             A11            A34     A43  ...               A101     A121   \n",
       "1             A12            A32     A43  ...               A101     A121   \n",
       "2             A14            A34     A46  ...               A101     A121   \n",
       "3             A11            A32     A42  ...               A103     A122   \n",
       "4             A11            A33     A40  ...               A101     A124   \n",
       "\n",
       "  installment_plans housing   job number_people_liable telephone  \\\n",
       "0              A143    A152  A173                    1      A192   \n",
       "1              A143    A152  A173                    1      A191   \n",
       "2              A143    A152  A172                    2      A191   \n",
       "3              A143    A153  A173                    2      A191   \n",
       "4              A143    A153  A173                    2      A191   \n",
       "\n",
       "   foreign_worker  gender label  \n",
       "0            A201    male     0  \n",
       "1            A201  female     1  \n",
       "2            A201    male     0  \n",
       "3            A201    male     0  \n",
       "4            A201    male     1  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all numeric columns need to be considered as numeric features. For example, `number_people_liable` only has two unique discrete values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    845\n",
       "2    155\n",
       "Name: number_people_liable, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.number_people_liable.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may therefore treat it as a categorical feature. Note, however, that we may need to reconsider this option if more values appear in testing phases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.number_people_liable = data.number_people_liable.astype('object')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding any further with model development, let us split the original dataset into two sets: a **baseline** set that will be used as a reference set, and a **sample** set which will mimic input data to the model once the model is in use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_baseline, df_sample = train_test_split(data, train_size=0.8, random_state=0)\n",
    "\n",
    "df_baseline.to_json('df_baseline.json', orient='records', lines=True)\n",
    "df_sample.to_json('df_sample.json', orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train a **Logistic Regression** classifier. Since our data contains categorical features, we will need to start our pipeline with an encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    OneHotEncoder(\n",
    "        handle_unknown='ignore', \n",
    "        sparse=True\n",
    "    ),\n",
    "    LogisticRegression(\n",
    "        max_iter=1000,\n",
    "        random_state=0\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Logistic Regression** has multiple parameters which can be tuned. Among these are `C`, `solver`, and `class_weight`, which will be optimized by **GridSearchCV**. We provide GridSearchCV a list of values for each of these parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = dict(\n",
    "    logisticregression__C=np.logspace(-4, 4, 50), # Inverse of regularization strength\n",
    "    logisticregression__solver=['liblinear', 'lbfgs', 'newton-cg'],\n",
    "    logisticregression__class_weight=['balanced', None]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data still contains non-predictive features, such as `id`, `label` and `gender` (excluded to remove explicit bias). We remove these below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictive_features = [\n",
    "    f for f in list(data.columns.values) \n",
    "    if f not in ['id', 'label', 'gender']\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check, let us see which features are automatically encoded as **numeric**, and which are encoded as **categorical**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = [\n",
    "    f for f in list(data.select_dtypes(include=['category', 'object'])) \n",
    "    if f in predictive_features\n",
    "]\n",
    "\n",
    "numerical_features = [\n",
    "    f for f in predictive_features \n",
    "    if f not in categorical_features\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Categorical features**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['checking_status', 'credit_history', 'purpose', 'savings_account', 'present_employment_since', 'debtors_guarantors', 'property', 'installment_plans', 'housing', 'job', 'number_people_liable', 'telephone', 'foreign_worker']\n"
     ]
    }
   ],
   "source": [
    "print(categorical_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Numerical features**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['duration_months', 'credit_amount', 'installment_rate', 'present_residence_since', 'age_years', 'number_existing_credits']\n"
     ]
    }
   ],
   "source": [
    "print(numerical_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything looks good; let us proceed with training. We need to specify **predictive** and **response** variables for each of the training and test sets. We set these by filtering the baseline and sample sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_baseline[predictive_features]\n",
    "X_test = df_sample[predictive_features]\n",
    "\n",
    "y_train = df_baseline['label']\n",
    "y_test = df_sample['label']\n",
    "\n",
    "X_train.to_json('X_train.json', orient='records', lines=True)\n",
    "X_test.to_json('X_test.json', orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may now fit the classifier to the training data. Since \"it is worse to classify a customer as good when they are bad, than it is to classify a customer as bad when they are good\", we will use an **F_beta metric**, with `beta=2`, to judge the performance of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=RepeatedStratifiedKFold(n_repeats=3, n_splits=10, random_state=0),\n",
       "             error_score=nan,\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('onehotencoder',\n",
       "                                        OneHotEncoder(categories='auto',\n",
       "                                                      drop=None,\n",
       "                                                      dtype=<class 'numpy.float64'>,\n",
       "                                                      handle_unknown='ignore',\n",
       "                                                      sparse=True)),\n",
       "                                       ('logisticregression',\n",
       "                                        LogisticRegression(C=1.0,\n",
       "                                                           class_weight=None,\n",
       "                                                           dual=False,\n",
       "                                                           fit_intercept=...\n",
       "       3.39322177e+02, 4.94171336e+02, 7.19685673e+02, 1.04811313e+03,\n",
       "       1.52641797e+03, 2.22299648e+03, 3.23745754e+03, 4.71486636e+03,\n",
       "       6.86648845e+03, 1.00000000e+04]),\n",
       "                         'logisticregression__class_weight': ['balanced', None],\n",
       "                         'logisticregression__solver': ['liblinear', 'lbfgs',\n",
       "                                                        'newton-cg']},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=make_scorer(fbeta_score, beta=2), verbose=0)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_GS = GridSearchCV(\n",
    "    estimator=pipeline, \n",
    "    param_grid=parameters,\n",
    "    n_jobs=-1,\n",
    "    scoring=make_scorer(fbeta_score, beta=2),\n",
    "    cv=RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=0)\n",
    ")\n",
    "clf_GS.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('onehotencoder',\n",
       "                 OneHotEncoder(categories='auto', drop=None,\n",
       "                               dtype=<class 'numpy.float64'>,\n",
       "                               handle_unknown='ignore', sparse=True)),\n",
       "                ('logisticregression',\n",
       "                 LogisticRegression(C=0.0020235896477251557,\n",
       "                                    class_weight='balanced', dual=False,\n",
       "                                    fit_intercept=True, intercept_scaling=1,\n",
       "                                    l1_ratio=None, max_iter=1000,\n",
       "                                    multi_class='auto', n_jobs=None,\n",
       "                                    penalty='l2', random_state=0,\n",
       "                                    solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                                    warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_GS.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the parameters of the best estimator more carefully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logisticregression__C': 0.0020235896477251557,\n",
       " 'logisticregression__class_weight': 'balanced',\n",
       " 'logisticregression__solver': 'lbfgs'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_GS.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that the best logistic regression classifier is one with a `solver='lbfgs'` and `class_weight='balanced'`. This classifier achived the best score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6720094431830962"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_GS.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**II - Model Evaluation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before saving our trained model for further use, let's look at some performance metrics. We will evaluate the model on both the training and test sets; we would like to see a stable performance.\n",
    "\n",
    "For repeatability, let's define a function which computes multiple metrics at-a-time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(y, y_preds):\n",
    "    \"\"\"\n",
    "    A function to evaluate a classification model\n",
    "    \n",
    "    param: y: true (actual) labels\n",
    "    param: y_preds: predicted labels (as scored by model)\n",
    "    \n",
    "    return: mutiple classification performance metrics\n",
    "    \"\"\"\n",
    "    \n",
    "    return [\n",
    "        accuracy_score(y, y_preds),\n",
    "        precision_score(y, y_preds),\n",
    "        recall_score(y, y_preds),\n",
    "        f1_score(y, y_preds),\n",
    "        fbeta_score(y, y_preds, beta=2),\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now compute predictions on both training and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_preds = clf_GS.best_estimator_.predict(X_test)\n",
    "y_train_preds = clf_GS.best_estimator_.predict(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will display performance metrics in a DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "preformance_df = pd.DataFrame(\n",
    "    data=[{}],\n",
    "    columns=['Accuracy', 'Precision', 'Recall', 'F1 score', 'F2 Score'],\n",
    "    index=['Training Set', 'Test Set']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "preformance_df.loc['Training Set',:] = compute_metrics(y=y_train, y_preds=y_train_preds)\n",
    "preformance_df.loc['Test Set',:] = compute_metrics(y=y_test, y_preds=y_test_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how our model performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1 score</th>\n",
       "      <th>F2 Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Training Set</th>\n",
       "      <td>0.725</td>\n",
       "      <td>0.531250</td>\n",
       "      <td>0.772727</td>\n",
       "      <td>0.629630</td>\n",
       "      <td>0.708333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Test Set</th>\n",
       "      <td>0.665</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>0.724138</td>\n",
       "      <td>0.556291</td>\n",
       "      <td>0.646154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Accuracy  Precision    Recall  F1 score  F2 Score\n",
       "Training Set     0.725   0.531250  0.772727  0.629630  0.708333\n",
       "Test Set         0.665   0.451613  0.724138  0.556291  0.646154"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preformance_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it's good to see that the performance on the training set is not too far off from the performance on the test set, further model improvements are needed to achieve better F2 scores. For now, we will contend with this model and use it to produce new predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**III - Saving and Loading the Trained Model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the model is **trained** and **evaluated**, we save it in a binary format. It will then be loaded and used to make new predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(clf_GS.best_estimator_, open(\"logreg_classifier.pickle\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is reloaded on-demand as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg_classifier = pickle.load(open(\"logreg_classifier.pickle\", 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predictions are produced on-demand by calling the `predict()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_preds = logreg_classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IV - Evaluating Bias on Protected Classes**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since `gender` is a protected class, we have excluded it from the list of predictive features. However, this does not guarantee that the model is not implicitly biased, as `gender` could potentially be inferred from other features. It is therefore imperative that we evaluate our model for Bias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To that end, let us produce some predictions and append them to our labeled baseline and sample sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_baseline_scored = df_baseline.copy(deep=True)\n",
    "df_baseline_scored[\"score\"] = logreg_classifier.predict(\n",
    "    df_baseline[predictive_features])\n",
    "\n",
    "df_sample_scored = df_sample.copy(deep=True)\n",
    "df_sample_scored[\"score\"] = logreg_classifier.predict(\n",
    "    df_sample[predictive_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aequitas library requires the true label to be encoded as 'label_value', so let us rename that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_baseline_scored.rename(columns={'label': 'label_value'}, inplace=True)\n",
    "df_sample_scored.rename(columns={'label': 'label_value'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save these two DataFrames before proceeding further:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_baseline_scored.to_json('df_baseline_scored.json', orient='records', lines=True)\n",
    "df_sample_scored.to_json('df_sample_scored.json', orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we call the aequitas preprocessing function on our datasets, filtered to the features we care about: `score` (prediction), `label_value` (true label), and `gender` (protected class):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_baseline_scored_processed, _ = preprocess_input_df(\n",
    "    df_baseline_scored.loc[:,['score', 'label_value', 'gender']]\n",
    ")\n",
    "df_sample_scored_processed, _ = preprocess_input_df(\n",
    "    df_sample_scored.loc[:,['score', 'label_value', 'gender']]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by computing some `Group` Metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_baseline, g_sample = Group(), Group()\n",
    "xtab_baseline, _ = g_baseline.get_crosstabs(df_baseline_scored_processed)\n",
    "xtab_sample, _ = g_sample.get_crosstabs(df_sample_scored_processed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "absolute_metrics_baseline = g_baseline.list_absolute_metrics(xtab_baseline)\n",
    "absolute_metrics_sample = g_sample.list_absolute_metrics(xtab_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the absolute metrics, computed on baseline and sample sets, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>tpr</th>\n",
       "      <th>tnr</th>\n",
       "      <th>for</th>\n",
       "      <th>fdr</th>\n",
       "      <th>fpr</th>\n",
       "      <th>fnr</th>\n",
       "      <th>npv</th>\n",
       "      <th>precision</th>\n",
       "      <th>ppr</th>\n",
       "      <th>pprev</th>\n",
       "      <th>prev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.34</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.51</td>\n",
       "      <td>0.66</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribute_name attribute_value   tpr   tnr   for   fdr   fpr   fnr   npv  \\\n",
       "0         gender          female  0.80  0.67  0.14  0.43  0.33  0.20  0.86   \n",
       "1         gender            male  0.76  0.72  0.12  0.49  0.28  0.24  0.88   \n",
       "\n",
       "   precision   ppr  pprev  prev  \n",
       "0       0.57  0.34   0.50  0.35  \n",
       "1       0.51  0.66   0.42  0.28  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtab_baseline[['attribute_name', 'attribute_value'] + absolute_metrics_baseline].round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>tpr</th>\n",
       "      <th>tnr</th>\n",
       "      <th>for</th>\n",
       "      <th>fdr</th>\n",
       "      <th>fpr</th>\n",
       "      <th>fnr</th>\n",
       "      <th>npv</th>\n",
       "      <th>precision</th>\n",
       "      <th>ppr</th>\n",
       "      <th>pprev</th>\n",
       "      <th>prev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.32</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.48</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribute_name attribute_value   tpr   tnr   for   fdr   fpr   fnr   npv  \\\n",
       "0         gender          female  0.68  0.70  0.20  0.45  0.30  0.32  0.80   \n",
       "1         gender            male  0.76  0.61  0.12  0.60  0.39  0.24  0.88   \n",
       "\n",
       "   precision   ppr  pprev  prev  \n",
       "0       0.55  0.33   0.43  0.35  \n",
       "1       0.40  0.67   0.48  0.26  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtab_sample[['attribute_name', 'attribute_value'] + absolute_metrics_sample].round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add some raw counts (group sizes) as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_id</th>\n",
       "      <th>score_threshold</th>\n",
       "      <th>k</th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>pp</th>\n",
       "      <th>pn</th>\n",
       "      <th>fp</th>\n",
       "      <th>fn</th>\n",
       "      <th>tn</th>\n",
       "      <th>tp</th>\n",
       "      <th>group_label_pos</th>\n",
       "      <th>group_label_neg</th>\n",
       "      <th>group_size</th>\n",
       "      <th>total_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>binary 0/1</td>\n",
       "      <td>352</td>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>118</td>\n",
       "      <td>120</td>\n",
       "      <td>51</td>\n",
       "      <td>17</td>\n",
       "      <td>103</td>\n",
       "      <td>67</td>\n",
       "      <td>84</td>\n",
       "      <td>154</td>\n",
       "      <td>238</td>\n",
       "      <td>800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>binary 0/1</td>\n",
       "      <td>352</td>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>234</td>\n",
       "      <td>328</td>\n",
       "      <td>114</td>\n",
       "      <td>38</td>\n",
       "      <td>290</td>\n",
       "      <td>120</td>\n",
       "      <td>158</td>\n",
       "      <td>404</td>\n",
       "      <td>562</td>\n",
       "      <td>800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   model_id score_threshold    k attribute_name attribute_value   pp   pn  \\\n",
       "0         0      binary 0/1  352         gender          female  118  120   \n",
       "1         0      binary 0/1  352         gender            male  234  328   \n",
       "\n",
       "    fp  fn   tn   tp  group_label_pos  group_label_neg  group_size  \\\n",
       "0   51  17  103   67               84              154         238   \n",
       "1  114  38  290  120              158              404         562   \n",
       "\n",
       "   total_entities  \n",
       "0             800  \n",
       "1             800  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtab_baseline[[col for col in xtab_baseline.columns if col not in absolute_metrics_baseline]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_id</th>\n",
       "      <th>score_threshold</th>\n",
       "      <th>k</th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>pp</th>\n",
       "      <th>pn</th>\n",
       "      <th>fp</th>\n",
       "      <th>fn</th>\n",
       "      <th>tn</th>\n",
       "      <th>tp</th>\n",
       "      <th>group_label_pos</th>\n",
       "      <th>group_label_neg</th>\n",
       "      <th>group_size</th>\n",
       "      <th>total_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>binary 0/1</td>\n",
       "      <td>93</td>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>31</td>\n",
       "      <td>41</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>33</td>\n",
       "      <td>17</td>\n",
       "      <td>25</td>\n",
       "      <td>47</td>\n",
       "      <td>72</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>binary 0/1</td>\n",
       "      <td>93</td>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>62</td>\n",
       "      <td>66</td>\n",
       "      <td>37</td>\n",
       "      <td>8</td>\n",
       "      <td>58</td>\n",
       "      <td>25</td>\n",
       "      <td>33</td>\n",
       "      <td>95</td>\n",
       "      <td>128</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   model_id score_threshold   k attribute_name attribute_value  pp  pn  fp  \\\n",
       "0         0      binary 0/1  93         gender          female  31  41  14   \n",
       "1         0      binary 0/1  93         gender            male  62  66  37   \n",
       "\n",
       "   fn  tn  tp  group_label_pos  group_label_neg  group_size  total_entities  \n",
       "0   8  33  17               25               47          72             200  \n",
       "1   8  58  25               33               95         128             200  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtab_sample[[col for col in xtab_sample.columns if col not in absolute_metrics_sample]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it for `Group` metrics. Let's move on to `Bias` metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "get_disparity_predefined_group()\n",
      "get_disparity_predefined_group()\n"
     ]
    }
   ],
   "source": [
    "b_baseline, b_sample = Bias(), Bias()\n",
    "\n",
    "bdf_baseline = b_baseline.get_disparity_predefined_groups(\n",
    "    xtab_baseline, \n",
    "    original_df=df_baseline_scored_processed, \n",
    "    ref_groups_dict={'gender':'male'}, alpha=0.05, mask_significance=True\n",
    ")\n",
    "\n",
    "bdf_sample = b_sample.get_disparity_predefined_groups(\n",
    "    xtab_sample, \n",
    "    original_df=df_sample_scored_processed, \n",
    "    ref_groups_dict={'gender':'male'}, alpha=0.05, mask_significance=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now compute **disparity** metrics as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "calculated_disparities_baseline = b_baseline.list_disparities(bdf_baseline)\n",
    "calculated_disparities_sample = b_sample.list_disparities(bdf_sample)\n",
    "\n",
    "disparity_metrics_df_baseline = bdf_baseline[\n",
    "    ['attribute_name', 'attribute_value'] + \\\n",
    "        calculated_disparities_baseline\n",
    "    ]\n",
    "disparity_metrics_df_sample = bdf_sample[\n",
    "    ['attribute_name', 'attribute_value'] + \\\n",
    "        calculated_disparities_sample\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the computed disparity metrics on baseline and sample sets, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>ppr_disparity</th>\n",
       "      <th>pprev_disparity</th>\n",
       "      <th>precision_disparity</th>\n",
       "      <th>fdr_disparity</th>\n",
       "      <th>for_disparity</th>\n",
       "      <th>fpr_disparity</th>\n",
       "      <th>fnr_disparity</th>\n",
       "      <th>tpr_disparity</th>\n",
       "      <th>tnr_disparity</th>\n",
       "      <th>npv_disparity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>0.504274</td>\n",
       "      <td>1.190763</td>\n",
       "      <td>1.107203</td>\n",
       "      <td>0.887154</td>\n",
       "      <td>1.222807</td>\n",
       "      <td>1.173616</td>\n",
       "      <td>0.841479</td>\n",
       "      <td>1.050198</td>\n",
       "      <td>0.931751</td>\n",
       "      <td>0.970805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribute_name attribute_value  ppr_disparity  pprev_disparity  \\\n",
       "0         gender          female       0.504274         1.190763   \n",
       "1         gender            male       1.000000         1.000000   \n",
       "\n",
       "   precision_disparity  fdr_disparity  for_disparity  fpr_disparity  \\\n",
       "0             1.107203       0.887154       1.222807       1.173616   \n",
       "1             1.000000       1.000000       1.000000       1.000000   \n",
       "\n",
       "   fnr_disparity  tpr_disparity  tnr_disparity  npv_disparity  \n",
       "0       0.841479       1.050198       0.931751       0.970805  \n",
       "1       1.000000       1.000000       1.000000       1.000000  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disparity_metrics_df_baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>ppr_disparity</th>\n",
       "      <th>pprev_disparity</th>\n",
       "      <th>precision_disparity</th>\n",
       "      <th>fdr_disparity</th>\n",
       "      <th>for_disparity</th>\n",
       "      <th>fpr_disparity</th>\n",
       "      <th>fnr_disparity</th>\n",
       "      <th>tpr_disparity</th>\n",
       "      <th>tnr_disparity</th>\n",
       "      <th>npv_disparity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>1.36</td>\n",
       "      <td>0.756757</td>\n",
       "      <td>1.609756</td>\n",
       "      <td>0.764807</td>\n",
       "      <td>1.32</td>\n",
       "      <td>0.8976</td>\n",
       "      <td>1.150037</td>\n",
       "      <td>0.915896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribute_name attribute_value  ppr_disparity  pprev_disparity  \\\n",
       "0         gender          female            0.5         0.888889   \n",
       "1         gender            male            1.0         1.000000   \n",
       "\n",
       "   precision_disparity  fdr_disparity  for_disparity  fpr_disparity  \\\n",
       "0                 1.36       0.756757       1.609756       0.764807   \n",
       "1                 1.00       1.000000       1.000000       1.000000   \n",
       "\n",
       "   fnr_disparity  tpr_disparity  tnr_disparity  npv_disparity  \n",
       "0           1.32         0.8976       1.150037       0.915896  \n",
       "1           1.00         1.0000       1.000000       1.000000  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disparity_metrics_df_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the disparity metrics above are worrisome! We might need to retrain the model, possibly with better feature engineering. That's an exercise for a later time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**V - Conforming Model Code to MOC Requirements**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conformance is best-demonstrated through and example. Let's look at the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pickle\n",
    "import copy\n",
    "from aequitas.preprocessing import preprocess_input_df\n",
    "from aequitas.group import Group\n",
    "from aequitas.bias import Bias\n",
    "\n",
    "\n",
    "# modelop.init\n",
    "def begin():\n",
    "    \n",
    "    global logreg_classifier\n",
    "    \n",
    "    # load pickled logistic regression model\n",
    "    logreg_classifier = pickle.load(open(\"logreg_classifier.pickle\", \"rb\"))\n",
    "\n",
    "    \n",
    "# modelop.score\n",
    "def action(data):\n",
    "    \n",
    "    # Turn data into DataFrame\n",
    "    data = pd.DataFrame([data])\n",
    "    \n",
    "    # There are only two unique values in data.number_people_liable.\n",
    "    # Treat it as a categorical feature\n",
    "    data.number_people_liable = data.number_people_liable.astype('object')\n",
    "\n",
    "    predictive_features = [\n",
    "        'duration_months', 'credit_amount', 'installment_rate',\n",
    "        'present_residence_since', 'age_years', 'number_existing_credits',\n",
    "        'checking_status', 'credit_history', 'purpose', 'savings_account',\n",
    "        'present_employment_since', 'debtors_guarantors', 'property',\n",
    "        'installment_plans', 'housing', 'job', 'number_people_liable',\n",
    "        'telephone', 'foreign_worker'\n",
    "    ]\n",
    "    \n",
    "    data[\"predicted_score\"] = logreg_classifier.predict(data[predictive_features])\n",
    "    \n",
    "    return data.to_dict(orient=\"records\")\n",
    "\n",
    "\n",
    "# modelop.metrics\n",
    "def metrics(data):\n",
    "    \n",
    "    data = pd.DataFrame(data)\n",
    "\n",
    "    # To measure Bias towards gender, filter DataFrame\n",
    "    # to \"score\", \"label_value\" (ground truth), and\n",
    "    # \"gender\" (protected attribute)\n",
    "    data_scored = data[[\"score\", \"label_value\", \"gender\"]]\n",
    "\n",
    "    # Process DataFrame\n",
    "    data_scored_processed, _ = preprocess_input_df(data_scored)\n",
    "\n",
    "    # Group Metrics\n",
    "    g = Group()\n",
    "    xtab, _ = g.get_crosstabs(data_scored_processed)\n",
    "\n",
    "    # Absolute metrics, such as 'tpr', 'tnr','precision', etc.\n",
    "    absolute_metrics = g.list_absolute_metrics(xtab)\n",
    "\n",
    "    # DataFrame of calculated absolute metrics for each sample population group\n",
    "    absolute_metrics_df = xtab[\n",
    "        ['attribute_name', 'attribute_value'] + absolute_metrics].round(2)\n",
    "\n",
    "    # For example:\n",
    "    \"\"\"\n",
    "        attribute_name  attribute_value     tpr     tnr  ... precision\n",
    "    0   gender          female              0.60    0.88 ... 0.75\n",
    "    1   gender          male                0.49    0.90 ... 0.64\n",
    "    \"\"\"\n",
    "\n",
    "    # Bias Metrics\n",
    "    b = Bias()\n",
    "\n",
    "    # Disparities calculated in relation gender for \"male\" and \"female\"\n",
    "    bias_df = b.get_disparity_predefined_groups(\n",
    "        xtab,\n",
    "        original_df=data_scored_processed,\n",
    "        ref_groups_dict={'gender': 'male'},\n",
    "        alpha=0.05, mask_significance=True\n",
    "    )\n",
    "\n",
    "    # Disparity metrics added to bias DataFrame\n",
    "    calculated_disparities = b.list_disparities(bias_df)\n",
    "\n",
    "    disparity_metrics_df = bias_df[\n",
    "        ['attribute_name', 'attribute_value'] + calculated_disparities]\n",
    "\n",
    "    # For example:\n",
    "    \"\"\"\n",
    "        attribute_name\tattribute_value    ppr_disparity   precision_disparity\n",
    "    0   gender          female             0.714286        1.41791\n",
    "    1   gender          male               1.000000        1.000000\n",
    "    \"\"\"\n",
    "\n",
    "    output_metrics_df = disparity_metrics_df # or absolute_metrics_df\n",
    "\n",
    "    # Output a JSON object of calculated metrics\n",
    "    \n",
    "    return output_metrics_df.to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four main sections that are standard to almost any model in MOC:\n",
    "1. Library imports\n",
    "2. `init` function\n",
    "3. `score` function\n",
    "4. `metrics` function\n",
    "\n",
    "**Library** imports are always at the top. We don't need to include all libraries that we used for training and model evaluation. We just need the libraries for processing and scoring.\n",
    "\n",
    "The **`init`** function runs once per deployment, and is used to load and persist into memory any variable that needs to be accessed at scoring time. For example, the init function is where we load the saved model binary. We make the variable global so it can be accessed from the scoring function.\n",
    "\n",
    "The **`score`** function is the function that runs anytime we make a scoring (prediction) request. This is where we put our prediction code. We have to remember to include any steps that were not captured by the pipeline, such as feature engineering or re-encoding.\n",
    "\n",
    "The **`metrics`** functions is where model evaluation is carried out. In our example, this is the place where we replicate the calculations of Group and/or Bias metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us test our source code to see if we missed anything. We will load input data and scored input data to test both the scoring and metrics functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_sample = pd.read_json('df_baseline.json', lines=True, orient='records')\n",
    "metrics_sample = pd.read_json('df_baseline_scored.json', lines=True, orient='records')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that the **`init`** function can load the trained model binary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "begin()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No errors from the **`init`** function. Let us now call the **`score`** function on input data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = action(test_sample.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>duration_months</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>installment_rate</th>\n",
       "      <th>present_residence_since</th>\n",
       "      <th>age_years</th>\n",
       "      <th>number_existing_credits</th>\n",
       "      <th>checking_status</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>...</th>\n",
       "      <th>property</th>\n",
       "      <th>installment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>job</th>\n",
       "      <th>number_people_liable</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "      <th>gender</th>\n",
       "      <th>label</th>\n",
       "      <th>predicted_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>687</td>\n",
       "      <td>36</td>\n",
       "      <td>2862</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>A12</td>\n",
       "      <td>A33</td>\n",
       "      <td>A40</td>\n",
       "      <td>...</td>\n",
       "      <td>A124</td>\n",
       "      <td>A143</td>\n",
       "      <td>A153</td>\n",
       "      <td>A173</td>\n",
       "      <td>1</td>\n",
       "      <td>A191</td>\n",
       "      <td>A201</td>\n",
       "      <td>male</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    id  duration_months  credit_amount  installment_rate  \\\n",
       "0  687               36           2862                 4   \n",
       "\n",
       "   present_residence_since  age_years  number_existing_credits  \\\n",
       "0                        3         30                        1   \n",
       "\n",
       "  checking_status credit_history purpose  ... property installment_plans  \\\n",
       "0             A12            A33     A40  ...     A124              A143   \n",
       "\n",
       "  housing   job number_people_liable telephone foreign_worker  gender label  \\\n",
       "0    A153  A173                    1      A191           A201    male     0   \n",
       "\n",
       "  predicted_score  \n",
       "0               1  \n",
       "\n",
       "[1 rows x 23 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(scores).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have scores! Last but not least, let's call the **`metrics`** function on scored data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bias = metrics(metrics_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attribute_name</th>\n",
       "      <th>attribute_value</th>\n",
       "      <th>ppr_disparity</th>\n",
       "      <th>pprev_disparity</th>\n",
       "      <th>precision_disparity</th>\n",
       "      <th>fdr_disparity</th>\n",
       "      <th>for_disparity</th>\n",
       "      <th>fpr_disparity</th>\n",
       "      <th>fnr_disparity</th>\n",
       "      <th>tpr_disparity</th>\n",
       "      <th>tnr_disparity</th>\n",
       "      <th>npv_disparity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gender</td>\n",
       "      <td>female</td>\n",
       "      <td>0.504274</td>\n",
       "      <td>1.190763</td>\n",
       "      <td>1.107203</td>\n",
       "      <td>0.887154</td>\n",
       "      <td>1.222807</td>\n",
       "      <td>1.173616</td>\n",
       "      <td>0.841479</td>\n",
       "      <td>1.050198</td>\n",
       "      <td>0.931751</td>\n",
       "      <td>0.970805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gender</td>\n",
       "      <td>male</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  attribute_name attribute_value  ppr_disparity  pprev_disparity  \\\n",
       "0         gender          female       0.504274         1.190763   \n",
       "1         gender            male       1.000000         1.000000   \n",
       "\n",
       "   precision_disparity  fdr_disparity  for_disparity  fpr_disparity  \\\n",
       "0             1.107203       0.887154       1.222807       1.173616   \n",
       "1             1.000000       1.000000       1.000000       1.000000   \n",
       "\n",
       "   fnr_disparity  tpr_disparity  tnr_disparity  npv_disparity  \n",
       "0       0.841479       1.050198       0.931751       0.970805  \n",
       "1       1.000000       1.000000       1.000000       1.000000  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(bias).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prefect!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.8 64-bit ('german_credit_398_env')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "ffe27ecf9aefb015465c8e22db62d00aad3a9132150f6b09a8bc3c86528aa33e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}