User Credit Data Project - Solution Development.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Solution Development for User Credit Data Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering the problem in the dataset, Lenders would definitely benefit from having some sort of application which can predict whether users will Default or not. This would definitely help the employees in determining whether or not to give someone credit/loans.\n",
    "To fulfill this requirement, I will build a model to predict defaulters and non-defaulters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import scipy.stats\n",
    "from statsmodels.stats.outliers_influence import variance_inflation_factor\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, StratifiedKFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from imblearn.over_sampling import SMOTE\n",
    "#Import Random Forest \n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score, f1_score, precision_score, recall_score, make_scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read data from csv\n",
    "fixed_df = pd.read_csv('data/fixed-data.csv',index_col=0)\n",
    "pd.set_option('display.float_format', lambda x: '%.5f' % x) #set option to fix annoying scientific notation in pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>payment_ratio</th>\n",
       "      <th>overlimit_percentage</th>\n",
       "      <th>payment_ratio_3month</th>\n",
       "      <th>payment_ratio_6month</th>\n",
       "      <th>years_since_card_issuing</th>\n",
       "      <th>remaining_bill_per_number_of_cards</th>\n",
       "      <th>remaining_bill_per_limit</th>\n",
       "      <th>total_usage_per_limit</th>\n",
       "      <th>total_3mo_usage_per_limit</th>\n",
       "      <th>total_6mo_usage_per_limit</th>\n",
       "      <th>...</th>\n",
       "      <th>outstanding</th>\n",
       "      <th>credit_limit</th>\n",
       "      <th>total_retail_usage</th>\n",
       "      <th>number_of_cards</th>\n",
       "      <th>X</th>\n",
       "      <th>X.1</th>\n",
       "      <th>x</th>\n",
       "      <th>branch_code</th>\n",
       "      <th>default_flag</th>\n",
       "      <th>delinquency_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.02190</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.74780</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>15.41667</td>\n",
       "      <td>13161.50000</td>\n",
       "      <td>0.00376</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.01172</td>\n",
       "      <td>0.01781</td>\n",
       "      <td>...</td>\n",
       "      <td>36158</td>\n",
       "      <td>7000000.00000</td>\n",
       "      <td>94.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1-a</td>\n",
       "      <td>I</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.75000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00010</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>...</td>\n",
       "      <td>268691</td>\n",
       "      <td>10000000.00000</td>\n",
       "      <td>1012.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2-a</td>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.00910</td>\n",
       "      <td>10.75000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.04052</td>\n",
       "      <td>0.04770</td>\n",
       "      <td>...</td>\n",
       "      <td>6769149</td>\n",
       "      <td>28000000.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3-a</td>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.95990</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.97490</td>\n",
       "      <td>0.99840</td>\n",
       "      <td>1.66667</td>\n",
       "      <td>2975932.50000</td>\n",
       "      <td>0.59519</td>\n",
       "      <td>0.26666</td>\n",
       "      <td>0.32303</td>\n",
       "      <td>0.13116</td>\n",
       "      <td>...</td>\n",
       "      <td>9402085</td>\n",
       "      <td>10000000.00000</td>\n",
       "      <td>2666558.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5-a</td>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.18470</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.24950</td>\n",
       "      <td>0.17890</td>\n",
       "      <td>4.66667</td>\n",
       "      <td>1657023.00000</td>\n",
       "      <td>0.82851</td>\n",
       "      <td>0.05760</td>\n",
       "      <td>0.01875</td>\n",
       "      <td>0.16667</td>\n",
       "      <td>...</td>\n",
       "      <td>3906290</td>\n",
       "      <td>4000000.00000</td>\n",
       "      <td>230400.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7-a</td>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   payment_ratio  overlimit_percentage  payment_ratio_3month  \\\n",
       "0        1.02190               0.00000               0.74780   \n",
       "1        0.00000               0.00000               0.00000   \n",
       "2        1.00000               0.00000               1.00000   \n",
       "4        0.95990               0.00000               0.97490   \n",
       "6        0.18470               0.00000               0.24950   \n",
       "\n",
       "   payment_ratio_6month  years_since_card_issuing  \\\n",
       "0               1.00000                  15.41667   \n",
       "1               0.00000                   0.75000   \n",
       "2               1.00910                  10.75000   \n",
       "4               0.99840                   1.66667   \n",
       "6               0.17890                   4.66667   \n",
       "\n",
       "   remaining_bill_per_number_of_cards  remaining_bill_per_limit  \\\n",
       "0                         13161.50000                   0.00376   \n",
       "1                             0.00000                   0.00000   \n",
       "2                             0.00000                   0.00000   \n",
       "4                       2975932.50000                   0.59519   \n",
       "6                       1657023.00000                   0.82851   \n",
       "\n",
       "   total_usage_per_limit  total_3mo_usage_per_limit  \\\n",
       "0                0.00001                    0.01172   \n",
       "1                0.00010                    0.00000   \n",
       "2                0.00000                    0.04052   \n",
       "4                0.26666                    0.32303   \n",
       "6                0.05760                    0.01875   \n",
       "\n",
       "   total_6mo_usage_per_limit  ...  outstanding   credit_limit  \\\n",
       "0                    0.01781  ...        36158  7000000.00000   \n",
       "1                    0.00000  ...       268691 10000000.00000   \n",
       "2                    0.04770  ...      6769149 28000000.00000   \n",
       "4                    0.13116  ...      9402085 10000000.00000   \n",
       "6                    0.16667  ...      3906290  4000000.00000   \n",
       "\n",
       "   total_retail_usage  number_of_cards  X  X.1    x branch_code default_flag  \\\n",
       "0            94.00000                2  1    1  1-a           I            0   \n",
       "1          1012.00000                2  2    2  2-a           A            0   \n",
       "2             0.00000                3  3    3  3-a           A            0   \n",
       "4       2666558.00000                2  5    5  5-a           A            0   \n",
       "6        230400.00000                2  7    7  7-a           A            0   \n",
       "\n",
       "   delinquency_score  \n",
       "0            0.00000  \n",
       "1            0.00000  \n",
       "2            0.00000  \n",
       "4            0.00000  \n",
       "6            0.00000  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fixed_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>payment_ratio</th>\n",
       "      <th>overlimit_percentage</th>\n",
       "      <th>payment_ratio_3month</th>\n",
       "      <th>payment_ratio_6month</th>\n",
       "      <th>years_since_card_issuing</th>\n",
       "      <th>remaining_bill_per_number_of_cards</th>\n",
       "      <th>remaining_bill_per_limit</th>\n",
       "      <th>total_usage_per_limit</th>\n",
       "      <th>total_3mo_usage_per_limit</th>\n",
       "      <th>total_6mo_usage_per_limit</th>\n",
       "      <th>utilization_6month</th>\n",
       "      <th>outstanding</th>\n",
       "      <th>credit_limit</th>\n",
       "      <th>total_retail_usage</th>\n",
       "      <th>number_of_cards</th>\n",
       "      <th>X</th>\n",
       "      <th>X.1</th>\n",
       "      <th>default_flag</th>\n",
       "      <th>delinquency_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>count</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "      <td>7461.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>mean</td>\n",
       "      <td>0.49218</td>\n",
       "      <td>0.05823</td>\n",
       "      <td>0.53582</td>\n",
       "      <td>0.59363</td>\n",
       "      <td>5.74686</td>\n",
       "      <td>1190503.27634</td>\n",
       "      <td>0.32280</td>\n",
       "      <td>0.04390</td>\n",
       "      <td>0.10054</td>\n",
       "      <td>0.12665</td>\n",
       "      <td>0.41660</td>\n",
       "      <td>3743940.50824</td>\n",
       "      <td>10322677.92521</td>\n",
       "      <td>387402.28803</td>\n",
       "      <td>2.21981</td>\n",
       "      <td>7761.63249</td>\n",
       "      <td>7761.63249</td>\n",
       "      <td>0.07345</td>\n",
       "      <td>0.03163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>std</td>\n",
       "      <td>0.47773</td>\n",
       "      <td>0.31863</td>\n",
       "      <td>0.36651</td>\n",
       "      <td>0.40482</td>\n",
       "      <td>3.71463</td>\n",
       "      <td>1617503.06540</td>\n",
       "      <td>0.36469</td>\n",
       "      <td>0.07012</td>\n",
       "      <td>0.10809</td>\n",
       "      <td>0.14228</td>\n",
       "      <td>0.34472</td>\n",
       "      <td>4080476.91620</td>\n",
       "      <td>8514204.36631</td>\n",
       "      <td>654254.53673</td>\n",
       "      <td>0.51746</td>\n",
       "      <td>4555.06907</td>\n",
       "      <td>4555.06907</td>\n",
       "      <td>0.26089</td>\n",
       "      <td>0.34395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>min</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.75000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>3000000.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25%</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.20400</td>\n",
       "      <td>0.20000</td>\n",
       "      <td>2.75000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.01360</td>\n",
       "      <td>0.01290</td>\n",
       "      <td>0.09290</td>\n",
       "      <td>750719.00000</td>\n",
       "      <td>5000000.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>2.00000</td>\n",
       "      <td>3799.00000</td>\n",
       "      <td>3799.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50%</td>\n",
       "      <td>0.28400</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>0.59800</td>\n",
       "      <td>5.17000</td>\n",
       "      <td>462871.00000</td>\n",
       "      <td>0.14200</td>\n",
       "      <td>0.00400</td>\n",
       "      <td>0.06530</td>\n",
       "      <td>0.07580</td>\n",
       "      <td>0.34900</td>\n",
       "      <td>2590903.00000</td>\n",
       "      <td>7000000.00000</td>\n",
       "      <td>45500.00000</td>\n",
       "      <td>2.00000</td>\n",
       "      <td>7724.00000</td>\n",
       "      <td>7724.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75%</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.86200</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>7.92000</td>\n",
       "      <td>1846563.00000</td>\n",
       "      <td>0.65830</td>\n",
       "      <td>0.06200</td>\n",
       "      <td>0.15400</td>\n",
       "      <td>0.19500</td>\n",
       "      <td>0.70000</td>\n",
       "      <td>5050904.00000</td>\n",
       "      <td>13000000.00000</td>\n",
       "      <td>502000.00000</td>\n",
       "      <td>2.00000</td>\n",
       "      <td>11773.00000</td>\n",
       "      <td>11773.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>max</td>\n",
       "      <td>2.44000</td>\n",
       "      <td>2.65000</td>\n",
       "      <td>1.95410</td>\n",
       "      <td>2.16000</td>\n",
       "      <td>18.80000</td>\n",
       "      <td>7743339.00000</td>\n",
       "      <td>1.59000</td>\n",
       "      <td>0.32000</td>\n",
       "      <td>0.54400</td>\n",
       "      <td>0.66225</td>\n",
       "      <td>1.77000</td>\n",
       "      <td>23423624.00000</td>\n",
       "      <td>47000000.00000</td>\n",
       "      <td>3533381.00000</td>\n",
       "      <td>4.00000</td>\n",
       "      <td>15642.00000</td>\n",
       "      <td>15642.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>5.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       payment_ratio  overlimit_percentage  payment_ratio_3month  \\\n",
       "count     7461.00000            7461.00000            7461.00000   \n",
       "mean         0.49218               0.05823               0.53582   \n",
       "std          0.47773               0.31863               0.36651   \n",
       "min          0.00000               0.00000               0.00000   \n",
       "25%          0.00000               0.00000               0.20400   \n",
       "50%          0.28400               0.00000               0.50000   \n",
       "75%          1.00000               0.00000               0.86200   \n",
       "max          2.44000               2.65000               1.95410   \n",
       "\n",
       "       payment_ratio_6month  years_since_card_issuing  \\\n",
       "count            7461.00000                7461.00000   \n",
       "mean                0.59363                   5.74686   \n",
       "std                 0.40482                   3.71463   \n",
       "min                 0.00000                   0.75000   \n",
       "25%                 0.20000                   2.75000   \n",
       "50%                 0.59800                   5.17000   \n",
       "75%                 1.00000                   7.92000   \n",
       "max                 2.16000                  18.80000   \n",
       "\n",
       "       remaining_bill_per_number_of_cards  remaining_bill_per_limit  \\\n",
       "count                          7461.00000                7461.00000   \n",
       "mean                        1190503.27634                   0.32280   \n",
       "std                         1617503.06540                   0.36469   \n",
       "min                               0.00000                   0.00000   \n",
       "25%                               0.00000                   0.00000   \n",
       "50%                          462871.00000                   0.14200   \n",
       "75%                         1846563.00000                   0.65830   \n",
       "max                         7743339.00000                   1.59000   \n",
       "\n",
       "       total_usage_per_limit  total_3mo_usage_per_limit  \\\n",
       "count             7461.00000                 7461.00000   \n",
       "mean                 0.04390                    0.10054   \n",
       "std                  0.07012                    0.10809   \n",
       "min                  0.00000                    0.00000   \n",
       "25%                  0.00000                    0.01360   \n",
       "50%                  0.00400                    0.06530   \n",
       "75%                  0.06200                    0.15400   \n",
       "max                  0.32000                    0.54400   \n",
       "\n",
       "       total_6mo_usage_per_limit  utilization_6month    outstanding  \\\n",
       "count                 7461.00000          7461.00000     7461.00000   \n",
       "mean                     0.12665             0.41660  3743940.50824   \n",
       "std                      0.14228             0.34472  4080476.91620   \n",
       "min                      0.00000             0.00000        0.00000   \n",
       "25%                      0.01290             0.09290   750719.00000   \n",
       "50%                      0.07580             0.34900  2590903.00000   \n",
       "75%                      0.19500             0.70000  5050904.00000   \n",
       "max                      0.66225             1.77000 23423624.00000   \n",
       "\n",
       "        credit_limit  total_retail_usage  number_of_cards           X  \\\n",
       "count     7461.00000          7461.00000       7461.00000  7461.00000   \n",
       "mean  10322677.92521        387402.28803          2.21981  7761.63249   \n",
       "std    8514204.36631        654254.53673          0.51746  4555.06907   \n",
       "min    3000000.00000             0.00000          1.00000     1.00000   \n",
       "25%    5000000.00000             0.00000          2.00000  3799.00000   \n",
       "50%    7000000.00000         45500.00000          2.00000  7724.00000   \n",
       "75%   13000000.00000        502000.00000          2.00000 11773.00000   \n",
       "max   47000000.00000       3533381.00000          4.00000 15642.00000   \n",
       "\n",
       "              X.1  default_flag  delinquency_score  \n",
       "count  7461.00000    7461.00000         7461.00000  \n",
       "mean   7761.63249       0.07345            0.03163  \n",
       "std    4555.06907       0.26089            0.34395  \n",
       "min       1.00000       0.00000            0.00000  \n",
       "25%    3799.00000       0.00000            0.00000  \n",
       "50%    7724.00000       0.00000            0.00000  \n",
       "75%   11773.00000       0.00000            0.00000  \n",
       "max   15642.00000       1.00000            5.00000  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fixed_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>payment_ratio</th>\n",
       "      <th>overlimit_percentage</th>\n",
       "      <th>payment_ratio_3month</th>\n",
       "      <th>payment_ratio_6month</th>\n",
       "      <th>years_since_card_issuing</th>\n",
       "      <th>remaining_bill_per_number_of_cards</th>\n",
       "      <th>remaining_bill_per_limit</th>\n",
       "      <th>total_usage_per_limit</th>\n",
       "      <th>total_3mo_usage_per_limit</th>\n",
       "      <th>total_6mo_usage_per_limit</th>\n",
       "      <th>utilization_6month</th>\n",
       "      <th>outstanding</th>\n",
       "      <th>credit_limit</th>\n",
       "      <th>total_retail_usage</th>\n",
       "      <th>number_of_cards</th>\n",
       "      <th>X</th>\n",
       "      <th>branch_code</th>\n",
       "      <th>default_flag</th>\n",
       "      <th>delinquency_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.02190</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.74780</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>15.41667</td>\n",
       "      <td>13161.50000</td>\n",
       "      <td>0.00376</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.01172</td>\n",
       "      <td>0.01781</td>\n",
       "      <td>0.02195</td>\n",
       "      <td>36158</td>\n",
       "      <td>7000000.00000</td>\n",
       "      <td>94.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>I</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.75000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00010</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00030</td>\n",
       "      <td>268691</td>\n",
       "      <td>10000000.00000</td>\n",
       "      <td>1012.00000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   payment_ratio  overlimit_percentage  payment_ratio_3month  \\\n",
       "0        1.02190               0.00000               0.74780   \n",
       "1        0.00000               0.00000               0.00000   \n",
       "\n",
       "   payment_ratio_6month  years_since_card_issuing  \\\n",
       "0               1.00000                  15.41667   \n",
       "1               0.00000                   0.75000   \n",
       "\n",
       "   remaining_bill_per_number_of_cards  remaining_bill_per_limit  \\\n",
       "0                         13161.50000                   0.00376   \n",
       "1                             0.00000                   0.00000   \n",
       "\n",
       "   total_usage_per_limit  total_3mo_usage_per_limit  \\\n",
       "0                0.00001                    0.01172   \n",
       "1                0.00010                    0.00000   \n",
       "\n",
       "   total_6mo_usage_per_limit  utilization_6month  outstanding   credit_limit  \\\n",
       "0                    0.01781             0.02195        36158  7000000.00000   \n",
       "1                    0.00000             0.00030       268691 10000000.00000   \n",
       "\n",
       "   total_retail_usage  number_of_cards  X branch_code  default_flag  \\\n",
       "0            94.00000                2  1           I             0   \n",
       "1          1012.00000                2  2           A             0   \n",
       "\n",
       "   delinquency_score  \n",
       "0            0.00000  \n",
       "1            0.00000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fixed_df = fixed_df.drop(['X.1','x'],axis=1)\n",
    "fixed_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Statement / Hypothesis\n",
    "\n",
    "This is a classification problem, as we need to identify which users belong to which group, from a total of 2 groups:\n",
    "    1. Defaulters     (1)\n",
    "    2. Non-Defaulters (0)\n",
    "Before I start, I define the baseline result here (taken from: https://machinelearningmastery.com/how-to-get-baseline-results-and-why-they-matter/):\n",
    "\n",
    "### Baseline Result for Classification Problem\n",
    "Since the classes are imbalanced and has more observations for non-defaulters, we \n",
    "select the class that has the most observations and use that class as the result for all predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7461\n",
      "0    6913\n",
      "1     548\n",
      "Name: default_flag, dtype: int64\n",
      "Baseline Classification Accuracy: 92.65514006165392\n"
     ]
    }
   ],
   "source": [
    "print(fixed_df['default_flag'].count()) #Total number of rows after removing outliers\n",
    "print(fixed_df['default_flag'].value_counts()) #Number of users who default vs who don't default\n",
    "\n",
    "print('Baseline Classification Accuracy:', 6913/7461 * 100 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Determine default_flag as my y dataset\n",
    "y = fixed_df.default_flag.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be helpful to reduce the number of independent variables. Therefore, at this point, I check for Multi-Collinearity using VIF (Variance Inflation Factor). \n",
    "And I will use the threshold of VIF > 10 to remove the variables that are indicated to be multicollinear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking for multi-collinearity steps\n",
    "df_model = pd.DataFrame() #create temp model to store our features\n",
    "for cols in fixed_df.columns:\n",
    "    if cols not in ['X', 'x', 'branch_code', 'default_flag']:\n",
    "        df_model[cols] = fixed_df[cols]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    VIF Factor                            features\n",
      "0      4.75589                       payment_ratio\n",
      "1      1.13676                overlimit_percentage\n",
      "2     11.62869                payment_ratio_3month\n",
      "3      6.69783                payment_ratio_6month\n",
      "4      4.00360            years_since_card_issuing\n",
      "5      9.49250  remaining_bill_per_number_of_cards\n",
      "6      8.59087            remaining_bill_per_limit\n",
      "7      4.96582               total_usage_per_limit\n",
      "8      3.83862           total_3mo_usage_per_limit\n",
      "9      3.25759           total_6mo_usage_per_limit\n",
      "10     5.77062                  utilization_6month\n",
      "11     8.38131                         outstanding\n",
      "12     4.98473                        credit_limit\n",
      "13     5.04616                  total_retail_usage\n",
      "14    11.45693                     number_of_cards\n",
      "15     1.01766                   delinquency_score\n"
     ]
    }
   ],
   "source": [
    "#Checking the VIF/ Multi-collinearity amongst the model dataframe\n",
    "vif = pd.DataFrame()\n",
    "vif[\"VIF Factor\"] = [variance_inflation_factor(df_model.values, i) for i in range(df_model.shape[1])]\n",
    "vif[\"features\"] = df_model.columns\n",
    "print(vif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Fix VIF \n",
    "df_model = df_model.drop('payment_ratio_3month',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    VIF Factor                            features\n",
      "0      3.29303                       payment_ratio\n",
      "1      1.13600                overlimit_percentage\n",
      "2      4.12117                payment_ratio_6month\n",
      "3      4.00010            years_since_card_issuing\n",
      "4      9.48794  remaining_bill_per_number_of_cards\n",
      "5      8.48698            remaining_bill_per_limit\n",
      "6      4.96492               total_usage_per_limit\n",
      "7      3.65325           total_3mo_usage_per_limit\n",
      "8      3.24599           total_6mo_usage_per_limit\n",
      "9      5.71856                  utilization_6month\n",
      "10     8.32397                         outstanding\n",
      "11     4.95581                        credit_limit\n",
      "12     5.03382                  total_retail_usage\n",
      "13    11.34698                     number_of_cards\n",
      "14     1.01766                   delinquency_score\n"
     ]
    }
   ],
   "source": [
    "#Calculate new VIF\n",
    "#Checking the VIF/ Multi-collinearity amongst the model dataframe\n",
    "vif = pd.DataFrame()\n",
    "vif[\"VIF Factor\"] = [variance_inflation_factor(df_model.values, i) for i in range(df_model.shape[1])]\n",
    "vif[\"features\"] = df_model.columns\n",
    "print(vif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_model = df_model.drop('number_of_cards',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    VIF Factor                            features\n",
      "0      3.20752                       payment_ratio\n",
      "1      1.13355                overlimit_percentage\n",
      "2      3.75560                payment_ratio_6month\n",
      "3      3.09702            years_since_card_issuing\n",
      "4      8.97238  remaining_bill_per_number_of_cards\n",
      "5      7.50145            remaining_bill_per_limit\n",
      "6      4.90205               total_usage_per_limit\n",
      "7      3.64620           total_3mo_usage_per_limit\n",
      "8      3.23625           total_6mo_usage_per_limit\n",
      "9      5.47827                  utilization_6month\n",
      "10     8.25821                         outstanding\n",
      "11     3.94774                        credit_limit\n",
      "12     4.97482                  total_retail_usage\n",
      "13     1.01178                   delinquency_score\n"
     ]
    }
   ],
   "source": [
    "#Calculate new VIF\n",
    "#Checking the VIF/ Multi-collinearity amongst the model dataframe\n",
    "vif = pd.DataFrame()\n",
    "vif[\"VIF Factor\"] = [variance_inflation_factor(df_model.values, i) for i in range(df_model.shape[1])]\n",
    "vif[\"features\"] = df_model.columns\n",
    "print(vif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solved Multicollinearity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting Data into Train and Test\n",
    "\n",
    "For our classification problem, we split the Data into Training Data and Test Data. With a ratio of 70 % Training and 30 % Testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TRAINING (70%) and TESTING (30%)\n",
    "# Stratified on y\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_model, y, test_size = 0.30, random_state = 5, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My data is imbalanced, and leaning heavily to non-defaulters. I will try to use the SMOTE method (https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/) in order to balance this out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scale X matrix with StandardScaler\n",
    "ss = StandardScaler()\n",
    "X_scaled_train = ss.fit_transform(X_train)\n",
    "\n",
    "X_scaled_test = ss.transform(X_test)\n",
    "\n",
    "# SMOTE the training set as the data set is skewed towards having more non_defaulters\n",
    "sm = SMOTE(random_state = 3, sampling_strategy = 'minority')\n",
    "X_scaled_sm_train, y_sm_train = sm.fit_sample(X_scaled_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Classifier Model\n",
    "I will use Random Forest Classifier to classify this problem of Defaulters vs Non-Defaulters. Random Forest is also selected because of its Features Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,\n",
       "                       criterion='gini', max_depth=5, max_features='auto',\n",
       "                       max_leaf_nodes=None, max_samples=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=1000,\n",
       "                       n_jobs=None, oob_score=False, random_state=3, verbose=0,\n",
       "                       warm_start=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random Forest Classifier -> estimator\n",
    "# Here, I try to use Random Forest Classifier\n",
    "rclf = RandomForestClassifier(n_estimators=1000, max_depth=5, random_state=3) #set my random random forest classifier params\n",
    "\n",
    "# train 30% of the training set\n",
    "rclf.fit(X_scaled_sm_train, y_sm_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use grid search to select the best parameters\n",
    "grid_search_params ={\n",
    "    'criterion': ['gini'],\n",
    "    'max_depth': [None,1,5,10],\n",
    "    'max_features': ['auto', 3, 7 ],\n",
    "    'n_estimators':[100,200, 500, 1000],\n",
    "    'random_state':[3]\n",
    "}\n",
    "\n",
    "rclf_grid_search = GridSearchCV(RandomForestClassifier(), grid_search_params, n_jobs = -1, verbose = 1, cv = 3, scoring='recall')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 48 candidates, totalling 144 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  26 tasks      | elapsed:   36.0s\n",
      "[Parallel(n_jobs=-1)]: Done 144 out of 144 | elapsed:  2.6min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score=nan,\n",
       "             estimator=RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
       "                                              class_weight=None,\n",
       "                                              criterion='gini', max_depth=None,\n",
       "                                              max_features='auto',\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              max_samples=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              n_estimators=100, n_jobs=None,\n",
       "                                              oob_score=False,\n",
       "                                              random_state=None, verbose=0,\n",
       "                                              warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'criterion': ['gini'], 'max_depth': [None, 1, 5, 10],\n",
       "                         'max_features': ['auto', 3, 7],\n",
       "                         'n_estimators': [100, 200, 500, 1000],\n",
       "                         'random_state': [3]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='recall', verbose=1)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rclf_grid_search.fit(X_scaled_sm_train, y_sm_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'criterion': 'gini',\n",
       " 'max_depth': None,\n",
       " 'max_features': 'auto',\n",
       " 'n_estimators': 1000,\n",
       " 'random_state': 3}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rclf_grid_search.best_params_ # Check the best parameters from the grid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the best parameters that were obtained\n",
    "rclf_grid_search_best_params = rclf_grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#evaulate model performance\n",
    "#create performance evaluation function, with f1 score, precision, recall, and auc \n",
    "def eval_perf(model, X_test, y_test):\n",
    "    model_yhat = model.predict(X_test)\n",
    "    model_score = model.score(X_test, y_test)\n",
    "    model_f1 = f1_score(y_test, model_yhat, average = 'binary')\n",
    "    model_precision = precision_score(y_test, model_yhat, average = 'binary')\n",
    "    model_recall = recall_score(y_test, model_yhat, average = 'binary')\n",
    "    model_auc = roc_auc_score(y_test, model_yhat, average = 'macro')\n",
    "    \n",
    "    return (model_score, model_precision, model_recall, model_f1, model_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_rclf, precision_rclf, recall_rclf, f1_rclf, auc_rclf = eval_perf(rclf, X_scaled_test, y_test) #Evaluate performance here \n",
    "#This is performance evaluation for the normal random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for the normal/base Random Forest Model\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      " non_default       0.98      0.79      0.87      2075\n",
      "     default       0.22      0.76      0.35       164\n",
      "\n",
      "    accuracy                           0.79      2239\n",
      "   macro avg       0.60      0.78      0.61      2239\n",
      "weighted avg       0.92      0.79      0.84      2239\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Results for the normal/base Random Forest Model')\n",
    "random_forest_classifier_yhat = rclf.predict(X_scaled_test)\n",
    "print (classification_report(y_test, random_forest_classifier_yhat, labels = [0,1], target_names=['non_default','default']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predict_non_default</th>\n",
       "      <th>predict_default</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>actual_non_default</td>\n",
       "      <td>1640</td>\n",
       "      <td>435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>actual_default</td>\n",
       "      <td>39</td>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    predict_non_default  predict_default\n",
       "actual_non_default                 1640              435\n",
       "actual_default                       39              125"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the confusion matrix, and put it into a confusion_matrix\n",
    "#This is for the normal random forest\n",
    "confusion_matrix_base = pd.DataFrame(confusion_matrix(y_test, random_forest_classifier_yhat, labels = [0,1]),\n",
    "                      index = ['actual_non_default','actual_default'],\n",
    "                      columns = ['predict_non_default','predict_default'])\n",
    "confusion_matrix_base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for the Grid Search Random Forest Model\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      " non_default       0.95      0.95      0.95      2075\n",
      "     default       0.38      0.40      0.39       164\n",
      "\n",
      "    accuracy                           0.91      2239\n",
      "   macro avg       0.67      0.68      0.67      2239\n",
      "weighted avg       0.91      0.91      0.91      2239\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Evaluate performance of best params random forest\n",
    "score_rclf_gs_best, precision_rclf_gs_best, recall_rclf_gs_best, f1_rclf_gs_best, auc_rclf_gs_best = eval_perf(rclf_grid_search_best_params, X_scaled_test, y_test)\n",
    "\n",
    "print(\"Results for the Grid Search Random Forest Model\")\n",
    "RF_best_yhat = rclf_grid_search_best_params.predict(X_scaled_test)\n",
    "print (classification_report(y_test, RF_best_yhat, labels = [0,1], target_names=['non_default','default']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predict_non_default</th>\n",
       "      <th>predict_default</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>actual_non_default</td>\n",
       "      <td>1640</td>\n",
       "      <td>435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>actual_default</td>\n",
       "      <td>39</td>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    predict_non_default  predict_default\n",
       "actual_non_default                 1640              435\n",
       "actual_default                       39              125"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random Forest with Grid Search Best Params Confusion Matrix\n",
    "confusion_matrix_gs = pd.DataFrame(confusion_matrix(y_test, random_forest_classifier_yhat, labels = [0,1]),\n",
    "                      index = ['actual_non_default','actual_default'],\n",
    "                      columns = ['predict_non_default','predict_default'])\n",
    "confusion_matrix_gs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Basic Random Forest</th>\n",
       "      <th>Grid Search Random Forest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Accuracy:</td>\n",
       "      <td>0.78830</td>\n",
       "      <td>0.90889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Precision:</td>\n",
       "      <td>0.22321</td>\n",
       "      <td>0.38372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Recall:</td>\n",
       "      <td>0.76220</td>\n",
       "      <td>0.40244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>F1 Score:</td>\n",
       "      <td>0.34530</td>\n",
       "      <td>0.39286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>AUC Score:</td>\n",
       "      <td>0.77628</td>\n",
       "      <td>0.67568</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Basic Random Forest  Grid Search Random Forest\n",
       "Accuracy:                0.78830                    0.90889\n",
       "Precision:               0.22321                    0.38372\n",
       "Recall:                  0.76220                    0.40244\n",
       "F1 Score:                0.34530                    0.39286\n",
       "AUC Score:               0.77628                    0.67568"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compare_models_df = pd.DataFrame({'Basic Random Forest':[score_rclf, precision_rclf, recall_rclf, f1_rclf, auc_rclf],\\\n",
    "                                     'Grid Search Random Forest':[score_rclf_gs_best, precision_rclf_gs_best, recall_rclf_gs_best, f1_rclf_gs_best, auc_rclf_gs_best]},\\\n",
    "                                    index = ['Accuracy: ','Precision: ','Recall: ','F1 Score: ','AUC Score: '])\n",
    "compare_models_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The accuracy is higher for my Grid Search Random Forest Model\n",
    "- Precision is also higher for my Grid Search Random Forest Model\n",
    "- The Recall score is the ratio of correctly predicted positive observations to the all observations in the actual class. It is observed to get lower in my grid search random forest model.\n",
    "- The F1 score got higher for my Grid Search Random Forest model\n",
    "- AUC score is negatively impacted, as my Grid Search Random Forest model only managed to get 0.68, whilst the previous model obtained 0.78.\n",
    "\n",
    "Regarding which score to use for model evaluation, Recall score should be the top priority. This is because a high recall score would really help in selecting the actual amount of defaulters. False negatives are costly because we could give credit to people who cannot pay back.\n",
    "\n",
    "We also need to balance the costs between False Positives and False Negatives.\n",
    "For instance, let's say the Lender gave loans to someone who is predicted to be a non-defaulter. But over time, it turns out that they eventually defaulted on their loans (False Negative). This could prove costly as the Lender won't get returns on credit.\n",
    "But on the other hand, if the Lender predicted a 'non-defaulter or someone who could pay back their credit' as a defaulter, they would lose on potential profits (False Positive). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.17439203, 0.00753956, 0.07120882, 0.00770928, 0.02910391,\n",
       "       0.02961103, 0.10053447, 0.2419262 , 0.02736667, 0.01897217,\n",
       "       0.07742609, 0.01658144, 0.15824828, 0.03938003])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rclf.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('payment_ratio', 0.17439203278375562)\n",
      "('overlimit_percentage', 0.007539562228966823)\n",
      "('payment_ratio_6month', 0.07120881734479163)\n",
      "('years_since_card_issuing', 0.007709284389305301)\n",
      "('remaining_bill_per_number_of_cards', 0.02910390983575271)\n",
      "('remaining_bill_per_limit', 0.02961103458683387)\n",
      "('total_usage_per_limit', 0.10053447370498568)\n",
      "('total_3mo_usage_per_limit', 0.24192619914956312)\n",
      "('total_6mo_usage_per_limit', 0.02736667442860117)\n",
      "('utilization_6month', 0.01897216699057531)\n",
      "('outstanding', 0.07742609353463578)\n",
      "('credit_limit', 0.016581442932270193)\n",
      "('total_retail_usage', 0.15824827563225774)\n",
      "('delinquency_score', 0.039380032457704994)\n"
     ]
    }
   ],
   "source": [
    "#get my feature labels for printing\n",
    "feature_labels = df_model.columns\n",
    "# print feature importance on my random forest classifier.\n",
    "list_features = []\n",
    "for feature in zip(feature_labels, rclf.feature_importances_):\n",
    "    print(feature)\n",
    "    list_features.append(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_importance_df = pd.DataFrame({'importance': rclf.feature_importances_, 'features':df_model.columns})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>importance</th>\n",
       "      <th>features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.24193</td>\n",
       "      <td>total_3mo_usage_per_limit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.17439</td>\n",
       "      <td>payment_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.15825</td>\n",
       "      <td>total_retail_usage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.10053</td>\n",
       "      <td>total_usage_per_limit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.07743</td>\n",
       "      <td>outstanding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.07121</td>\n",
       "      <td>payment_ratio_6month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.03938</td>\n",
       "      <td>delinquency_score</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.02961</td>\n",
       "      <td>remaining_bill_per_limit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.02910</td>\n",
       "      <td>remaining_bill_per_number_of_cards</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.02737</td>\n",
       "      <td>total_6mo_usage_per_limit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.01897</td>\n",
       "      <td>utilization_6month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.01658</td>\n",
       "      <td>credit_limit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.00771</td>\n",
       "      <td>years_since_card_issuing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.00754</td>\n",
       "      <td>overlimit_percentage</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    importance                            features\n",
       "0      0.24193           total_3mo_usage_per_limit\n",
       "1      0.17439                       payment_ratio\n",
       "2      0.15825                  total_retail_usage\n",
       "3      0.10053               total_usage_per_limit\n",
       "4      0.07743                         outstanding\n",
       "5      0.07121                payment_ratio_6month\n",
       "6      0.03938                   delinquency_score\n",
       "7      0.02961            remaining_bill_per_limit\n",
       "8      0.02910  remaining_bill_per_number_of_cards\n",
       "9      0.02737           total_6mo_usage_per_limit\n",
       "10     0.01897                  utilization_6month\n",
       "11     0.01658                        credit_limit\n",
       "12     0.00771            years_since_card_issuing\n",
       "13     0.00754                overlimit_percentage"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_importance_df.sort_values('importance', ascending = False).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 3 most important features to classify someone as \"Defaulters\" or \"Non-Defaulters\" are:\n",
    "    1. Total usage per limit in the last 3 months\n",
    "    2. Payment ratio in the last month\n",
    "    3. Total Retail Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "- I created 2 Random Forest Classification Models made using base Random Forest (First model) and used Grid Search to tune params (Second Model):\n",
    "    - The First model has better Recall and AUC score\n",
    "    - The Second Model has better Accuracy, F1 score, and Precision\n",
    "- The first model obtained an accuracy of 78%, the second model got 91%\n",
    "- Both of these models could not beat the baseline accuracy I defined above, which is 92.65 %\n",
    "- Total usage (both retail usage and total usage per limit in 3months) and payment ratio are the most important features to predict Default/Non-Default\n",
    "- Select models based on recall score\n",
    "- Need to balance costs of false positives vs false negatives in the prediction model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}