notebooks/XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosted Trees with XGBoost\n",
    "\n",
    "Jay Urbain, PhD\n",
    "\n",
    "Credits:   \n",
    "https://xgboost.readthedocs.io/en/latest/  \n",
    "https://github.com/tqchen/xgboost/tree/master/demo/guide-python   \n",
    "https://jessesw.com/XG-Boost/    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[XGBoost](https://github.com/dmlc/xgboost) has become incredibly popular on [Kaggle](https://github.com/dmlc/xgboost/tree/master/demo#machine-learning-challenge-winning-solutions) for any problems dealing with structured (tabular) data. \n",
    "\n",
    "scikit-learn implementation of Gradient Boosted Trees: [sklearn's version](http://scikit-learn.org/stable/modules/ensemble.html#gradient-boosting).\n",
    "\n",
    "There are several technical issues why you may want to consider the xgboost implementation. The most significant is speed.\n",
    "\n",
    "Reference [this paper](http://www.jmlr.org/proceedings/papers/v42/chen14.pdf) on XGBoost's first major debut in the [Higgs-Boson Kaggle competition](https://www.kaggle.com/c/higgs-boson) that XGBoost has a nice speed advantage over sklearn. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Szilard Pafka has done an [excellent benchmark](https://github.com/szilard/benchm-ml) of a variety of machine learning libraries and he also found XGBoost to be much faster. \n",
    "\n",
    "What about XGBoost makes it faster? Gradient boosted trees, have to be built in series so that a step of gradient descent can be taken in order to minimize a loss function. Unlike Random Forests, you can't simply build the trees in parallel. XGBoost, however, builds the tree **itself** in a parallel fashion. More details about this are available in the XGBoost paper by Chen and Guestrin [here](https://arxiv.org/pdf/1603.02754v1.pdf). Essentially the information contained in each feature column can have statistics calculated on it in parallel.\n",
    "\n",
    "What you get in return for this speed is a much faster grid search for optimizing hyperparameters in model tuning. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting and Preprocessing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparative purposes, we will work with the [\"Adult Data Set\"](https://archive.ics.uci.edu/ml/datasets/Adult) used in our LogisticRegression lab.\n",
    "\n",
    "Our task is to predict whether a certain individual has an income of greater than 50,000 based on their demographic information. In the dataset description found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.names), we can see that the best model they came up with at the time had an accuracy of 85.95% (14.05% error on the test set). \n",
    "\n",
    "First, load in numpy and pandas, and download the data, which is already split into train/test sets. Skip the header row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load in the same train and test sets used by the original authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', header = None)\n",
    "test_set = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test',\n",
    "                      skiprows = 1, header = None) # Make sure to skip a row for the test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                  1       2           3   4                    5   \\\n",
       "0  39          State-gov   77516   Bachelors  13        Never-married   \n",
       "1  50   Self-emp-not-inc   83311   Bachelors  13   Married-civ-spouse   \n",
       "2  38            Private  215646     HS-grad   9             Divorced   \n",
       "3  53            Private  234721        11th   7   Married-civ-spouse   \n",
       "4  28            Private  338409   Bachelors  13   Married-civ-spouse   \n",
       "\n",
       "                   6               7       8        9     10  11  12  \\\n",
       "0        Adm-clerical   Not-in-family   White     Male  2174   0  40   \n",
       "1     Exec-managerial         Husband   White     Male     0   0  13   \n",
       "2   Handlers-cleaners   Not-in-family   White     Male     0   0  40   \n",
       "3   Handlers-cleaners         Husband   Black     Male     0   0  40   \n",
       "4      Prof-specialty            Wife   Black   Female     0   0  40   \n",
       "\n",
       "               13      14  \n",
       "0   United-States   <=50K  \n",
       "1   United-States   <=50K  \n",
       "2   United-States   <=50K  \n",
       "3   United-States   <=50K  \n",
       "4            Cuba   <=50K  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>Private</td>\n",
       "      <td>226802</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>89814</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>336951</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44</td>\n",
       "      <td>Private</td>\n",
       "      <td>160323</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18</td>\n",
       "      <td>?</td>\n",
       "      <td>103497</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>?</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0           1       2              3   4                    5   \\\n",
       "0  25     Private  226802           11th   7        Never-married   \n",
       "1  38     Private   89814        HS-grad   9   Married-civ-spouse   \n",
       "2  28   Local-gov  336951     Assoc-acdm  12   Married-civ-spouse   \n",
       "3  44     Private  160323   Some-college  10   Married-civ-spouse   \n",
       "4  18           ?  103497   Some-college  10        Never-married   \n",
       "\n",
       "                   6           7       8        9     10  11  12  \\\n",
       "0   Machine-op-inspct   Own-child   Black     Male     0   0  40   \n",
       "1     Farming-fishing     Husband   White     Male     0   0  50   \n",
       "2     Protective-serv     Husband   White     Male     0   0  40   \n",
       "3   Machine-op-inspct     Husband   Black     Male  7688   0  40   \n",
       "4                   ?   Own-child   White   Female     0   0  30   \n",
       "\n",
       "               13       14  \n",
       "0   United-States   <=50K.  \n",
       "1   United-States   <=50K.  \n",
       "2   United-States    >50K.  \n",
       "3   United-States    >50K.  \n",
       "4   United-States   <=50K.  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few potential issues:\n",
    "\n",
    "- We don't have a column header for our data.\n",
    "- There seem to be some unknown values in the fifth row of the test set (the question marks). \n",
    "- The target values have periods at the end in the test set but do not in the training set (<=50K. vs. <=50K)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the accompanying dataset description, we can see the column names. Add the column names to the train and test first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_labels = ['age', 'workclass', 'fnlwgt', 'education', 'education_num', 'marital_status', 'occupation', \n",
    "              'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',\n",
    "             'wage_class']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply names to both dataframes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set.columns = col_labels\n",
    "test_set.columns = col_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's check to see if pandas has identified any of these missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Data columns (total 15 columns):\n",
      "age               32561 non-null int64\n",
      "workclass         32561 non-null object\n",
      "fnlwgt            32561 non-null int64\n",
      "education         32561 non-null object\n",
      "education_num     32561 non-null int64\n",
      "marital_status    32561 non-null object\n",
      "occupation        32561 non-null object\n",
      "relationship      32561 non-null object\n",
      "race              32561 non-null object\n",
      "sex               32561 non-null object\n",
      "capital_gain      32561 non-null int64\n",
      "capital_loss      32561 non-null int64\n",
      "hours_per_week    32561 non-null int64\n",
      "native_country    32561 non-null object\n",
      "wage_class        32561 non-null object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 3.7+ MB\n"
     ]
    }
   ],
   "source": [
    "train_set.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 16281 entries, 0 to 16280\n",
      "Data columns (total 15 columns):\n",
      "age               16281 non-null int64\n",
      "workclass         16281 non-null object\n",
      "fnlwgt            16281 non-null int64\n",
      "education         16281 non-null object\n",
      "education_num     16281 non-null int64\n",
      "marital_status    16281 non-null object\n",
      "occupation        16281 non-null object\n",
      "relationship      16281 non-null object\n",
      "race              16281 non-null object\n",
      "sex               16281 non-null object\n",
      "capital_gain      16281 non-null int64\n",
      "capital_loss      16281 non-null int64\n",
      "hours_per_week    16281 non-null int64\n",
      "native_country    16281 non-null object\n",
      "wage_class        16281 non-null object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 1.9+ MB\n"
     ]
    }
   ],
   "source": [
    "test_set.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the accompanying notes, the original datasets had 32561 in train and 16281 with test. However, unknowns are included and have been labeled with a question mark (?). The test results documented were done after removing all unknowns. Therefore, to see if we can beat their best results, we need to remove the same unknown rows. \n",
    "\n",
    "If we do so, we should have 30162 in train and 15060 in test. Let's see if we can remove all of these unknown rows. \n",
    "\n",
    "It turns out the question mark was actually entered with a space first. Let's do a simple test to see what would happen if we dropped all rows that contain an unknown marked with a ' ?'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(30162, 15)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.replace(' ?', np.nan).dropna().shape "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15060, 15)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set.replace(' ?', np.nan).dropna().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These must be our missing rows since the numbers add up now if we drop them. Let's apply this change to our test and training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_nomissing = train_set.replace(' ?', np.nan).dropna()\n",
    "test_nomissing = test_set.replace(' ?', np.nan).dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have taken care of the missing value problem, we still have an issue with the target income thresholds being encoded slightly differently in test vs. train. \n",
    "\n",
    "We need these to match up appropriately, so we are going to need to fix either the test or training set to make them match up. \n",
    "\n",
    "Replace all of the '<=50K.' with '<=50K' and the same for '>50K.' with '>50K', we are just dropping the periods. \n",
    "\n",
    "We can use the replace method from pandas to fix this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_nomissing['wage_class'] = test_nomissing.wage_class.replace({' <=50K.': ' <=50K', ' >50K.':' >50K'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the unique values from each set, we can see if they now match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' <=50K', ' >50K'], dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_nomissing.wage_class.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' <=50K', ' >50K'], dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_nomissing.wage_class.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to make sure everything encoded as a string is turned into a variable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Applying  Encoding to Categoricals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is some debate as to whether you are technically required to use one-hot encoding for categorical features (since there isn't supposed to be any inherent ordering). [This discussion](https://github.com/szilard/benchm-ml/issues/1) on the subject interesting. \n",
    "\n",
    "Essentially, one-hot encoding isn't necessary for tree-based algorithms, but you may run into a problem using this on linear models.  \n",
    "\n",
    "First, combine them together into a single dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_set = pd.concat([train_nomissing, test_nomissing], axis = 0) # Stacks them vertically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 45222 entries, 0 to 16280\n",
      "Data columns (total 15 columns):\n",
      "age               45222 non-null int64\n",
      "workclass         45222 non-null object\n",
      "fnlwgt            45222 non-null int64\n",
      "education         45222 non-null object\n",
      "education_num     45222 non-null int64\n",
      "marital_status    45222 non-null object\n",
      "occupation        45222 non-null object\n",
      "relationship      45222 non-null object\n",
      "race              45222 non-null object\n",
      "sex               45222 non-null object\n",
      "capital_gain      45222 non-null int64\n",
      "capital_loss      45222 non-null int64\n",
      "hours_per_week    45222 non-null int64\n",
      "native_country    45222 non-null object\n",
      "wage_class        45222 non-null object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 5.5+ MB\n"
     ]
    }
   ],
   "source": [
    "combined_set.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the feature is not already numerical, we need to encode it as one. We can use pandas Categorical codes for this task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "for feature in combined_set.columns: # Loop through all columns in the dataframe\n",
    "    if combined_set[feature].dtype == 'object': # Only apply for columns with categorical strings\n",
    "        combined_set[feature] = pd.Categorical(combined_set[feature]).codes # Replace strings with an integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 45222 entries, 0 to 16280\n",
      "Data columns (total 15 columns):\n",
      "age               45222 non-null int64\n",
      "workclass         45222 non-null int8\n",
      "fnlwgt            45222 non-null int64\n",
      "education         45222 non-null int8\n",
      "education_num     45222 non-null int64\n",
      "marital_status    45222 non-null int8\n",
      "occupation        45222 non-null int8\n",
      "relationship      45222 non-null int8\n",
      "race              45222 non-null int8\n",
      "sex               45222 non-null int8\n",
      "capital_gain      45222 non-null int64\n",
      "capital_loss      45222 non-null int64\n",
      "hours_per_week    45222 non-null int64\n",
      "native_country    45222 non-null int8\n",
      "wage_class        45222 non-null int8\n",
      "dtypes: int64(6), int8(9)\n",
      "memory usage: 2.8 MB\n"
     ]
    }
   ],
   "source": [
    "combined_set.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all of our features encoded, we need to split these back into their original train/test sizes. Since they haven't been shuffled we just need to retrieve the same indices as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_train = combined_set[:train_nomissing.shape[0]] # Up to the last initial training set row\n",
    "final_test = combined_set[train_nomissing.shape[0]:]  # Past the last initial training set row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initial Model Setup and Grid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the target value inside the train and test frames prior to feeding into XGBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = final_train.pop('wage_class')\n",
    "y_test = final_test.pop('wage_class')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the libraries we will need to do grid search for XGBoost. We can use sklearn's API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To review the parameters that can be tuned in XGBoost, have a look at the [parameter documentation](https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst). \n",
    "\n",
    "A useful example is available [here](https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_objective.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_params = {'max_depth': [3,5,7], 'min_child_weight': [1,3,5]}\n",
    "ind_params = {'learning_rate': 0.1, 'n_estimators': 1000, 'seed':0, 'subsample': 0.8, 'colsample_bytree': 0.8, \n",
    "             'objective': 'binary:logistic'}\n",
    "optimized_GBM = GridSearchCV(xgb.XGBClassifier(**ind_params), \n",
    "                             cv_params, \n",
    "                             scoring = 'accuracy', cv = 5, n_jobs = -1,\n",
    "                             return_train_score=True) \n",
    "# Optimize for accuracy since that is the metric used in the Adult Data Set notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run our grid search with 5-fold cross-validation and see which parameters perform the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=1, missing=None, n_estimators=1000,\n",
       "       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=0, silent=True,\n",
       "       subsample=0.8),\n",
       "       fit_params=None, iid='warn', n_jobs=-1,\n",
       "       param_grid={'max_depth': [3, 5, 7], 'min_child_weight': [1, 3, 5]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimized_GBM.fit(final_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check our grid scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([38.05417356, 37.84070706, 36.05836725, 49.30986862, 48.47288327,\n",
       "        47.06869841, 63.41675344, 58.68249044, 47.37163076]),\n",
       " 'std_fit_time': array([0.22499025, 0.04537635, 1.41610977, 0.08346701, 0.43032541,\n",
       "        0.22646438, 0.15052846, 2.39786558, 5.08487728]),\n",
       " 'mean_score_time': array([0.33268046, 0.31883764, 0.30720892, 0.65682526, 0.55268512,\n",
       "        0.5184598 , 1.16563668, 0.67396722, 0.46826453]),\n",
       " 'std_score_time': array([0.01196734, 0.00549443, 0.02788719, 0.021732  , 0.04582518,\n",
       "        0.00964167, 0.01760205, 0.0733309 , 0.05362652]),\n",
       " 'param_max_depth': masked_array(data=[3, 3, 3, 5, 5, 5, 7, 7, 7],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_min_child_weight': masked_array(data=[1, 3, 5, 1, 3, 5, 1, 3, 5],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'max_depth': 3, 'min_child_weight': 1},\n",
       "  {'max_depth': 3, 'min_child_weight': 3},\n",
       "  {'max_depth': 3, 'min_child_weight': 5},\n",
       "  {'max_depth': 5, 'min_child_weight': 1},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 5},\n",
       "  {'max_depth': 7, 'min_child_weight': 1},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 5}],\n",
       " 'split0_test_score': array([0.86258909, 0.86490966, 0.86258909, 0.86060003, 0.85877673,\n",
       "        0.86159456, 0.85380408, 0.85678767, 0.85695342]),\n",
       " 'split1_test_score': array([0.86391513, 0.86341787, 0.86457815, 0.85827946, 0.85745069,\n",
       "        0.85811371, 0.8496602 , 0.85098624, 0.851152  ]),\n",
       " 'split2_test_score': array([0.87037958, 0.86839052, 0.87037958, 0.86408089, 0.86192607,\n",
       "        0.86258909, 0.85894248, 0.8562904 , 0.85678767]),\n",
       " 'split3_test_score': array([0.87267905, 0.87317639, 0.87367374, 0.86571618, 0.86521883,\n",
       "        0.86571618, 0.85692971, 0.86223475, 0.86024536]),\n",
       " 'split4_test_score': array([0.86668877, 0.86867849, 0.86784944, 0.86254353, 0.86254353,\n",
       "        0.86337258, 0.85674018, 0.85690599, 0.85591113]),\n",
       " 'mean_test_score': array([0.86725018, 0.86771434, 0.86781381, 0.86224388, 0.86118295,\n",
       "        0.86227704, 0.85521517, 0.85664081, 0.8562098 ]),\n",
       " 'std_test_score': array([0.00380608, 0.00339256, 0.00396718, 0.00260543, 0.00277171,\n",
       "        0.00248697, 0.00322524, 0.0035632 , 0.00292728]),\n",
       " 'rank_test_score': array([3, 2, 1, 5, 6, 4, 9, 7, 8], dtype=int32),\n",
       " 'split0_train_score': array([0.89543703, 0.89295039, 0.89191429, 0.940528  , 0.92871648,\n",
       "        0.92113225, 0.98532886, 0.9680053 , 0.95594513]),\n",
       " 'split1_train_score': array([0.89510547, 0.89365494, 0.89141697, 0.9425173 , 0.93215633,\n",
       "        0.92511086, 0.98624062, 0.97103071, 0.95810021]),\n",
       " 'split2_train_score': array([0.89618302, 0.89266028, 0.89091964, 0.94102532, 0.93070579,\n",
       "        0.92237556, 0.98400265, 0.9666791 , 0.95470181]),\n",
       " 'split3_train_score': array([0.89602155, 0.89303771, 0.89021964, 0.9413593 , 0.9294654 ,\n",
       "        0.92171571, 0.98537091, 0.96920845, 0.95524244]),\n",
       " 'split4_train_score': array([0.89552857, 0.89362231, 0.89130165, 0.9417347 , 0.9310845 ,\n",
       "        0.92180183, 0.98445982, 0.9702043 , 0.95686047]),\n",
       " 'mean_train_score': array([0.89565513, 0.89318513, 0.89115444, 0.94143292, 0.9304257 ,\n",
       "        0.92242724, 0.98508057, 0.96902557, 0.95617001]),\n",
       " 'std_train_score': array([0.00039462, 0.00039093, 0.00056514, 0.00067192, 0.00121253,\n",
       "        0.00139849, 0.0007796 , 0.00154857, 0.00120573])}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimized_GBM.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first hyperparameter combination performed best and we've already exceeded our target of 85.95% accuracy in our cross-validation. \n",
    "\n",
    "We can try optimizing some other hyperparameters now to see if we can beat a mean of 86.78% accuracy. This time, we will play around with subsampling along with lowering the learning rate to see if that helps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_params = {'learning_rate': [0.1, 0.01], 'subsample': [0.7,0.8,0.9]}\n",
    "ind_params = {'n_estimators': 1000, 'seed':0, 'colsample_bytree': 0.8, \n",
    "             'objective': 'binary:logistic', 'max_depth': 3, 'min_child_weight': 1}\n",
    "\n",
    "\n",
    "optimized_GBM = GridSearchCV(xgb.XGBClassifier(**ind_params), \n",
    "                            cv_params, \n",
    "                             scoring = 'accuracy', cv = 5, n_jobs = -1)\n",
    "optimized_GBM.fit(final_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, check the grid scores. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_GBM.grid_scores_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No real improvement. We may be able to optimize a little further by utilizing XGBoost's built-in cv which allows early stopping to prevent overfitting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Early stopping CV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the CV testing performed earlier, we want to utilize the following parameters:\n",
    "\n",
    "- Learning_rate (eta) = 0.1\n",
    "- Subsample, colsample_bytree = 0.8\n",
    "- Max_depth = 3\n",
    "- Min_child_weight = 1\n",
    "\n",
    "There are a few other parameters we could tune in theory to squeeze out a little more performance, but this is a good enough starting point. \n",
    "\n",
    "To increase the performance of XGBoost's speed through many iterations of the training set, we can create a DMatrix. The DMatrix sorts the data to optimize for XGBoost when it builds trees, making the algorithm more efficient. To create a DMatrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgdmat = xgb.DMatrix(final_train, y_train) # Create our DMatrix to make XGBoost more efficient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's specify our parameters (with slightly different syntax in some places for the XGBoost native API) and set our stopping criteria. For now, let's be aggressive with the stopping and say we don't want the accuracy to improve for at least 100 new trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "our_params = {'eta': 0.1, 'seed':0, 'subsample': 0.8, 'colsample_bytree': 0.8, \n",
    "             'objective': 'binary:logistic', 'max_depth':3, 'min_child_weight':1} \n",
    "# Grid Search CV optimized settings\n",
    "\n",
    "cv_xgb = xgb.cv(params = our_params, dtrain = xgdmat, num_boost_round = 3000, nfold = 5,\n",
    "                metrics = ['error'], # Make sure you enter metrics inside a list or you may encounter issues!\n",
    "                early_stopping_rounds = 100) # Look for early stopping that minimizes error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at our CV results to see how accurate we were with these settings. The output is automatically saved into a pandas dataframe for us. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_xgb.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our CV test error at this number of iterations is 12.897%,  or 87.103% accuracy. \n",
    "\n",
    "Now that we have our best settings, let's create this as an XGBoost object model that we can reference later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "our_params = {'eta': 0.1, 'seed':0, 'subsample': 0.8, 'colsample_bytree': 0.8, \n",
    "             'objective': 'binary:logistic', 'max_depth':3, 'min_child_weight':1} \n",
    "\n",
    "final_gb = xgb.train(our_params, xgdmat, num_boost_round = 432)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that it trains very quickly. With our XG model object, we can then plot our feature importances using a built-in method. This is similar to the feature importances found in sklearn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set(font_scale = 1.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.plot_importance(final_gb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will tell us which features were most important in the series of trees. The 'fnlwgt' feature seems to have the most importance. Filing capital gains was also important, which makes sense given that only those with greater incomes have the ability to invest. Race and sex were not as important. This may be because we are just predicting a specific threshold (below/above 50000 a year) instead of a precise income. \n",
    "\n",
    "If the built-in feature importance method isn't what you wanted, you can make your own chart of feature importances using the get_fscore method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importances = final_gb.get_fscore()\n",
    "importances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given this dictionary, we can now make a plot of it ourselves. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_frame = pd.DataFrame({'Importance': list(importances.values()), 'Feature': list(importances.keys())})\n",
    "importance_frame.sort_values(by = 'Importance', inplace = True)\n",
    "importance_frame.plot(kind = 'barh', x = 'Feature', figsize = (8,8), color = 'orange')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an understanding of the feature importances, we can better understand what is driving the splits most for the trees and where we may be able to make some improvements in feature engineering if possible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analyzing Performance on Test Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has now been tuned using cross-validation grid search through the sklearn API and early stopping through the built-in XGBoost API. Now, we can see how it finally performs on the test set. Does it match our CV performance? First, create another DMatrix (this time for the test data). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "testdmat = xgb.DMatrix(final_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use sklearn's accuracy metric to see how well we did on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "y_pred = final_gb.predict(testdmat) # Predict using our testdmat\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the predict function for XGBoost outputs probabilities by default and not actual class labels. To calculate accuracy we need to convert these to a 0/1 label. We will set 0.5 probability as our threshold. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred[y_pred > 0.5] = 1\n",
    "y_pred[y_pred <= 0.5] = 0\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can calculate our accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(y_pred, y_test), 1-accuracy_score(y_pred, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our final accuracy is 86.94%, or a 13.05% error rate. We beat our goal by a one percentage point! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Further improvement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Make sure to read the [documentation](https://xgboost.readthedocs.org/en/latest/) thoroughly if you want to utilize some of its more advanced features\n",
    "- This is a rapidly evolving library, so additional features may be introduced that affect its functionality\n",
    "- XGBoost is available in most of the other common languages for data science as well such as Julia and R if those are preferable\n",
    "- XGBoost can also now be utilized at scale, running on Spark and Flink with the recently released [XGBoost4J](http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html)\n",
    "\n",
    "Some other helpful information:\n",
    "\n",
    "- This [YouTube video](https://www.youtube.com/watch?v=ufHo8vbk6g4) given by Tong He on XGBoost in detail. This describes the R version, but the API is very similar and there is a lot of great information in here on how XGBoost actually works.\n",
    "- Here is his [slide deck](http://www.slideshare.net/ShangxuanZhang/xgboost) that covers the same material.\n",
    "- Another tutorial guide on hyperparameter tuning from Aarshay Jain [here](http://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}