Predicting_CKD_XGB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting the Hyper-local Prevalence of Chronic Kidney Disease\n",
    "\n",
    "Using Census and CDC data, we may be able to better identify neighborhoods where aggressive public health campaigns and healthcare initiatives can positively affect the early detection and treatment of CKD.\n",
    "\n",
    "## 1.0 Load Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import os\n",
    "import sys\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%load_ext watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define project root directory\n",
    "PROJ_ROOT = os.path.join(os.pardir)\n",
    "\n",
    "# define library directory\n",
    "sys.path.append(PROJ_ROOT)\n",
    "\n",
    "load_dotenv(find_dotenv(), verbose=True)\n",
    "\n",
    "# Census Bureau API key\n",
    "API_KEY = os.environ.get(\"CENSUS_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import project modules\n",
    "from predict_ckd.data import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "E. Chris Lynch 2019-03-19 23:58:44 \n",
      "\n",
      "CPython 3.6.6\n",
      "IPython 7.1.1\n",
      "\n",
      "numpy 1.15.4\n",
      "pandas 0.24.1\n",
      "matplotlib 3.0.3\n",
      "requests 2.20.0\n",
      "sklearn 0.20.0\n"
     ]
    }
   ],
   "source": [
    "%watermark -a \"E. Chris Lynch\" -d -t -v -p numpy,pandas,matplotlib,requests,sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Load Data\n",
    "This function will load 2016 data that was collected from the Census Bureau API and cleaned in order to train a predictive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load train and test data\n",
    "X_train, X_test, y_train, y_test = load_and_split_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Train the Model\n",
    "We will use the data to create a model using stochastic XGBoost. This model will be saved in order to predict CKD on 2017 and future data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2: 0.8577\n",
      "MSE: 0.1276\n",
      "Adjusted R2: 0.8526\n"
     ]
    }
   ],
   "source": [
    "# model data with XGBoost\n",
    "params = {'n_estimators':700, 'max_depth': 5, 'eta':0.1,\n",
    "          'colsample_bytree':0.1, 'random_state':22}\n",
    "\n",
    "xgb_model = xgb_model(params=params, features_train=X_train, \n",
    "                      labels_train=y_train, features_test=X_test, \n",
    "                      labels_test=y_test)\n",
    "xgb_model.save_model('data/xgb.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Make Predictions\n",
    "We will now make predictions using 2017 data for Contra Costa County in California. If you don't know the FIPS code for your county, you can use the `get_county_code` module. You must have a valid Census Bureau API key to run the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "192    013\n",
       "Name: county_code, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get code for example county\n",
    "code = get_county_code(state='CA', county='Contra Costa County')\n",
    "code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get tract-level Census data for given county\n",
    "contra_costa = get_county_data(state='CA', county='013', key=API_KEY, year='2017')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S0101_C02_002E</th>\n",
       "      <th>state</th>\n",
       "      <th>county</th>\n",
       "      <th>tract</th>\n",
       "      <th>S0101_C02_003E</th>\n",
       "      <th>S0101_C02_004E</th>\n",
       "      <th>S0101_C02_005E</th>\n",
       "      <th>S0101_C02_006E</th>\n",
       "      <th>S0101_C02_007E</th>\n",
       "      <th>S0101_C02_008E</th>\n",
       "      <th>...</th>\n",
       "      <th>S2701_C05_002E</th>\n",
       "      <th>S2701_C05_005E</th>\n",
       "      <th>S2701_C05_011E</th>\n",
       "      <th>S2703_C03_003E</th>\n",
       "      <th>S2703_C03_004E</th>\n",
       "      <th>S2703_C03_005E</th>\n",
       "      <th>S2704_C03_003E</th>\n",
       "      <th>S2704_C03_004E</th>\n",
       "      <th>S2704_C03_005E</th>\n",
       "      <th>ckd_prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>351102</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>1.85</td>\n",
       "      <td>66.65</td>\n",
       "      <td>34.5</td>\n",
       "      <td>42.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.9</td>\n",
       "      <td>99.1</td>\n",
       "      <td>5.480069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.1</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>339002</td>\n",
       "      <td>2.9</td>\n",
       "      <td>4.5</td>\n",
       "      <td>4.8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>15.9</td>\n",
       "      <td>7.7</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.8</td>\n",
       "      <td>4.60</td>\n",
       "      <td>62.70</td>\n",
       "      <td>70.5</td>\n",
       "      <td>31.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>92.6</td>\n",
       "      <td>3.700408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.6</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355115</td>\n",
       "      <td>11.1</td>\n",
       "      <td>9.5</td>\n",
       "      <td>6.1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.7</td>\n",
       "      <td>8.6</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1.40</td>\n",
       "      <td>75.70</td>\n",
       "      <td>79.3</td>\n",
       "      <td>11.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>92.4</td>\n",
       "      <td>2.949047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.5</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355116</td>\n",
       "      <td>12.4</td>\n",
       "      <td>13.8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>2.1</td>\n",
       "      <td>1.9</td>\n",
       "      <td>3.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.40</td>\n",
       "      <td>87.10</td>\n",
       "      <td>86.0</td>\n",
       "      <td>33.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>89.8</td>\n",
       "      <td>3.426589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.6</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355117</td>\n",
       "      <td>13.4</td>\n",
       "      <td>15.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>3.7</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>91.50</td>\n",
       "      <td>89.4</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>87.6</td>\n",
       "      <td>2.829141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 239 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   S0101_C02_002E  state  county   tract  S0101_C02_003E  S0101_C02_004E  \\\n",
       "0             0.0      6      13  351102             0.0             0.0   \n",
       "1             5.1      6      13  339002             2.9             4.5   \n",
       "2             8.6      6      13  355115            11.1             9.5   \n",
       "3             5.5      6      13  355116            12.4            13.8   \n",
       "4             5.6      6      13  355117            13.4            15.0   \n",
       "\n",
       "   S0101_C02_005E  S0101_C02_006E  S0101_C02_007E  S0101_C02_008E  ...  \\\n",
       "0             0.0             0.0             0.0             0.0  ...   \n",
       "1             4.8             8.1            15.9             7.7  ...   \n",
       "2             6.1             4.0             5.7             8.6  ...   \n",
       "3             8.3             2.1             1.9             3.5  ...   \n",
       "4             6.8             3.7             1.4             1.5  ...   \n",
       "\n",
       "   S2701_C05_002E  S2701_C05_005E  S2701_C05_011E  S2703_C03_003E  \\\n",
       "0             0.0            11.5            1.85           66.65   \n",
       "1             0.0             4.8            4.60           62.70   \n",
       "2             0.0             8.2            1.40           75.70   \n",
       "3             0.0             0.0            0.40           87.10   \n",
       "4             0.0             0.0            0.50           91.50   \n",
       "\n",
       "   S2703_C03_004E  S2703_C03_005E  S2704_C03_003E  S2704_C03_004E  \\\n",
       "0            34.5            42.5             0.0             3.9   \n",
       "1            70.5            31.1             0.0             0.6   \n",
       "2            79.3            11.1             0.0             0.6   \n",
       "3            86.0            33.3             0.0             0.8   \n",
       "4            89.4            22.5             0.0             1.1   \n",
       "\n",
       "   S2704_C03_005E  ckd_prediction  \n",
       "0            99.1        5.480069  \n",
       "1            92.6        3.700408  \n",
       "2            92.4        2.949047  \n",
       "3            89.8        3.426589  \n",
       "4            87.6        2.829141  \n",
       "\n",
       "[5 rows x 239 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict CKD prevalence\n",
    "cc_preds = predict_county_ckd(model='data/xgb.model', data=contra_costa)\n",
    "cc_preds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>S0101_C02_002E</th>\n",
       "      <th>state</th>\n",
       "      <th>county</th>\n",
       "      <th>tract</th>\n",
       "      <th>S0101_C02_003E</th>\n",
       "      <th>S0101_C02_004E</th>\n",
       "      <th>S0101_C02_005E</th>\n",
       "      <th>S0101_C02_006E</th>\n",
       "      <th>S0101_C02_007E</th>\n",
       "      <th>S0101_C02_008E</th>\n",
       "      <th>...</th>\n",
       "      <th>S2701_C05_002E</th>\n",
       "      <th>S2701_C05_005E</th>\n",
       "      <th>S2701_C05_011E</th>\n",
       "      <th>S2703_C03_003E</th>\n",
       "      <th>S2703_C03_004E</th>\n",
       "      <th>S2703_C03_005E</th>\n",
       "      <th>S2704_C03_003E</th>\n",
       "      <th>S2704_C03_004E</th>\n",
       "      <th>S2704_C03_005E</th>\n",
       "      <th>ckd_prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>351102</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>1.85</td>\n",
       "      <td>66.65</td>\n",
       "      <td>34.5</td>\n",
       "      <td>42.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.9</td>\n",
       "      <td>99.1</td>\n",
       "      <td>5.480069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.1</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>339002</td>\n",
       "      <td>2.9</td>\n",
       "      <td>4.5</td>\n",
       "      <td>4.8</td>\n",
       "      <td>8.1</td>\n",
       "      <td>15.9</td>\n",
       "      <td>7.7</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.8</td>\n",
       "      <td>4.60</td>\n",
       "      <td>62.70</td>\n",
       "      <td>70.5</td>\n",
       "      <td>31.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>92.6</td>\n",
       "      <td>3.700408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.6</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355115</td>\n",
       "      <td>11.1</td>\n",
       "      <td>9.5</td>\n",
       "      <td>6.1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.7</td>\n",
       "      <td>8.6</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.2</td>\n",
       "      <td>1.40</td>\n",
       "      <td>75.70</td>\n",
       "      <td>79.3</td>\n",
       "      <td>11.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>92.4</td>\n",
       "      <td>2.949047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.5</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355116</td>\n",
       "      <td>12.4</td>\n",
       "      <td>13.8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>2.1</td>\n",
       "      <td>1.9</td>\n",
       "      <td>3.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.40</td>\n",
       "      <td>87.10</td>\n",
       "      <td>86.0</td>\n",
       "      <td>33.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>89.8</td>\n",
       "      <td>3.426589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.6</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>355117</td>\n",
       "      <td>13.4</td>\n",
       "      <td>15.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>3.7</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>91.50</td>\n",
       "      <td>89.4</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>87.6</td>\n",
       "      <td>2.829141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 239 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   S0101_C02_002E  state  county   tract  S0101_C02_003E  S0101_C02_004E  \\\n",
       "0             0.0      6      13  351102             0.0             0.0   \n",
       "1             5.1      6      13  339002             2.9             4.5   \n",
       "2             8.6      6      13  355115            11.1             9.5   \n",
       "3             5.5      6      13  355116            12.4            13.8   \n",
       "4             5.6      6      13  355117            13.4            15.0   \n",
       "\n",
       "   S0101_C02_005E  S0101_C02_006E  S0101_C02_007E  S0101_C02_008E  ...  \\\n",
       "0             0.0             0.0             0.0             0.0  ...   \n",
       "1             4.8             8.1            15.9             7.7  ...   \n",
       "2             6.1             4.0             5.7             8.6  ...   \n",
       "3             8.3             2.1             1.9             3.5  ...   \n",
       "4             6.8             3.7             1.4             1.5  ...   \n",
       "\n",
       "   S2701_C05_002E  S2701_C05_005E  S2701_C05_011E  S2703_C03_003E  \\\n",
       "0             0.0            11.5            1.85           66.65   \n",
       "1             0.0             4.8            4.60           62.70   \n",
       "2             0.0             8.2            1.40           75.70   \n",
       "3             0.0             0.0            0.40           87.10   \n",
       "4             0.0             0.0            0.50           91.50   \n",
       "\n",
       "   S2703_C03_004E  S2703_C03_005E  S2704_C03_003E  S2704_C03_004E  \\\n",
       "0            34.5            42.5             0.0             3.9   \n",
       "1            70.5            31.1             0.0             0.6   \n",
       "2            79.3            11.1             0.0             0.6   \n",
       "3            86.0            33.3             0.0             0.8   \n",
       "4            89.4            22.5             0.0             1.1   \n",
       "\n",
       "   S2704_C03_005E  ckd_prediction  \n",
       "0            99.1        5.480069  \n",
       "1            92.6        3.700408  \n",
       "2            92.4        2.949047  \n",
       "3            89.8        3.426589  \n",
       "4            87.6        2.829141  \n",
       "\n",
       "[5 rows x 239 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "contra_costa_full = contra_costa.merge(pd.Series(ckd_preds).rename('ckd_prediction'), left_index=True, right_index=True)\n",
    "contra_costa_full.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "contra_costa_full.to_csv('data/contra_costa.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}