notebooks/5 - Imputing Data.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Agenda\n",
      "\n",
      "- Define the problem and the approach\n",
      "- Data basics: loading data, looking at your data, basic commands\n",
      "- <p style=\"color: red\">Handling missing values</p>\n",
      "- Intro to scikit-learn\n",
      "- Grouping and aggregating data\n",
      "- Feature selection\n",
      "- Fitting and evaluating a model\n",
      "- Deploying your work"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#In this workbook you will\n",
      "- Learn about imputation and what it's used for\n",
      "- Use the KNearestNeighbors algorithm to impute data\n",
      "- Come up with your own imputation strategy"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Imputation\n",
      "Imputation is fairly intuitive. For the missing data in our dataset, we're going to replace it with values that come from similar records in our dataset that aren't null.\n",
      "To do this we're going to use the NearestNeighbors algorithm."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "import numpy as np\n",
      "import pylab as pl"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.read_csv(\"./data/credit-data-post-import.csv\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 33
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Cross Validation\n",
      "We're going to use the simplest type of cross validation. we'll simply split our data into 2 groups: training and test. we'll use the training set to calibrate our model and then use the test set to  evaluate how effective it is."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "is_test = np.random.uniform(0, 1, len(df)) > 0.75\n",
      "train = df[is_test==False]\n",
      "test = df[is_test==True]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(train), len(test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "(112415, 37585)"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Be sure to calibrate the imputation with the training set"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.neighbors import KNeighborsRegressor\n",
      "\n",
      "income_imputer = KNeighborsRegressor(n_neighbors=1)\n",
      "\n",
      "#split our data into 2 groups; data containing nulls and data \n",
      "# not containing nulls we'll train on the latter and make\n",
      "# 'predictions' on the null data to impute monthly_income\n",
      "train_w_monthly_income = train[train.monthly_income.isnull()==False]\n",
      "train_w_null_monthly_income = train[train.monthly_income.isnull()==True]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train_w_monthly_income.corr()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <th>age</th>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <th>debt_ratio</th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <th>number_of_dependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.001968</td>\n",
        "      <td>-0.102272</td>\n",
        "      <td> 0.120134</td>\n",
        "      <td>-0.002382</td>\n",
        "      <td>-0.019663</td>\n",
        "      <td>-0.026876</td>\n",
        "      <td> 0.108921</td>\n",
        "      <td>-0.005976</td>\n",
        "      <td> 0.091995</td>\n",
        "      <td> 0.047273</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <td>-0.001968</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.003601</td>\n",
        "      <td>-0.001269</td>\n",
        "      <td> 0.000075</td>\n",
        "      <td> 0.007388</td>\n",
        "      <td>-0.010750</td>\n",
        "      <td>-0.001091</td>\n",
        "      <td> 0.007419</td>\n",
        "      <td>-0.000975</td>\n",
        "      <td> 0.002690</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>age</th>\n",
        "      <td>-0.102272</td>\n",
        "      <td>-0.003601</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.052251</td>\n",
        "      <td> 0.000083</td>\n",
        "      <td> 0.037199</td>\n",
        "      <td> 0.186420</td>\n",
        "      <td>-0.051162</td>\n",
        "      <td> 0.066435</td>\n",
        "      <td>-0.046745</td>\n",
        "      <td>-0.206503</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <td> 0.120134</td>\n",
        "      <td>-0.001269</td>\n",
        "      <td>-0.052251</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.001488</td>\n",
        "      <td>-0.010458</td>\n",
        "      <td>-0.047053</td>\n",
        "      <td> 0.977052</td>\n",
        "      <td>-0.026220</td>\n",
        "      <td> 0.981492</td>\n",
        "      <td> 0.007043</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>debt_ratio</th>\n",
        "      <td>-0.002382</td>\n",
        "      <td> 0.000075</td>\n",
        "      <td> 0.000083</td>\n",
        "      <td>-0.001488</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.026406</td>\n",
        "      <td> 0.008032</td>\n",
        "      <td>-0.002738</td>\n",
        "      <td> 0.015818</td>\n",
        "      <td>-0.001608</td>\n",
        "      <td> 0.005896</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>monthly_income</th>\n",
        "      <td>-0.019663</td>\n",
        "      <td> 0.007388</td>\n",
        "      <td> 0.037199</td>\n",
        "      <td>-0.010458</td>\n",
        "      <td>-0.026406</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.089260</td>\n",
        "      <td>-0.012778</td>\n",
        "      <td> 0.123512</td>\n",
        "      <td>-0.011264</td>\n",
        "      <td> 0.064060</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <td>-0.026876</td>\n",
        "      <td>-0.010750</td>\n",
        "      <td> 0.186420</td>\n",
        "      <td>-0.047053</td>\n",
        "      <td> 0.008032</td>\n",
        "      <td> 0.089260</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.074116</td>\n",
        "      <td> 0.426890</td>\n",
        "      <td>-0.063968</td>\n",
        "      <td> 0.041384</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <td> 0.108921</td>\n",
        "      <td>-0.001091</td>\n",
        "      <td>-0.051162</td>\n",
        "      <td> 0.977052</td>\n",
        "      <td>-0.002738</td>\n",
        "      <td>-0.012778</td>\n",
        "      <td>-0.074116</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.041557</td>\n",
        "      <td> 0.990128</td>\n",
        "      <td>-0.000703</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <td>-0.005976</td>\n",
        "      <td> 0.007419</td>\n",
        "      <td> 0.066435</td>\n",
        "      <td>-0.026220</td>\n",
        "      <td> 0.015818</td>\n",
        "      <td> 0.123512</td>\n",
        "      <td> 0.426890</td>\n",
        "      <td>-0.041557</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.035835</td>\n",
        "      <td> 0.119559</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <td> 0.091995</td>\n",
        "      <td>-0.000975</td>\n",
        "      <td>-0.046745</td>\n",
        "      <td> 0.981492</td>\n",
        "      <td>-0.001608</td>\n",
        "      <td>-0.011264</td>\n",
        "      <td>-0.063968</td>\n",
        "      <td> 0.990128</td>\n",
        "      <td>-0.035835</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td>-0.002197</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_dependents</th>\n",
        "      <td> 0.047273</td>\n",
        "      <td> 0.002690</td>\n",
        "      <td>-0.206503</td>\n",
        "      <td> 0.007043</td>\n",
        "      <td> 0.005896</td>\n",
        "      <td> 0.064060</td>\n",
        "      <td> 0.041384</td>\n",
        "      <td>-0.000703</td>\n",
        "      <td> 0.119559</td>\n",
        "      <td>-0.002197</td>\n",
        "      <td> 1.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 37,
       "text": [
        "                                             serious_dlqin2yrs  \\\n",
        "serious_dlqin2yrs                                     1.000000   \n",
        "revolving_utilization_of_unsecured_lines             -0.001968   \n",
        "age                                                  -0.102272   \n",
        "number_of_time30-59_days_past_due_not_worse           0.120134   \n",
        "debt_ratio                                           -0.002382   \n",
        "monthly_income                                       -0.019663   \n",
        "number_of_open_credit_lines_and_loans                -0.026876   \n",
        "number_of_times90_days_late                           0.108921   \n",
        "number_real_estate_loans_or_lines                    -0.005976   \n",
        "number_of_time60-89_days_past_due_not_worse           0.091995   \n",
        "number_of_dependents                                  0.047273   \n",
        "\n",
        "                                             revolving_utilization_of_unsecured_lines  \\\n",
        "serious_dlqin2yrs                                                           -0.001968   \n",
        "revolving_utilization_of_unsecured_lines                                     1.000000   \n",
        "age                                                                         -0.003601   \n",
        "number_of_time30-59_days_past_due_not_worse                                 -0.001269   \n",
        "debt_ratio                                                                   0.000075   \n",
        "monthly_income                                                               0.007388   \n",
        "number_of_open_credit_lines_and_loans                                       -0.010750   \n",
        "number_of_times90_days_late                                                 -0.001091   \n",
        "number_real_estate_loans_or_lines                                            0.007419   \n",
        "number_of_time60-89_days_past_due_not_worse                                 -0.000975   \n",
        "number_of_dependents                                                         0.002690   \n",
        "\n",
        "                                                  age  \\\n",
        "serious_dlqin2yrs                           -0.102272   \n",
        "revolving_utilization_of_unsecured_lines    -0.003601   \n",
        "age                                          1.000000   \n",
        "number_of_time30-59_days_past_due_not_worse -0.052251   \n",
        "debt_ratio                                   0.000083   \n",
        "monthly_income                               0.037199   \n",
        "number_of_open_credit_lines_and_loans        0.186420   \n",
        "number_of_times90_days_late                 -0.051162   \n",
        "number_real_estate_loans_or_lines            0.066435   \n",
        "number_of_time60-89_days_past_due_not_worse -0.046745   \n",
        "number_of_dependents                        -0.206503   \n",
        "\n",
        "                                             number_of_time30-59_days_past_due_not_worse  \\\n",
        "serious_dlqin2yrs                                                               0.120134   \n",
        "revolving_utilization_of_unsecured_lines                                       -0.001269   \n",
        "age                                                                            -0.052251   \n",
        "number_of_time30-59_days_past_due_not_worse                                     1.000000   \n",
        "debt_ratio                                                                     -0.001488   \n",
        "monthly_income                                                                 -0.010458   \n",
        "number_of_open_credit_lines_and_loans                                          -0.047053   \n",
        "number_of_times90_days_late                                                     0.977052   \n",
        "number_real_estate_loans_or_lines                                              -0.026220   \n",
        "number_of_time60-89_days_past_due_not_worse                                     0.981492   \n",
        "number_of_dependents                                                            0.007043   \n",
        "\n",
        "                                             debt_ratio  monthly_income  \\\n",
        "serious_dlqin2yrs                             -0.002382       -0.019663   \n",
        "revolving_utilization_of_unsecured_lines       0.000075        0.007388   \n",
        "age                                            0.000083        0.037199   \n",
        "number_of_time30-59_days_past_due_not_worse   -0.001488       -0.010458   \n",
        "debt_ratio                                     1.000000       -0.026406   \n",
        "monthly_income                                -0.026406        1.000000   \n",
        "number_of_open_credit_lines_and_loans          0.008032        0.089260   \n",
        "number_of_times90_days_late                   -0.002738       -0.012778   \n",
        "number_real_estate_loans_or_lines              0.015818        0.123512   \n",
        "number_of_time60-89_days_past_due_not_worse   -0.001608       -0.011264   \n",
        "number_of_dependents                           0.005896        0.064060   \n",
        "\n",
        "                                             number_of_open_credit_lines_and_loans  \\\n",
        "serious_dlqin2yrs                                                        -0.026876   \n",
        "revolving_utilization_of_unsecured_lines                                 -0.010750   \n",
        "age                                                                       0.186420   \n",
        "number_of_time30-59_days_past_due_not_worse                              -0.047053   \n",
        "debt_ratio                                                                0.008032   \n",
        "monthly_income                                                            0.089260   \n",
        "number_of_open_credit_lines_and_loans                                     1.000000   \n",
        "number_of_times90_days_late                                              -0.074116   \n",
        "number_real_estate_loans_or_lines                                         0.426890   \n",
        "number_of_time60-89_days_past_due_not_worse                              -0.063968   \n",
        "number_of_dependents                                                      0.041384   \n",
        "\n",
        "                                             number_of_times90_days_late  \\\n",
        "serious_dlqin2yrs                                               0.108921   \n",
        "revolving_utilization_of_unsecured_lines                       -0.001091   \n",
        "age                                                            -0.051162   \n",
        "number_of_time30-59_days_past_due_not_worse                     0.977052   \n",
        "debt_ratio                                                     -0.002738   \n",
        "monthly_income                                                 -0.012778   \n",
        "number_of_open_credit_lines_and_loans                          -0.074116   \n",
        "number_of_times90_days_late                                     1.000000   \n",
        "number_real_estate_loans_or_lines                              -0.041557   \n",
        "number_of_time60-89_days_past_due_not_worse                     0.990128   \n",
        "number_of_dependents                                           -0.000703   \n",
        "\n",
        "                                             number_real_estate_loans_or_lines  \\\n",
        "serious_dlqin2yrs                                                    -0.005976   \n",
        "revolving_utilization_of_unsecured_lines                              0.007419   \n",
        "age                                                                   0.066435   \n",
        "number_of_time30-59_days_past_due_not_worse                          -0.026220   \n",
        "debt_ratio                                                            0.015818   \n",
        "monthly_income                                                        0.123512   \n",
        "number_of_open_credit_lines_and_loans                                 0.426890   \n",
        "number_of_times90_days_late                                          -0.041557   \n",
        "number_real_estate_loans_or_lines                                     1.000000   \n",
        "number_of_time60-89_days_past_due_not_worse                          -0.035835   \n",
        "number_of_dependents                                                  0.119559   \n",
        "\n",
        "                                             number_of_time60-89_days_past_due_not_worse  \\\n",
        "serious_dlqin2yrs                                                               0.091995   \n",
        "revolving_utilization_of_unsecured_lines                                       -0.000975   \n",
        "age                                                                            -0.046745   \n",
        "number_of_time30-59_days_past_due_not_worse                                     0.981492   \n",
        "debt_ratio                                                                     -0.001608   \n",
        "monthly_income                                                                 -0.011264   \n",
        "number_of_open_credit_lines_and_loans                                          -0.063968   \n",
        "number_of_times90_days_late                                                     0.990128   \n",
        "number_real_estate_loans_or_lines                                              -0.035835   \n",
        "number_of_time60-89_days_past_due_not_worse                                     1.000000   \n",
        "number_of_dependents                                                           -0.002197   \n",
        "\n",
        "                                             number_of_dependents  \n",
        "serious_dlqin2yrs                                        0.047273  \n",
        "revolving_utilization_of_unsecured_lines                 0.002690  \n",
        "age                                                     -0.206503  \n",
        "number_of_time30-59_days_past_due_not_worse              0.007043  \n",
        "debt_ratio                                               0.005896  \n",
        "monthly_income                                           0.064060  \n",
        "number_of_open_credit_lines_and_loans                    0.041384  \n",
        "number_of_times90_days_late                             -0.000703  \n",
        "number_real_estate_loans_or_lines                        0.119559  \n",
        "number_of_time60-89_days_past_due_not_worse             -0.002197  \n",
        "number_of_dependents                                     1.000000  "
       ]
      }
     ],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train_w_monthly_income.corr().ix[:,5]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 38,
       "text": [
        "serious_dlqin2yrs                             -0.019663\n",
        "revolving_utilization_of_unsecured_lines       0.007388\n",
        "age                                            0.037199\n",
        "number_of_time30-59_days_past_due_not_worse   -0.010458\n",
        "debt_ratio                                    -0.026406\n",
        "monthly_income                                 1.000000\n",
        "number_of_open_credit_lines_and_loans          0.089260\n",
        "number_of_times90_days_late                   -0.012778\n",
        "number_real_estate_loans_or_lines              0.123512\n",
        "number_of_time60-89_days_past_due_not_worse   -0.011264\n",
        "number_of_dependents                           0.064060\n",
        "Name: monthly_income"
       ]
      }
     ],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cols = ['number_real_estate_loans_or_lines', 'number_of_open_credit_lines_and_loans']\n",
      "income_imputer.fit(train_w_monthly_income[cols], train_w_monthly_income.monthly_income)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 39,
       "text": [
        "KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',\n",
        "          n_neighbors=1, p=2, weights='uniform')"
       ]
      }
     ],
     "prompt_number": 39
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Replace the mising values"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "new_values = income_imputer.predict(train_w_null_monthly_income[cols])\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 40
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train_w_null_monthly_income['monthly_income'] = new_values\n",
      "new_values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 41,
       "text": [
        "array([  7666.,   3235.,   9666., ...,   2155.,   3771.,  10000.])"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#combine the data back together\n",
      "train = train_w_monthly_income.append(train_w_null_monthly_income)\n",
      "len(train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 42,
       "text": [
        "112415"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "test['monthly_income_imputed'] = income_imputer.predict(test[cols])\n",
      "test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <th>age</th>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <th>debt_ratio</th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <th>number_of_dependents</th>\n",
        "      <th>monthly_income_imputed</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.233810</td>\n",
        "      <td> 30</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.036050</td>\n",
        "      <td>  3300</td>\n",
        "      <td>  5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6017</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.964673</td>\n",
        "      <td> 40</td>\n",
        "      <td> 3</td>\n",
        "      <td>    0.382965</td>\n",
        "      <td> 13700</td>\n",
        "      <td>  9</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 2</td>\n",
        "      <td> 2850</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.061086</td>\n",
        "      <td> 78</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2058.000000</td>\n",
        "      <td>   NaN</td>\n",
        "      <td> 10</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2500</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.075427</td>\n",
        "      <td> 32</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.085512</td>\n",
        "      <td>  7916</td>\n",
        "      <td>  6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 4145</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.046560</td>\n",
        "      <td> 58</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.241622</td>\n",
        "      <td>  2416</td>\n",
        "      <td>  9</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2850</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 43,
       "text": [
        "    serious_dlqin2yrs  revolving_utilization_of_unsecured_lines  age  \\\n",
        "3                   0                                  0.233810   30   \n",
        "13                  1                                  0.964673   40   \n",
        "16                  0                                  0.061086   78   \n",
        "23                  0                                  0.075427   32   \n",
        "24                  0                                  0.046560   58   \n",
        "\n",
        "    number_of_time30-59_days_past_due_not_worse   debt_ratio  monthly_income  \\\n",
        "3                                             0     0.036050            3300   \n",
        "13                                            3     0.382965           13700   \n",
        "16                                            0  2058.000000             NaN   \n",
        "23                                            0     0.085512            7916   \n",
        "24                                            0     0.241622            2416   \n",
        "\n",
        "    number_of_open_credit_lines_and_loans  number_of_times90_days_late  \\\n",
        "3                                       5                            0   \n",
        "13                                      9                            3   \n",
        "16                                     10                            0   \n",
        "23                                      6                            0   \n",
        "24                                      9                            0   \n",
        "\n",
        "    number_real_estate_loans_or_lines  \\\n",
        "3                                   0   \n",
        "13                                  1   \n",
        "16                                  2   \n",
        "23                                  0   \n",
        "24                                  1   \n",
        "\n",
        "    number_of_time60-89_days_past_due_not_worse  number_of_dependents  \\\n",
        "3                                             0                     0   \n",
        "13                                            1                     2   \n",
        "16                                            0                     0   \n",
        "23                                            0                     0   \n",
        "24                                            0                     0   \n",
        "\n",
        "    monthly_income_imputed  \n",
        "3                     6017  \n",
        "13                    2850  \n",
        "16                    2500  \n",
        "23                    4145  \n",
        "24                    2850  "
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "test['monthly_income'] = np.where(test.monthly_income.isnull(), test.monthly_income_imputed,\n",
      "                                  test.monthly_income)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 44
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print pd.value_counts(train.monthly_income.isnull())\n",
      "print pd.value_counts(test.monthly_income.isnull())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "False    112415\n",
        "False    37585\n"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "train.to_csv(\"./data/credit-data-trainingset.csv\", index=False)\n",
      "test.to_csv(\"./data/credit-data-testset.csv\", index=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 46
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Trying your own imputation strategy\n",
      "Partner with the person next to you and think of other ways you might be able to impute missing valus for the `monthly_income` column.\n",
      "\n",
      "Things to consider:\n",
      "\n",
      "- Are `number_of_open_credit_lines_and_loans` and `number_real_estate_loans_or_lines` the best predictors?\n",
      "- What are some other methods you could use for replacing nulls?\n",
      "- What about handling outlying values? Should a montly income of $3MM be treated as missing?\n",
      "- Check out the [scikit-learn docs](http://scikit-learn.org/stable/auto_examples/imputation.html) for more examples\n",
      "- How might you evaluate the effectiveness of your imputation?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##We just did the following\n",
      "\n",
      "- Split our data into a training set for building our model and a test set for evaluating its performance\n",
      "- Used KNearestNeighbors to fill in missing values for `monthly_income`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}