notebooks/1.0-full-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# From Raw Data to Predictions\n",
    "\n",
    "This notebook is designed as a follow-up to the [Machine Learning with the Experts: School Budgets](https://www.datacamp.com/courses/machine-learning-with-the-experts-school-budgets) course on Datacamp. We won't explain all the tools and techniques we use here. If you're curious about any of the tools, code, or methods used here, make sure to check out the course!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "# ignore deprecation warnings in sklearn\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import os\n",
    "import sys\n",
    "\n",
    "# add the 'src' directory as one where we can import modules\n",
    "src_dir = os.path.join(os.getcwd(), os.pardir, 'src')\n",
    "sys.path.append(src_dir)\n",
    "\n",
    "from data.multilabel import multilabel_sample_dataframe, multilabel_train_test_split\n",
    "from features.SparseInteractions import SparseInteractions\n",
    "from models.metrics import multi_multi_log_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Load Data\n",
    "\n",
    "First, we'll load the entire training data set available from DrivenData. In order to make this notebook run, you will need to: \n",
    " - [Sign up for an account on DrivenData](http://www.drivendata.org)\n",
    " - [Join the Box-plots for education competition](https://www.drivendata.org/competitions/46/box-plots-for-education-reboot/)\n",
    " - Download the competition data to the `data` folder in this repository. Files should be named `TrainingSet.csv` and `TestSet.csv`.\n",
    " - Enjoy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(400277, 25)\n"
     ]
    }
   ],
   "source": [
    "path_to_training_data = os.path.join(os.pardir,\n",
    "                                     'data',\n",
    "                                     'TrainingSet.csv')\n",
    "\n",
    "df = pd.read_csv(path_to_training_data, index_col=0)\n",
    "\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Resample Data\n",
    "\n",
    "400,277 rows is too many to work with locally while we develop our approach. We'll sample down to 10,000 rows so that it is easy and quick to run our analysis.\n",
    "\n",
    "We'll also create dummy variables for our labels and split our sampled dataset into a training set and a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "LABELS = ['Function',\n",
    "          'Use',\n",
    "          'Sharing',\n",
    "          'Reporting',\n",
    "          'Student_Type',\n",
    "          'Position_Type',\n",
    "          'Object_Type', \n",
    "          'Pre_K',\n",
    "          'Operating_Status']\n",
    "\n",
    "NON_LABELS = [c for c in df.columns if c not in LABELS]\n",
    "\n",
    "SAMPLE_SIZE = 40000\n",
    "\n",
    "sampling = multilabel_sample_dataframe(df,\n",
    "                                       pd.get_dummies(df[LABELS]),\n",
    "                                       size=SAMPLE_SIZE,\n",
    "                                       min_count=25,\n",
    "                                       seed=43)\n",
    "\n",
    "dummy_labels = pd.get_dummies(sampling[LABELS])\n",
    "\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(sampling[NON_LABELS],\n",
    "                                                               dummy_labels,\n",
    "                                                               0.2,\n",
    "                                                               min_count=3,\n",
    "                                                               seed=43)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Create preprocessing tools\n",
    "\n",
    "We need tools to preprocess our text and numeric data. We'll create those tools here. The `combine_text_columns` function will take a DataFrame of text columns and return a single series where all of the text in the columns has been joined together.\n",
    "\n",
    "We'll then create `FunctionTransformer` objects that select our text and numeric data from the dataframe.\n",
    "\n",
    "Finally, we create a custom scoring method that uses the `multi_multi_log_loss` function that is the evaluation metric for the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "NUMERIC_COLUMNS = ['FTE', \"Total\"]\n",
    "\n",
    "def combine_text_columns(data_frame, to_drop=NUMERIC_COLUMNS + LABELS):\n",
    "    \"\"\" Takes the dataset as read in, drops the non-feature, non-text columns and\n",
    "        then combines all of the text columns into a single vector that has all of\n",
    "        the text for a row.\n",
    "        \n",
    "        :param data_frame: The data as read in with read_csv (no preprocessing necessary)\n",
    "        :param to_drop (optional): Removes the numeric and label columns by default.\n",
    "    \"\"\"\n",
    "    # drop non-text columns that are in the df\n",
    "    to_drop = set(to_drop) & set(data_frame.columns.tolist())\n",
    "    text_data = data_frame.drop(to_drop, axis=1)\n",
    "    \n",
    "    # replace nans with blanks\n",
    "    text_data.fillna(\"\", inplace=True)\n",
    "    \n",
    "    # joins all of the text items in a row (axis=1)\n",
    "    # with a space in between\n",
    "    return text_data.apply(lambda x: \" \".join(x), axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS], validate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38     OTHER PURCHASED SERVICES  SCHOOL-WIDE SCHOOL P...\n",
       "70     Extra Duty Pay/Overtime For Support Personnel ...\n",
       "198    Supplemental *  Operation and Maintenance of P...\n",
       "209    REPAIR AND MAINTENANCE SERVICES  PUPIL TRANSPO...\n",
       "614     GENERAL EDUCATION LOCAL EDUCATIONAL AIDE,70 H...\n",
       "dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_text_data.fit_transform(sampling.head(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FTE</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NaN</td>\n",
       "      <td>653.460000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2153.530000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-8291.860000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>NaN</td>\n",
       "      <td>618.290000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614</th>\n",
       "      <td>0.71</td>\n",
       "      <td>21747.666875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      FTE         Total\n",
       "38    NaN    653.460000\n",
       "70    NaN   2153.530000\n",
       "198   NaN  -8291.860000\n",
       "209   NaN    618.290000\n",
       "614  0.71  21747.666875"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_numeric_data.fit_transform(sampling.head(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.scorer import make_scorer\n",
    "\n",
    "log_loss_scorer = make_scorer(multi_multi_log_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train model pipeline\n",
    "\n",
    "Now we'll train the final pipeline from the course that takes text and numeric data, does the necessary preprocessing, and trains the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import chi2, SelectKBest\n",
    "\n",
    "from sklearn.pipeline import Pipeline, FeatureUnion\n",
    "\n",
    "from sklearn.preprocessing import Imputer\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from sklearn.preprocessing import MaxAbsScaler\n",
    "\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logloss score of trained pipeline:  2.14831287426\n",
      "CPU times: user 21min 28s, sys: 53.8 s, total: 22min 22s\n",
      "Wall time: 6min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# set a reasonable number of features before adding interactions\n",
    "chi_k = 300\n",
    "\n",
    "# create the pipeline object\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', HashingVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                                     non_negative=True, norm=None, binary=False,\n",
    "                                                     ngram_range=(1, 2))),\n",
    "                    ('dim_red', SelectKBest(chi2, chi_k))\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('int', SparseInteractions(degree=2)),\n",
    "        ('scale', MaxAbsScaler()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# fit the pipeline to our training data\n",
    "pl.fit(X_train, y_train.values)\n",
    "\n",
    "# print the score of our trained pipeline on our test set\n",
    "print(\"Logloss score of trained pipeline: \", log_loss_scorer(pl, X_test, y_test.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict holdout set and write submission\n",
    "\n",
    "Finally, we want to use our trained pipeline to predict the holdout dataset. We will write our predictions to a file, `predictions.csv`, that we can submit on [DrivenData](http://www.drivendata.org)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "path_to_holdout_data = os.path.join(os.pardir,\n",
    "                                    'data',\n",
    "                                    'TestSet.csv')\n",
    "\n",
    "# Load holdout data\n",
    "holdout = pd.read_csv(path_to_holdout_data, index_col=0)\n",
    "\n",
    "# Make predictions\n",
    "predictions = pl.predict_proba(holdout)\n",
    "\n",
    "# Format correctly in new DataFrame: prediction_df\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(df[LABELS]).columns,\n",
    "                             index=holdout.index,\n",
    "                             data=predictions)\n",
    "\n",
    "\n",
    "# Save prediction_df to csv called \"predictions.csv\"\n",
    "prediction_df.to_csv(\"predictions.csv\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}