PISAMEX.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Science 180\n",
    "\n",
    "\n",
    "\n",
    "![Alt text](http://thegeektown.com/wp-content/uploads/2015/03/data-scientist.jpg)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "An adventure through 15 dimensions of data wrangling, visualization and modeling at mind-bending speeds.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## In a world with too much data sitting around and not enough insight, to whom will we turn for help?\n",
    "\n",
    "![Alt](http://nextviewventures.com/blog/wp-content/uploads/2014/07/control-content-marekting-for-startups.jpg)\n",
    "\n",
    "##YOU!    (Neo was busy...)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### You must learn how to wrangle data in the next few hours in order to save the education system. If you fail, we're all doomed...\n",
    "\n",
    "### You have been given a dataset of test results and metadata, along with a laptop computer. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#Good luck, everything depends on you.\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "exam_data = pd.read_csv('data/PISA2009_Scored_Tests_MEX.csv')\n",
    "bio_data = pd.read_csv('data/PISA2009_Questionnaire_MEX.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Meet the data\n",
    "- 2009 OECD Programme for International Student Assessment\n",
    "- This presentation focused on Mexico, most participants\n",
    "- One file containing results of student test results\n",
    "- Another file containing questionnaires completed by students"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MATH - P2000 Population Pyramids (Q02)</th>\n",
       "      <th>MATH - P2000 Population Pyramids (Q03)</th>\n",
       "      <th>MATH - P2000 Population Pyramids (Q04)</th>\n",
       "      <th>MATH - P2000 Containers (Q01)</th>\n",
       "      <th>MATH - P2000 Pipelines (Q01)</th>\n",
       "      <th>MATH - P2003 Running Tracks (Q01)</th>\n",
       "      <th>MATH - P2003 Running Tracks (Q02)</th>\n",
       "      <th>MATH - P2003 Lotteries (Q01)</th>\n",
       "      <th>MATH - P2003 Diving (Q01)</th>\n",
       "      <th>MATH - P2003 Diving (Q02)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MATH - P2000 Population Pyramids (Q02)  \\\n",
       "0                                     NaN   \n",
       "1                                     NaN   \n",
       "2                                     NaN   \n",
       "3                                       1   \n",
       "4                                     NaN   \n",
       "\n",
       "   MATH - P2000 Population Pyramids (Q03)  \\\n",
       "0                                     NaN   \n",
       "1                                     NaN   \n",
       "2                                     NaN   \n",
       "3                                       0   \n",
       "4                                     NaN   \n",
       "\n",
       "   MATH - P2000 Population Pyramids (Q04)  MATH - P2000 Containers (Q01)  \\\n",
       "0                                     NaN                              0   \n",
       "1                                     NaN                            NaN   \n",
       "2                                     NaN                            NaN   \n",
       "3                                       0                            NaN   \n",
       "4                                     NaN                            NaN   \n",
       "\n",
       "   MATH - P2000 Pipelines (Q01)  MATH - P2003 Running Tracks (Q01)  \\\n",
       "0                             0                                  1   \n",
       "1                           NaN                                NaN   \n",
       "2                             1                                NaN   \n",
       "3                           NaN                                NaN   \n",
       "4                             0                                NaN   \n",
       "\n",
       "   MATH - P2003 Running Tracks (Q02)  MATH - P2003 Lotteries (Q01)  \\\n",
       "0                                  0                             0   \n",
       "1                                NaN                           NaN   \n",
       "2                                NaN                             1   \n",
       "3                                NaN                           NaN   \n",
       "4                                NaN                             0   \n",
       "\n",
       "   MATH - P2003 Diving (Q01)  MATH - P2003 Diving (Q02)  \n",
       "0                        NaN                        NaN  \n",
       "1                        NaN                        NaN  \n",
       "2                        NaN                        NaN  \n",
       "3                          0                          0  \n",
       "4                        NaN                        NaN  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exam_cols = exam_data.columns\n",
    "exam_data[exam_cols[10:20]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Attend &lt;ISCED 0&gt;</th>\n",
       "      <th>Age at &lt;ISCED 1&gt;</th>\n",
       "      <th>Repeat &lt;ISCED 1&gt;</th>\n",
       "      <th>Repeat &lt;ISCED 2&gt;</th>\n",
       "      <th>Repeat &lt;ISCED 3&gt;</th>\n",
       "      <th>At Home - Mother</th>\n",
       "      <th>At Home - Father</th>\n",
       "      <th>At Home - Brothers</th>\n",
       "      <th>At Home - Sisters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Female</td>\n",
       "      <td>Yes, more than one year</td>\n",
       "      <td>6</td>\n",
       "      <td>Yes, once</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>6</td>\n",
       "      <td>Yes, twice or more</td>\n",
       "      <td>No, never</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Female</td>\n",
       "      <td>Yes, more than one year</td>\n",
       "      <td>6</td>\n",
       "      <td>Yes, twice or more</td>\n",
       "      <td>No, never</td>\n",
       "      <td>No, never</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Female</td>\n",
       "      <td>Yes, more than one year</td>\n",
       "      <td>7</td>\n",
       "      <td>Yes, once</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes, twice or more</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Sex         Attend <ISCED 0>  Age at <ISCED 1>    Repeat <ISCED 1>  \\\n",
       "0  Female  Yes, more than one year                 6           Yes, once   \n",
       "1  Female                       No                 6  Yes, twice or more   \n",
       "2  Female  Yes, more than one year                 6  Yes, twice or more   \n",
       "3  Female  Yes, more than one year                 7           Yes, once   \n",
       "4  Female                       No               NaN                 NaN   \n",
       "\n",
       "     Repeat <ISCED 2> Repeat <ISCED 3> At Home - Mother At Home - Father  \\\n",
       "0                 NaN              NaN              Yes              Yes   \n",
       "1           No, never              NaN              Yes              Yes   \n",
       "2           No, never        No, never              Yes              Yes   \n",
       "3                 NaN              NaN              Yes               No   \n",
       "4  Yes, twice or more              NaN              NaN              NaN   \n",
       "\n",
       "  At Home - Brothers At Home - Sisters  \n",
       "0                Yes               Yes  \n",
       "1                NaN               NaN  \n",
       "2                Yes                No  \n",
       "3                Yes                No  \n",
       "4                NaN               NaN  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bio_cols = bio_data.columns\n",
    "bio_data[bio_cols[10:20]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###Where do we begin? \n",
    "\n",
    "\n",
    "###You know Python right? It does data stuff, right? OK, let's get started then.\n",
    "\n",
    "###First thing to figure out is how to get the data files on your machine into Python in the first place.\n",
    "\n",
    "\n",
    "###You've heard of this library called Pandas from another Agent -- it once saved them in a pinch. Lacking any better ideas, let's open up an editor and see if we can't at least cross the starting line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Reading data from disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "exam_data = pd.read_csv('data/PISA2009_Scored_Tests_MEX.csv')\n",
    "bio_data = pd.read_csv('data/PISA2009_Questionnaire_MEX.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###What did that just do? \n",
    "\n",
    "\n",
    "\n",
    "We called \"read_csv\", which presumably reads CSV files... and does what with them? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##pd.read_csv does a magical thing \n",
    "\n",
    "\n",
    "It reads a CSV file into a DataFrame. \n",
    "\n",
    "DataFrames are mystical creatures in Data Science. \n",
    "\n",
    "Popularized by R, they provide a standardized MATRIX-style format for interacting with your data. Most data can fit into this row and column format: financial transactions, iPhone app user records, medical histories, etc.\n",
    "\n",
    "(And you thought the Matrix references were just for fun)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Alt](http://www.bigdataexaminer.com/wp-content/uploads/2014/12/screen-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Since you were wondering\n",
    "\n",
    "##Pandas has support for many formats\n",
    "\n",
    "CSV, Text (tab separated, pipe separated, etc.), Excel, JSON, HTML, SQL, Stuff copied to your clipboard, HDF5..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Hold up. What's really going on with these DataFrames?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Two data structures: Series and DataFrame\n",
    "\n",
    "###Series\n",
    "Think of this as one column of your data - one data type.\n",
    "\n",
    "### DataFrame\n",
    "All of the columns in your data. Mixed data types. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Many Series can be combined and represented as a DataFrame object.\n",
    "\n",
    "#A DataFrame can be represented as many Series objects. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#Pandas provides tons of functions to:\n",
    "\n",
    "###slice, dice, merge, join, group by, select, append, find, transform, sort, reverse, pivot and anything else you want to do\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#... for both Series and DataFrames. \n",
    "\n",
    "Most functions are designed to work with either type or even combinations of the two, just like you would intuitively expect:\n",
    "\n",
    "i.e. A concat function can contatenate arbitrary combinations of 0 to n Series and DataFrames."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#Accessing data in a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Get one column (Series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "exam_data['Student ID 5-digit'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Get a subset of columns (DataFrame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "exam_data[['Student ID 5-digit', 'School ID 5-digit']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Get a subset of rows using a boolean array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "exam_data[exam_data['School ID 5-digit'] == 2].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#So you know a bit about DataFrames?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "exam_data = pd.read_csv('data/PISA2009_Scored_Tests_MEX.csv')\n",
    "bio_data = pd.read_csv('data/PISA2009_Questionnaire_MEX.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Fine. What's next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Exploratory Data Analysis. \n",
    "\n",
    "What is in those files anyway?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does it look like test data should? \n",
    "\n",
    "Is it completely empty? Full? Lots of missing values and NaNs?\n",
    "\n",
    "What are in the rows? columns?\n",
    "\n",
    "Does it have appropriate features? (characteristics common to records belonging to a dataset)\n",
    "\n",
    "###It's impossible to make good decisions moving forward until we know more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can just output the entire dataframe to the console, but that doesn't scale beyond a couple hundred rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<pre>\n",
    "In [1]: df = DataFrame(np.random.randn(10, 4))\n",
    "\n",
    "In [2]: df\n",
    "Out[2]: \n",
    "          0         1         2         3\n",
    "0  0.469112 -0.282863 -1.509059 -1.135632\n",
    "1  1.212112 -0.173215  0.119209 -1.044236\n",
    "2 -0.861849 -2.104569 -0.494929  1.071804\n",
    "3  0.721555 -0.706771 -1.039575  0.271860\n",
    "4 -0.424972  0.567020  0.276232 -1.087401\n",
    "5 -0.673690  0.113648 -1.478427  0.524988\n",
    "6  0.404705  0.577046 -1.715002 -1.039268\n",
    "7 -0.370647 -1.157892 -1.344312  0.844885\n",
    "8  1.075770 -0.109050  1.643563 -1.469388\n",
    "9  0.357021 -0.674600 -1.776904 -0.968914\n",
    "</pre>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Pandas gives us a number of tools: \n",
    "\n",
    "<br>\n",
    "    \n",
    "    .head(n)\n",
    "    .info()\n",
    "    .describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "exam_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "exam_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "exam_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#We have two files, and both of them have a feature named 'Student ID 5-digit'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#Using this unique ID as our guide, we can match the exam scores and biographical data for a single student."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#This task comes up a lot in data wrangling, since different kinds of data will be stored in different data stores. Often one of the first steps is to combine the relevant sections from each repository of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "useless = {\n",
    "    u' Version of cognitive database and date of release',\n",
    "    u'3-character country code ',\n",
    "    u'Adjudicated sub-region',\n",
    "}\n",
    "\n",
    "not_questions = {u'Booklet', \n",
    "                 u'School ID 5-digit', \n",
    "                 u'Student ID 5-digit',\n",
    "                 u'OECD country',\n",
    "                 u'Country code ISO 3-digit',}\n",
    "\n",
    "score_mapping = {\n",
    "    'Score 0': 0,\n",
    "    'Score 1': 1,\n",
    "    'Score 2': 2,\n",
    "    'Not reached': 0,\n",
    "}\n",
    "\n",
    "questions = set(exam_data.columns) - not_questions - useless\n",
    "\n",
    "for question in questions:\n",
    "    exam_data[question] = exam_data[question].map(score_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "math_qs = {q for q in questions if q.startswith('MATH')}\n",
    "read_qs = {q for q in questions if q.startswith('READ')}\n",
    "scie_qs = {q for q in questions if q.startswith('SCIE')}\n",
    "    \n",
    "totals = exam_data[list(questions)].sum(axis=1)\n",
    "math_score = exam_data[list(math_qs)].sum(axis=1)\n",
    "read_score = exam_data[list(read_qs)].sum(axis=1)\n",
    "scie_score = exam_data[list(scie_qs)].sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Let's merge into one DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "score_df = pd.DataFrame({'Total Score': totals, \n",
    "                         'Math Score': math_score, \n",
    "                         'Reading Score': read_score, \n",
    "                         'Science Score': scie_score,\n",
    "                         'Student ID 5-digit': exam_data['Student ID 5-digit']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data = pd.merge(score_df, bio_data, on='Student ID 5-digit')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#Get summary info for any column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data['Reading Enjoyment Time'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Particularly important for later\n",
    "\n",
    "Looking for Missing Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Now that you can load data and look at it\n",
    "\n",
    "#It's time for exercise #1 \n",
    "\n",
    "Go to https://github.com/dsakagi/odsc-pisa to clone the code for this talk\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<pre>\n",
    "$ mkvirtualenv odsc\n",
    "$ git clone https://github.com/dsakagi/odsc-pisa.git\n",
    "$ cd odsc-pisa\n",
    "$ mkdir data\n",
    "$ make\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#What about relationships between variables?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "##Should we compute covariance?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#Nah, let's make some plots!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#A quick aside.\n",
    "##IPython Notebooks (like this one) are great.\n",
    "##Plots in your notebook!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Two plotting weapons\n",
    "\n",
    "###Matplotlib\n",
    "\n",
    "- The historical go-to for plotting\n",
    "- allows lots of fine-grained control\n",
    "- built with numpy in mind (Numpy and its cousin Scipy are the number-crunching go-to's in Python)\n",
    "\n",
    "###Seaborn\n",
    "\n",
    "- Expressive power\n",
    "- built with pandas in mind\n",
    "- trendy newcomer, but gaining a loyal following"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We will mainly use seaborn examples in this presentation. It's very intuitive and powerful to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Scatterplots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "View relationship between two continuous variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sns.jointplot(data['Index of economic, social and cultural status (WLE)'], \n",
    "              data['Home Possessions'], kind=\"hex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In this dataset, several variables can stand as proxies for socio-economic status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Histogram\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Much better than simple averages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data.groupby('Sex')['Total Score'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "groups = data.groupby('Sex').groups\n",
    "for key, row_ids in groups.iteritems():\n",
    "    pylab.hist(data['Total Score'][row_ids].values,\n",
    "               normed=True,\n",
    "               bins=np.linspace(0, 70, 11),\n",
    "               alpha=0.35,\n",
    "               label=str(key))\n",
    "pylab.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Doesn't work as well for more than two levels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "groups = data.groupby('Grade').groups\n",
    "for key, row_ids in groups.iteritems():\n",
    "    pylab.hist(data['Total Score'][row_ids].values,\n",
    "               normed=True,\n",
    "               bins=np.linspace(0, 70, 11),\n",
    "               alpha=0.35,\n",
    "               label=str(key))\n",
    "pylab.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Violin Plots work better for comparing several distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "nonnull_subset = data['Total Score'].notnull()\n",
    "plt.figure(figsize=(12, 6))\n",
    "sns.violinplot(data['Total Score'][nonnull_subset], \n",
    "               data['Father  <Highest Schooling>'][nonnull_subset], \n",
    "               inner='box',\n",
    "               bw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Alternatively, use `FacetGrid`\n",
    "###Visualize interactions of variables on response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "no_nans = data[data['Sex'].notnull() & \n",
    "               data['At Home - Mother'].notnull()]\n",
    "\n",
    "g = sns.FacetGrid(no_nans, row=\"Sex\", col=\"At Home - Mother\", \n",
    "                  margin_titles=True, dropna=True)\n",
    "bins = np.linspace(0, 67, 13)\n",
    "g.map(plt.hist, \"Total Score\", color=\"steelblue\", bins=bins, \n",
    "      lw=0, normed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###If distribution not required, try factor plot to get a simpler comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "no_nans = data[data['Sex'].notnull() & \n",
    "               data['At Home - Father'].notnull()]\n",
    "\n",
    "g = sns.factorplot(\"At Home - Father\", \"Total Score\", \"Sex\",\n",
    "                    data=no_nans, kind=\"bar\",\n",
    "                    size=6, palette=\"muted\", dropna=True)\n",
    "g.set_ylabels(\"Mean Score\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "no_nans = data[data['Sex'].notnull() & \n",
    "               data['At Home - Father'].notnull() &\n",
    "               data['At Home - Mother'].notnull()]\n",
    "\n",
    "g = sns.factorplot(\"Sex\", \"Total Score\", \"Sex\",\n",
    "                   row=\"At Home - Mother\",\n",
    "                   col=\"At Home - Father\",\n",
    "                   data=no_nans, kind=\"bar\",\n",
    "                   size=6, palette=\"muted\",\n",
    "                   dropna=True)\n",
    "g.set_ylabels(\"Mean Score\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Heatmaps are a visual tool to look at the distribution of observations _across_ settings of two variables (as opposed to _within_ a setting like `FacetGrid`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Need to use ``pivot_table`` to get data in expected order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "ptable = pd.pivot_table(\n",
    "    data, \n",
    "    values='Total Score', \n",
    "    index='Like Read - Fiction', \n",
    "    columns='Like Read - Non-fiction books')\n",
    "sns.heatmap(ptable, annot=True, fmt=\"f\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "###Not very useful if not in order..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Heatmaps - Round 2\n",
    "####Effects of variables over a range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "display_order = [\n",
    "    u'Never or almost never', u'A few times a year', \n",
    "    u'About once a month', u'Several times a month', \n",
    "    u'Several times a week'\n",
    "]\n",
    "display_table = ptable[display_order].reindex(reversed(display_order))\n",
    "sns.heatmap(display_table, annot=True, fmt=\"f\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Pivot tables can do other aggregations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "count_table = pd.pivot_table(\n",
    "    data, values='Total Score', \n",
    "    index='Like Read - Fiction', \n",
    "    columns='Like Read - Non-fiction books',\n",
    "    aggfunc=np.count_nonzero)  # <--- Look here\n",
    "\n",
    "sns.heatmap(count_table[display_order].reindex(reversed(display_order)), annot=True, fmt='f')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Scatterplot\n",
    "\n",
    "What is the relationship between X and Y? (If any!)\n",
    "\n",
    "Positive, Negative, Linear, Non-linear...\n",
    "\n",
    "Are there clusters of similar observations?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Looking for any general relationship"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "sns.jointplot(\n",
    "    \"Total Score\", \n",
    "    'Index of economic, social and cultural status (WLE)', \n",
    "    data, kind='hex')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Now for exercise 2\n",
    "\n",
    "###Use your newfound knowledge\n",
    "###Make a few plots using seaborn\n",
    "###Find interesting relationships",
    "When you find something cool, call us over!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Scikit-Learn\n",
    "\n",
    "# Widely used machine learning package\n",
    "\n",
    "- Classification Models\n",
    "- Regression Models\n",
    "- Clustering techniques\n",
    "- Dimensionality Reduction\n",
    "- Preprocessing\n",
    "- ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Experimental Design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Build a model that is useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Can only build model on data that has a known output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Our data has missing values in the target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Today, we'll drop those observations. \n",
    "\n",
    "Just as a note, sometimes this isn't a good thing to do (think of a clinical study...)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model_data = data[data['Total Score'].notnull()]\n",
    "target = model_data.pop('Total Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Avoiding overfitting\n",
    "- Need to keep data we train on from data we validate on\n",
    "- Otherwise the results will be overly optimistic\n",
    "- ... and ultimately, the model will perform poorly on new data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sklearn.cross_validation\n",
    "\n",
    "(train_data, \n",
    " test_data, \n",
    " train_target, \n",
    " test_target) = sklearn.cross_validation.train_test_split(\n",
    "    model_data, target, test_size=0.2, random_state=1337\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "####Today we will use simple train/test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "####Recommend using multiple cross-validation folds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Variable Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###Most statistical models require numeric encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "###Many will choke on missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "###Need some massaging before fitting a statistical learner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "data['Repeat <ISCED 1>'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "data['Mother  <Highest Schooling>'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import sklearn.preprocessing\n",
    "import sklearn.feature_extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "encoder = DictVectorizer(sparse=False)\n",
    "\n",
    "our_categ_vars = [\n",
    "    'Repeat <ISCED 1>',\n",
    "    'Mother  <Highest Schooling>',\n",
    "    'Possessions Internet',\n",
    "    'Online - Reading Emails',\n",
    "    'Sex',\n",
    "    'Reading Tasks - Memorise text',\n",
    "]\n",
    "\n",
    "vardata = train_data[our_categ_vars].fillna('MISSING')\n",
    "encoder.fit(vardata.to_dict(orient='records'))\n",
    "\n",
    "train_catdata = encoder.transform(vardata.to_dict(orient='records'))\n",
    "\n",
    "test_vardata = test_data[our_categ_vars].fillna('MISSING')\n",
    "test_catdata = encoder.transform(\n",
    "    test_data[our_categ_vars].to_dict(orient='records'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Missing values in numeric columns\n",
    "\n",
    "###We will impute with the median value\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Imputer\n",
    "imputer = Imputer(strategy='median')\n",
    "\n",
    "our_num_vars = [\n",
    "    'Grade',\n",
    "    'Home Possessions',\n",
    "    'Joy/Like Reading',\n",
    "    'Diversity reading',\n",
    "    'No of ALL <class period> a week',\n",
    "    'Reading for School: Traditional literature courses',\n",
    "    'Min in <class period> for <Maths>'\n",
    "]\n",
    "\n",
    "numdata = train_data[our_num_vars]\n",
    "imputer.fit(numdata)\n",
    "\n",
    "train_numdata = imputer.transform(numdata)\n",
    "test_numdata = imputer.transform(test_data[our_num_vars])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Now, we put our training data back together again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "train_this = np.hstack([train_numdata, train_catdata])\n",
    "test_this = np.hstack([test_numdata, test_catdata])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Finally, ready to build a model\n",
    "###Linear Regression is a sensible first choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(train_this, train_target)\n",
    "\n",
    "lr_predictions = pd.Series(lr.predict(test_this),\n",
    "                           name='Linear Regression')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "p_df = pd.DataFrame({'Prediction': lr_predictions,\n",
    "                     'Actual': test_target.values})\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "sns.jointplot('Actual', 'Prediction', data=p_df, \n",
    "              kind=\"hex\", color=sns.color_palette()[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Was our model any good?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "test_metrics = {\n",
    "    'Explained Variance': metrics.explained_variance_score,\n",
    "    'MAE': metrics.mean_absolute_error,\n",
    "    'MSE': metrics.mean_squared_error,\n",
    "    'MedAE': metrics.median_absolute_error,\n",
    "    'R2': metrics.r2_score\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def metrics_report(*predictions):\n",
    "    records = []\n",
    "    for prediction_set in predictions:\n",
    "        record = {'name': prediction_set.name}\n",
    "        for metric_name in sorted(test_metrics.keys()):\n",
    "            metric_func = test_metrics[metric_name]\n",
    "            record[metric_name] = metric_func(test_target, prediction_set)\n",
    "        records.append(record)\n",
    "    frame = pd.DataFrame.from_records(records).set_index('name')\n",
    "    return frame\n",
    "        \n",
    "metrics_report(lr_predictions)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##These numbers don't really tell you if a model is good\n",
    "###Not by themselves, at least\n",
    "###Keep them in mind while we do more modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Now, just for a baseline\n",
    "###Let's look at the mean and median predictors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "mean_response = np.mean(train_target)\n",
    "mean_predictions = pd.Series(np.ones_like(test_target) * mean_response,\n",
    "                             name='Mean Response')\n",
    "\n",
    "median_response = np.median(train_target)\n",
    "median_predictions = pd.Series(np.ones_like(test_target) * median_response,\n",
    "                               name='Median Response')\n",
    "\n",
    "metrics_report(mean_predictions, \n",
    "               median_predictions, \n",
    "               lr_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Ordinary linear regression is boring\n",
    "### Let's try something more exotic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "estimator = linear_model.ElasticNet()\n",
    "\n",
    "parameters = {\n",
    "    'alpha': np.linspace(0.1, 2, 10, endpoint=True),\n",
    "    'l1_ratio': np.linspace(0, 1, 10, endpoint=True)\n",
    "}\n",
    "\n",
    "enet = GridSearchCV(estimator, parameters)\n",
    "enet.fit(train_this, train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "enet_predictions = pd.Series(enet.predict(test_this),\n",
    "                             name='Elastic Net')\n",
    "p_df = pd.DataFrame({'Enet Prediction': enet_predictions,\n",
    "                     'Actual': test_target.values})\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "sns.jointplot('Actual', 'Enet Prediction', data=p_df, kind=\"hex\",\n",
    "              color=sns.color_palette()[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "metrics_report(mean_predictions, median_predictions, lr_predictions, enet_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Let's do a RandomForest too. \n",
    "\n",
    "##Because why not?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "estimator = RandomForestRegressor()\n",
    "\n",
    "parameters = {'n_estimators': (5, 10, 15, 20),\n",
    "              'min_samples_split': (4, 8, 16),\n",
    "              'min_samples_leaf': (1, 2, 4),\n",
    "             }\n",
    "rfr = GridSearchCV(estimator, parameters)\n",
    "rfr.fit(train_this, train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "rfr_predictions = pd.Series(rfr.predict(test_this),\n",
    "                            name='Random Forest')\n",
    "\n",
    "p_df = pd.DataFrame({'RF Prediction': rfr_predictions,\n",
    "                     'Actual': test_target.values})\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "sns.jointplot('Actual', 'RF Prediction', data=p_df, kind=\"hex\",\n",
    "              color=sns.color_palette()[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "metrics_report(mean_predictions,\n",
    "               median_predictions,\n",
    "               lr_predictions,\n",
    "               enet_predictions,\n",
    "               rfr_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##These models seem very close\n",
    "###Are they doing anything differently?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "lr_diffs = lr_predictions - test_target\n",
    "lr_diffs.name = 'LinearRegression Error'\n",
    "rfr_diffs = rfr_predictions - test_target\n",
    "rfr_diffs.name = 'RandomForest Error'\n",
    "\n",
    "sns.jointplot(lr_diffs, rfr_diffs, kind='hex', color=sns.color_palette()[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###They are making the same kinds of errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "###Not unexpected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Feature Engineering (lite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Simple transformations\n",
    "### Quadratic combinations of existing features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "degree = 2\n",
    "\n",
    "model = make_pipeline(PolynomialFeatures(degree), Lasso())\n",
    "model.fit(train_this, train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "poly_preds = pd.Series(model.predict(test_this),\n",
    "                       name='Polynomial Lasso',\n",
    "                       index=test_target.index)\n",
    "\n",
    "\n",
    "sns.jointplot(test_target, \n",
    "              poly_preds,\n",
    "              kind='hex',\n",
    "              color=sns.color_palette()[5])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "metrics_report(mean_predictions,\n",
    "               median_predictions,\n",
    "               lr_predictions,\n",
    "               enet_predictions,\n",
    "               rfr_predictions,\n",
    "               poly_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#What next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Try more variables\n",
    "##Try different transforms\n",
    "##Ask lots of questions\n",
    "###See where you end up!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#Exercise 3\n",
    "\n",
    "##Make a model that gets a better MSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#Wrap up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##Pandas is great for manipulating data\n",
    "##Matplotlib and Seaborn make sweet visualizations\n",
    "##Scikit-Learn has great ML tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#Thanks for coming!\n",
    "##Enjoy ODSC!\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}