02a_Pandas (optional).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to the Pandas package \n",
    "This is a companion notebook for video content presented as part of the Data Mining course at SMU.\n",
    "\n",
    "In this tutorial we will be looking at a number of different parts of the Pandas package for data analysis, including:\n",
    "- Data Frames\n",
    " - loading data\n",
    " - head and tail commands\n",
    "- Munging\n",
    " - indexing operations\n",
    " - basic statistics\n",
    " - encoding\n",
    " - imputation (optional, but recommended)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Frames\n",
    "Data frames in Pandas are basically like tables of data that you can do some really interesting relational database operations upon. There are many built in methods for aggregation and visualization, but we will cover those next time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets load a typical table of data from a csv file. You can download the file from here:\n",
    "https://archive.ics.uci.edu/ml/datasets/Heart+Disease\n",
    "\n",
    "Make sure to place it in this directory or adjust the path for the file.\n",
    "### Reading Data from CSV with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# let's print out the first five rows inside a csv file\n",
    "\n",
    "# NOTE: you may need to change the path to the file, \n",
    "#       depending on where you saved the data\n",
    "with open('data/heart_disease.csv') as fid:\n",
    "    for idx, row in enumerate(fid):\n",
    "        print(row,end='')\n",
    "        if idx >= 4:\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# now let's read in the same data using pandas to save it as a dataframe\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('data/heart_disease.csv') # read in the csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site</th>\n",
       "      <th>age</th>\n",
       "      <th>is_male</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>high_blood_sugar</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>exer_angina</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cleve</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cleve</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cleve</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cleve</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cleve</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    site  age  is_male  chest_pain rest_blood_press cholesterol  \\\n",
       "0  cleve   63        1           1              145         233   \n",
       "1  cleve   67        1           4              160         286   \n",
       "2  cleve   67        1           4              120         229   \n",
       "3  cleve   37        1           3              130         250   \n",
       "4  cleve   41        0           2              130         204   \n",
       "\n",
       "  high_blood_sugar rest_ecg max_heart_rate exer_angina ST_depression  \\\n",
       "0                1        2            150           0           2.3   \n",
       "1                0        2            108           1           1.5   \n",
       "2                0        2            129           1           2.6   \n",
       "3                0        0            187           0           3.5   \n",
       "4                0        2            172           0           1.4   \n",
       "\n",
       "  Peak_ST_seg major_vessels thal  has_heart_disease  \n",
       "0           3             0    6                  0  \n",
       "1           2             3    3                  2  \n",
       "2           2             2    7                  1  \n",
       "3           3             0    3                  0  \n",
       "4           1             0    3                  0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now lets look at the data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 15 columns):\n",
      "site                 920 non-null object\n",
      "age                  920 non-null int64\n",
      "is_male              920 non-null int64\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     920 non-null object\n",
      "cholesterol          920 non-null object\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null object\n",
      "max_heart_rate       920 non-null object\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null object\n",
      "Peak_ST_seg          920 non-null object\n",
      "major_vessels        920 non-null object\n",
      "thal                 920 non-null object\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: int64(4), object(11)\n",
      "memory usage: 107.9+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# now let's a get a summary of the variables \n",
    "print(df.info())\n",
    "# we can see that most of the data \n",
    "#  is saved as an integer or as a nominal object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data has been read into working memory and is known as a DataFrame.\n",
    "\n",
    "### Reading Data from SQLite3 with Pandas\n",
    "We can also connect to a sqlite3 database using the built in sqlite3 package that ships with python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site</th>\n",
       "      <th>age</th>\n",
       "      <th>is_male</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>high_blood_sugar</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>exer_angina</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cleve</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cleve</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cleve</td>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cleve</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cleve</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    site age is_male chest_pain rest_blood_press cholesterol high_blood_sugar  \\\n",
       "0  cleve  63       1          1              145         233                1   \n",
       "1  cleve  67       1          4              160         286                0   \n",
       "2  cleve  67       1          4              120         229                0   \n",
       "3  cleve  37       1          3              130         250                0   \n",
       "4  cleve  41       0          2              130         204                0   \n",
       "\n",
       "  rest_ecg max_heart_rate exer_angina ST_depression Peak_ST_seg major_vessels  \\\n",
       "0        2            150           0           2.3           3             0   \n",
       "1        2            108           1           1.5           2             3   \n",
       "2        2            129           1           2.6           2             2   \n",
       "3        0            187           0           3.5           3             0   \n",
       "4        2            172           0           1.4           1             0   \n",
       "\n",
       "  thal has_heart_disease  \n",
       "0    6                 0  \n",
       "1    3                 2  \n",
       "2    7                 1  \n",
       "3    3                 0  \n",
       "4    3                 0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# but csv files are not the only thing we can work with\n",
    "# what if the data was actually in a sqlite database?\n",
    "del df\n",
    "import sqlite3\n",
    "\n",
    "con = sqlite3.connect('data/heart_disease_sql') # again this file is in the same directory\n",
    "df = pd.read_sql('SELECT * FROM heart_disease', con)  # the table name is heart_disease\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 15 columns):\n",
      "site                 920 non-null object\n",
      "age                  920 non-null object\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null object\n",
      "rest_blood_press     920 non-null object\n",
      "cholesterol          920 non-null object\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null object\n",
      "max_heart_rate       920 non-null object\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null object\n",
      "Peak_ST_seg          920 non-null object\n",
      "major_vessels        920 non-null object\n",
      "thal                 920 non-null object\n",
      "has_heart_disease    920 non-null object\n",
      "dtypes: object(15)\n",
      "memory usage: 107.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()\n",
    "# notice now, however, that the data types are all objects!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Working with DataFrames\n",
    " We can index in to DataFrame in a number of ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0      63\n",
      "1      67\n",
      "2      67\n",
      "3      37\n",
      "4      41\n",
      "5      56\n",
      "6      62\n",
      "7      57\n",
      "8      63\n",
      "9      53\n",
      "10     57\n",
      "11     56\n",
      "12     56\n",
      "13     44\n",
      "14     52\n",
      "15     57\n",
      "16     48\n",
      "17     54\n",
      "18     48\n",
      "19     49\n",
      "20     64\n",
      "21     58\n",
      "22     58\n",
      "23     58\n",
      "24     60\n",
      "25     50\n",
      "26     58\n",
      "27     66\n",
      "28     43\n",
      "29     40\n",
      "       ..\n",
      "890    52\n",
      "891    53\n",
      "892    53\n",
      "893    54\n",
      "894    55\n",
      "895    55\n",
      "896    55\n",
      "897    56\n",
      "898    56\n",
      "899    56\n",
      "900    58\n",
      "901    59\n",
      "902    59\n",
      "903    65\n",
      "904    66\n",
      "905    41\n",
      "906    43\n",
      "907    44\n",
      "908    47\n",
      "909    47\n",
      "910    49\n",
      "911    49\n",
      "912    50\n",
      "913    50\n",
      "914    52\n",
      "915    52\n",
      "916    54\n",
      "917    56\n",
      "918    58\n",
      "919    65\n",
      "Name: age, dtype: object\n",
      "0      63\n",
      "1      67\n",
      "2      67\n",
      "3      37\n",
      "4      41\n",
      "5      56\n",
      "6      62\n",
      "7      57\n",
      "8      63\n",
      "9      53\n",
      "10     57\n",
      "11     56\n",
      "12     56\n",
      "13     44\n",
      "14     52\n",
      "15     57\n",
      "16     48\n",
      "17     54\n",
      "18     48\n",
      "19     49\n",
      "20     64\n",
      "21     58\n",
      "22     58\n",
      "23     58\n",
      "24     60\n",
      "25     50\n",
      "26     58\n",
      "27     66\n",
      "28     43\n",
      "29     40\n",
      "       ..\n",
      "890    52\n",
      "891    53\n",
      "892    53\n",
      "893    54\n",
      "894    55\n",
      "895    55\n",
      "896    55\n",
      "897    56\n",
      "898    56\n",
      "899    56\n",
      "900    58\n",
      "901    59\n",
      "902    59\n",
      "903    65\n",
      "904    66\n",
      "905    41\n",
      "906    43\n",
      "907    44\n",
      "908    47\n",
      "909    47\n",
      "910    49\n",
      "911    49\n",
      "912    50\n",
      "913    50\n",
      "914    52\n",
      "915    52\n",
      "916    54\n",
      "917    56\n",
      "918    58\n",
      "919    65\n",
      "Name: age, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# the variable names are embedded into the structure\n",
    "print (df.age)\n",
    "print (df['age']) # but can also be accessed using strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 4 inf\n"
     ]
    }
   ],
   "source": [
    "print (df.chest_pain.min(), df.chest_pain.max(), df.chest_pain.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null object\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null object\n",
      "rest_blood_press     920 non-null object\n",
      "cholesterol          920 non-null object\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null object\n",
      "max_heart_rate       920 non-null object\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null object\n",
      "Peak_ST_seg          920 non-null object\n",
      "major_vessels        920 non-null object\n",
      "thal                 920 non-null object\n",
      "has_heart_disease    920 non-null object\n",
      "dtypes: object(14)\n",
      "memory usage: 100.7+ KB\n"
     ]
    }
   ],
   "source": [
    "# lets get rid of the 'site' variable\n",
    "if 'site' in df:\n",
    "    del df['site']\n",
    "\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null int64\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     920 non-null float64\n",
      "cholesterol          920 non-null float64\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null int64\n",
      "max_heart_rate       920 non-null float64\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null float64\n",
      "Peak_ST_seg          920 non-null int64\n",
      "major_vessels        920 non-null int64\n",
      "thal                 920 non-null int64\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: float64(4), int64(7), object(3)\n",
      "memory usage: 100.7+ KB\n"
     ]
    }
   ],
   "source": [
    "# Notice that all of the data is stored as a non-null object\n",
    "# That's not good. It means we need to change those data types\n",
    "# in order to encode the variables properly. Right now Pandas\n",
    "# thinks all of our variables are nominal!\n",
    "\n",
    "import numpy as np\n",
    "# replace '?' with -1, we will deal with missing values later\n",
    "df = df.replace(to_replace='?',value=-1) # don't set to NaN, becasue that's an error if data isn't float\n",
    "\n",
    "# let's start by first changing the numeric values to be floats\n",
    "continuous_features = ['rest_blood_press', 'cholesterol', \n",
    "                       'max_heart_rate', 'ST_depression']\n",
    "\n",
    "# and the oridnal values to be integers\n",
    "ordinal_features = ['age','major_vessels','chest_pain',\n",
    "                    'rest_ecg','Peak_ST_seg','thal','has_heart_disease']\n",
    "\n",
    "# we won't touch these variables, keep them as categorical\n",
    "categ_features = ['is_male','high_blood_sugar','exer_angina'];\n",
    "\n",
    "# use the \"astype\" function to change the variable type\n",
    "df[continuous_features] = df[continuous_features].astype(np.float64)\n",
    "df[ordinal_features] = df[ordinal_features].astype(np.int64)\n",
    "\n",
    "df.info() # now our data looks better!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>is_male</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>high_blood_sugar</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>exer_angina</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age is_male  chest_pain  rest_blood_press  cholesterol high_blood_sugar  \\\n",
       "0   63       1           1             145.0        233.0                1   \n",
       "1   67       1           4             160.0        286.0                0   \n",
       "2   67       1           4             120.0        229.0                0   \n",
       "3   37       1           3             130.0        250.0                0   \n",
       "4   41       0           2             130.0        204.0                0   \n",
       "\n",
       "   rest_ecg  max_heart_rate exer_angina  ST_depression  Peak_ST_seg  \\\n",
       "0         2           150.0           0            2.3            3   \n",
       "1         2           108.0           1            1.5            2   \n",
       "2         2           129.0           1            2.6            2   \n",
       "3         0           187.0           0            3.5            3   \n",
       "4         2           172.0           0            1.4            1   \n",
       "\n",
       "   major_vessels  thal  has_heart_disease  \n",
       "0              0     6                  0  \n",
       "1              3     3                  2  \n",
       "2              2     7                  1  \n",
       "3              0     3                  0  \n",
       "4              0     3                  0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get summary of all attributes in the frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>53.510870</td>\n",
       "      <td>3.250000</td>\n",
       "      <td>123.594565</td>\n",
       "      <td>192.604348</td>\n",
       "      <td>0.601087</td>\n",
       "      <td>129.263043</td>\n",
       "      <td>0.752174</td>\n",
       "      <td>0.840217</td>\n",
       "      <td>-0.436957</td>\n",
       "      <td>1.871739</td>\n",
       "      <td>0.995652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.424685</td>\n",
       "      <td>0.930969</td>\n",
       "      <td>37.484705</td>\n",
       "      <td>114.615011</td>\n",
       "      <td>0.808415</td>\n",
       "      <td>41.376773</td>\n",
       "      <td>1.154353</td>\n",
       "      <td>1.403211</td>\n",
       "      <td>0.959656</td>\n",
       "      <td>3.313649</td>\n",
       "      <td>1.142693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-2.600000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>47.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>115.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>54.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>221.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>138.000000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>267.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>156.000000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>603.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age  chest_pain  rest_blood_press  cholesterol    rest_ecg  \\\n",
       "count  920.000000  920.000000        920.000000   920.000000  920.000000   \n",
       "mean    53.510870    3.250000        123.594565   192.604348    0.601087   \n",
       "std      9.424685    0.930969         37.484705   114.615011    0.808415   \n",
       "min     28.000000    1.000000         -1.000000    -1.000000   -1.000000   \n",
       "25%     47.000000    3.000000        120.000000   164.000000    0.000000   \n",
       "50%     54.000000    4.000000        130.000000   221.000000    0.000000   \n",
       "75%     60.000000    4.000000        140.000000   267.000000    1.000000   \n",
       "max     77.000000    4.000000        200.000000   603.000000    2.000000   \n",
       "\n",
       "       max_heart_rate  ST_depression  Peak_ST_seg  major_vessels        thal  \\\n",
       "count      920.000000     920.000000   920.000000     920.000000  920.000000   \n",
       "mean       129.263043       0.752174     0.840217      -0.436957    1.871739   \n",
       "std         41.376773       1.154353     1.403211       0.959656    3.313649   \n",
       "min         -1.000000      -2.600000    -1.000000      -1.000000   -1.000000   \n",
       "25%        115.000000       0.000000    -1.000000      -1.000000   -1.000000   \n",
       "50%        138.000000       0.200000     1.000000      -1.000000   -1.000000   \n",
       "75%        156.000000       1.500000     2.000000       0.000000    6.000000   \n",
       "max        202.000000       6.200000     3.000000       3.000000    7.000000   \n",
       "\n",
       "       has_heart_disease  \n",
       "count         920.000000  \n",
       "mean            0.995652  \n",
       "std             1.142693  \n",
       "min             0.000000  \n",
       "25%             0.000000  \n",
       "50%             1.000000  \n",
       "75%             2.000000  \n",
       "max             4.000000  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe() # will get summary of continuous or the nominals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 920 entries in this data frame. Notice that this data frame has a number of missing values denoted by the value -1 (that we changed the '?' value to before). We need to either remove the missing values from the dataset OR we need to fill in with our best guess for those values. Let's first drop all the rows with missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null int64\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     861 non-null float64\n",
      "cholesterol          890 non-null float64\n",
      "high_blood_sugar     830 non-null object\n",
      "rest_ecg             918 non-null float64\n",
      "max_heart_rate       865 non-null float64\n",
      "exer_angina          865 non-null object\n",
      "ST_depression        856 non-null float64\n",
      "Peak_ST_seg          611 non-null float64\n",
      "major_vessels        309 non-null float64\n",
      "thal                 434 non-null float64\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: float64(8), int64(3), object(3)\n",
      "memory usage: 100.7+ KB\n",
      "None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile\n",
      "  RuntimeWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>920.000000</td>\n",
       "      <td>920.000000</td>\n",
       "      <td>861.000000</td>\n",
       "      <td>890.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>865.000000</td>\n",
       "      <td>856.000000</td>\n",
       "      <td>611.000000</td>\n",
       "      <td>309.000000</td>\n",
       "      <td>434.000000</td>\n",
       "      <td>920.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>53.510870</td>\n",
       "      <td>3.250000</td>\n",
       "      <td>132.132404</td>\n",
       "      <td>199.130337</td>\n",
       "      <td>0.604575</td>\n",
       "      <td>137.545665</td>\n",
       "      <td>0.883178</td>\n",
       "      <td>1.770867</td>\n",
       "      <td>0.676375</td>\n",
       "      <td>5.087558</td>\n",
       "      <td>0.995652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.424685</td>\n",
       "      <td>0.930969</td>\n",
       "      <td>19.066070</td>\n",
       "      <td>110.780810</td>\n",
       "      <td>0.805827</td>\n",
       "      <td>25.926276</td>\n",
       "      <td>1.088707</td>\n",
       "      <td>0.619256</td>\n",
       "      <td>0.935653</td>\n",
       "      <td>1.919075</td>\n",
       "      <td>1.142693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>-2.600000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>47.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>54.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>603.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age  chest_pain  rest_blood_press  cholesterol    rest_ecg  \\\n",
       "count  920.000000  920.000000        861.000000   890.000000  918.000000   \n",
       "mean    53.510870    3.250000        132.132404   199.130337    0.604575   \n",
       "std      9.424685    0.930969         19.066070   110.780810    0.805827   \n",
       "min     28.000000    1.000000          0.000000     0.000000    0.000000   \n",
       "25%     47.000000    3.000000               NaN          NaN         NaN   \n",
       "50%     54.000000    4.000000               NaN          NaN         NaN   \n",
       "75%     60.000000    4.000000               NaN          NaN         NaN   \n",
       "max     77.000000    4.000000        200.000000   603.000000    2.000000   \n",
       "\n",
       "       max_heart_rate  ST_depression  Peak_ST_seg  major_vessels        thal  \\\n",
       "count      865.000000     856.000000   611.000000     309.000000  434.000000   \n",
       "mean       137.545665       0.883178     1.770867       0.676375    5.087558   \n",
       "std         25.926276       1.088707     0.619256       0.935653    1.919075   \n",
       "min         60.000000      -2.600000     1.000000       0.000000    3.000000   \n",
       "25%               NaN            NaN          NaN            NaN         NaN   \n",
       "50%               NaN            NaN          NaN            NaN         NaN   \n",
       "75%               NaN            NaN          NaN            NaN         NaN   \n",
       "max        202.000000       6.200000     3.000000       3.000000    7.000000   \n",
       "\n",
       "       has_heart_disease  \n",
       "count         920.000000  \n",
       "mean            0.995652  \n",
       "std             1.142693  \n",
       "min             0.000000  \n",
       "25%             0.000000  \n",
       "50%             1.000000  \n",
       "75%             2.000000  \n",
       "max             4.000000  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many value have the -1 (which we set as the missing values) \n",
    "import numpy as np\n",
    "\n",
    "# let's set those values to NaN, so that Pandas understand they are missing\n",
    "df = df.replace(to_replace=-1,value=np.nan) # replace -1 with NaN (not a number)\n",
    "print (df.info())\n",
    "df.describe() # scroll over to see the values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow. Notice how the number of attributes went down in the description function. Looks like we need to impute values. If we drop the rows with missing data, we will be throwing away almost 80% of the data collected. No way!!\n",
    "\n",
    "### Imputation of NaN values (Optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age                   54.0\n",
       "is_male                1.0\n",
       "chest_pain             4.0\n",
       "rest_blood_press     130.0\n",
       "cholesterol          223.0\n",
       "high_blood_sugar       0.0\n",
       "rest_ecg               0.0\n",
       "max_heart_rate       140.0\n",
       "exer_angina            0.0\n",
       "ST_depression          0.5\n",
       "Peak_ST_seg            2.0\n",
       "major_vessels          0.0\n",
       "thal                   6.0\n",
       "has_heart_disease      1.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets look at some stats of the data\n",
    "df.median() # only calculates for numeric data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null int64\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     920 non-null float64\n",
      "cholesterol          920 non-null float64\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null float64\n",
      "max_heart_rate       920 non-null float64\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null float64\n",
      "Peak_ST_seg          920 non-null float64\n",
      "major_vessels        920 non-null float64\n",
      "thal                 920 non-null float64\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: float64(8), int64(3), object(3)\n",
      "memory usage: 100.7+ KB\n"
     ]
    }
   ],
   "source": [
    "# the 'fillna' function will take the given series (the output above)\n",
    "# and fill in the missing values for the columns it has\n",
    "df_imputed = df.fillna(df.median()) # note that to do this all values must be numeric\n",
    "df_imputed.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the object variables are unchanged, but all the numeric/ordinal values have been filled in with the median of the columns. Let's try something (slightly) smarter, and fill in the oridinals with the median and the continuous with the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is_male                1.000000\n",
      "high_blood_sugar       0.000000\n",
      "exer_angina            0.000000\n",
      "age                   54.000000\n",
      "major_vessels          0.000000\n",
      "chest_pain             4.000000\n",
      "rest_ecg               0.000000\n",
      "Peak_ST_seg            2.000000\n",
      "thal                   6.000000\n",
      "has_heart_disease      1.000000\n",
      "rest_blood_press     132.132404\n",
      "cholesterol          199.130337\n",
      "max_heart_rate       137.545665\n",
      "ST_depression          0.883178\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# make  one series for imputing with\n",
    "series_mean = df[continuous_features].mean()\n",
    "series_median = df[categ_features+ordinal_features].median()\n",
    "cat_series = pd.concat((series_median,series_mean))\n",
    "\n",
    "print (cat_series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null int64\n",
      "is_male              920 non-null object\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     920 non-null float64\n",
      "cholesterol          920 non-null float64\n",
      "high_blood_sugar     920 non-null object\n",
      "rest_ecg             920 non-null float64\n",
      "max_heart_rate       920 non-null float64\n",
      "exer_angina          920 non-null object\n",
      "ST_depression        920 non-null float64\n",
      "Peak_ST_seg          920 non-null float64\n",
      "major_vessels        920 non-null float64\n",
      "thal                 920 non-null float64\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: float64(8), int64(3), object(3)\n",
      "memory usage: 100.7+ KB\n"
     ]
    }
   ],
   "source": [
    "# now let's impute the numbers a bit differently\n",
    "\n",
    "df_imputed = df.fillna(value=cat_series)\n",
    "df_imputed.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_male</th>\n",
       "      <th>high_blood_sugar</th>\n",
       "      <th>exer_angina</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>920</td>\n",
       "      <td>830</td>\n",
       "      <td>865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>726</td>\n",
       "      <td>692</td>\n",
       "      <td>528</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       is_male high_blood_sugar exer_angina\n",
       "count      920              830         865\n",
       "unique       2                2           2\n",
       "top          1                0           0\n",
       "freq       726              692         528"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed[categ_features].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# what if we want to impute values after grouping? Split-Impute-Combine\n",
    "df_grouped = df.groupby(by=['is_male','high_blood_sugar'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 11 columns):\n",
      "age                  920 non-null int64\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     919 non-null float64\n",
      "cholesterol          920 non-null float64\n",
      "rest_ecg             919 non-null float64\n",
      "max_heart_rate       920 non-null float64\n",
      "ST_depression        915 non-null float64\n",
      "Peak_ST_seg          899 non-null float64\n",
      "major_vessels        835 non-null float64\n",
      "thal                 858 non-null float64\n",
      "has_heart_disease    920 non-null int64\n",
      "dtypes: float64(8), int64(3)\n",
      "memory usage: 79.1 KB\n",
      "None\n",
      "------------\n",
      "Index(['exer_angina', 'high_blood_sugar', 'is_male'], dtype='object')\n",
      "------------\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 920 entries, 0 to 919\n",
      "Data columns (total 14 columns):\n",
      "age                  920 non-null int64\n",
      "chest_pain           920 non-null int64\n",
      "rest_blood_press     919 non-null float64\n",
      "cholesterol          920 non-null float64\n",
      "rest_ecg             919 non-null float64\n",
      "max_heart_rate       920 non-null float64\n",
      "ST_depression        915 non-null float64\n",
      "Peak_ST_seg          899 non-null float64\n",
      "major_vessels        835 non-null float64\n",
      "thal                 858 non-null float64\n",
      "has_heart_disease    920 non-null int64\n",
      "exer_angina          865 non-null object\n",
      "high_blood_sugar     830 non-null object\n",
      "is_male              920 non-null object\n",
      "dtypes: float64(8), int64(3), object(3)\n",
      "memory usage: 100.7+ KB\n",
      "None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/ipykernel/__main__.py:7: FutureWarning: using '-' to provide set differences with Indexes is deprecated, use .difference()\n"
     ]
    }
   ],
   "source": [
    "# now use this grouping to fill the data set in each group, then transform back\n",
    "df_imputed = df_grouped.transform(lambda grp: grp.fillna(grp.mean()))\n",
    "print(df_imputed.info())\n",
    "print('------------')\n",
    "\n",
    "# the above process can remove columns, so let's find that and fix it\n",
    "names_removed = df.columns - df_imputed.columns # use list like a set operator\n",
    "print(names_removed)\n",
    "print('------------')\n",
    "\n",
    "df_imputed[names_removed] = df[names_removed]\n",
    "print(df_imputed.info())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing logically into Data Frames\n",
    "Let's now say that we are only interested in the summary of the dataframe when the patient has heart disease. We can achieve this using a few line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile\n",
      "  RuntimeWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>411.000000</td>\n",
       "      <td>405.000000</td>\n",
       "      <td>398.000000</td>\n",
       "      <td>398.000000</td>\n",
       "      <td>411.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>50.547445</td>\n",
       "      <td>2.761557</td>\n",
       "      <td>130.075507</td>\n",
       "      <td>227.713509</td>\n",
       "      <td>0.547445</td>\n",
       "      <td>148.235344</td>\n",
       "      <td>0.445843</td>\n",
       "      <td>1.610641</td>\n",
       "      <td>0.491867</td>\n",
       "      <td>4.486785</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.433700</td>\n",
       "      <td>0.903425</td>\n",
       "      <td>16.486825</td>\n",
       "      <td>74.217690</td>\n",
       "      <td>0.805204</td>\n",
       "      <td>23.175084</td>\n",
       "      <td>0.707479</td>\n",
       "      <td>0.474372</td>\n",
       "      <td>0.462333</td>\n",
       "      <td>1.368367</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>-1.100000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>43.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>201.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>135.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>227.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>57.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>266.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>165.000000</td>\n",
       "      <td>0.938866</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>76.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>190.000000</td>\n",
       "      <td>564.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>4.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age  chest_pain  rest_blood_press  cholesterol    rest_ecg  \\\n",
       "count  411.000000  411.000000        411.000000   411.000000  411.000000   \n",
       "mean    50.547445    2.761557        130.075507   227.713509    0.547445   \n",
       "std      9.433700    0.903425         16.486825    74.217690    0.805204   \n",
       "min     28.000000    1.000000         80.000000     0.000000    0.000000   \n",
       "25%     43.000000    2.000000        120.000000   201.000000    0.000000   \n",
       "50%     51.000000    3.000000        130.000000   227.000000    0.000000   \n",
       "75%     57.000000    4.000000        140.000000   266.500000    1.000000   \n",
       "max     76.000000    4.000000        190.000000   564.000000    2.000000   \n",
       "\n",
       "       max_heart_rate  ST_depression  Peak_ST_seg  major_vessels        thal  \\\n",
       "count      411.000000     411.000000   405.000000     398.000000  398.000000   \n",
       "mean       148.235344       0.445843     1.610641       0.491867    4.486785   \n",
       "std         23.175084       0.707479     0.474372       0.462333    1.368367   \n",
       "min         69.000000      -1.100000     1.000000       0.000000    3.000000   \n",
       "25%        135.000000       0.000000          NaN            NaN         NaN   \n",
       "50%        150.000000       0.000000          NaN            NaN         NaN   \n",
       "75%        165.000000       0.938866          NaN            NaN         NaN   \n",
       "max        202.000000       4.200000     3.000000       3.000000    7.000000   \n",
       "\n",
       "       has_heart_disease  \n",
       "count              411.0  \n",
       "mean                 0.0  \n",
       "std                  0.0  \n",
       "min                  0.0  \n",
       "25%                  0.0  \n",
       "50%                  0.0  \n",
       "75%                  0.0  \n",
       "max                  0.0  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed[df_imputed.has_heart_disease==0].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>has_heart_disease</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51</td>\n",
       "      <td>3</td>\n",
       "      <td>130.0</td>\n",
       "      <td>227.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.782991</td>\n",
       "      <td>0.674286</td>\n",
       "      <td>3.734043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>4</td>\n",
       "      <td>130.0</td>\n",
       "      <td>226.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.674286</td>\n",
       "      <td>5.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>58</td>\n",
       "      <td>4</td>\n",
       "      <td>134.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>1.40</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.674286</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>4</td>\n",
       "      <td>132.0</td>\n",
       "      <td>212.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>122.0</td>\n",
       "      <td>1.20</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.805893</td>\n",
       "      <td>5.759259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59</td>\n",
       "      <td>4</td>\n",
       "      <td>137.0</td>\n",
       "      <td>218.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>126.5</td>\n",
       "      <td>2.45</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.937500</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   age  chest_pain  rest_blood_press  cholesterol  rest_ecg  \\\n",
       "has_heart_disease                                                             \n",
       "0                   51           3             130.0        227.0       0.0   \n",
       "1                   55           4             130.0        226.0       0.0   \n",
       "2                   58           4             134.0        193.0       0.0   \n",
       "3                   60           4             132.0        212.0       1.0   \n",
       "4                   59           4             137.0        218.5       1.0   \n",
       "\n",
       "                   max_heart_rate  ST_depression  Peak_ST_seg  major_vessels  \\\n",
       "has_heart_disease                                                              \n",
       "0                           150.0           0.00     1.782991       0.674286   \n",
       "1                           130.0           1.00     2.000000       0.674286   \n",
       "2                           130.0           1.40     2.000000       0.674286   \n",
       "3                           122.0           1.20     2.000000       0.805893   \n",
       "4                           126.5           2.45     2.000000       0.937500   \n",
       "\n",
       "                       thal  \n",
       "has_heart_disease            \n",
       "0                  3.734043  \n",
       "1                  5.428571  \n",
       "2                  6.000000  \n",
       "3                  5.759259  \n",
       "4                  6.000000  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or we can use the extremely useful \"groupby\" function\n",
    "df_imputed.groupby(by='has_heart_disease').median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>has_heart_disease</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>50.547445</td>\n",
       "      <td>2.761557</td>\n",
       "      <td>130.075507</td>\n",
       "      <td>227.713509</td>\n",
       "      <td>0.547445</td>\n",
       "      <td>148.235344</td>\n",
       "      <td>0.445843</td>\n",
       "      <td>1.610641</td>\n",
       "      <td>0.491867</td>\n",
       "      <td>4.486785</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>55.903733</td>\n",
       "      <td>3.644401</td>\n",
       "      <td>134.100951</td>\n",
       "      <td>177.191037</td>\n",
       "      <td>0.650749</td>\n",
       "      <td>128.757144</td>\n",
       "      <td>1.251465</td>\n",
       "      <td>1.908871</td>\n",
       "      <td>0.863240</td>\n",
       "      <td>5.697933</td>\n",
       "      <td>1.799607</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         age  chest_pain  rest_blood_press  cholesterol  \\\n",
       "has_heart_disease                                                         \n",
       "False              50.547445    2.761557        130.075507   227.713509   \n",
       "True               55.903733    3.644401        134.100951   177.191037   \n",
       "\n",
       "                   rest_ecg  max_heart_rate  ST_depression  Peak_ST_seg  \\\n",
       "has_heart_disease                                                         \n",
       "False              0.547445      148.235344       0.445843     1.610641   \n",
       "True               0.650749      128.757144       1.251465     1.908871   \n",
       "\n",
       "                   major_vessels      thal  has_heart_disease  \n",
       "has_heart_disease                                              \n",
       "False                   0.491867  4.486785           0.000000  \n",
       "True                    0.863240  5.697933           1.799607  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed.groupby(by=df_imputed.has_heart_disease>0).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>chest_pain</th>\n",
       "      <th>rest_blood_press</th>\n",
       "      <th>cholesterol</th>\n",
       "      <th>rest_ecg</th>\n",
       "      <th>max_heart_rate</th>\n",
       "      <th>ST_depression</th>\n",
       "      <th>Peak_ST_seg</th>\n",
       "      <th>major_vessels</th>\n",
       "      <th>thal</th>\n",
       "      <th>has_heart_disease</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>major_vessels</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>53.368889</td>\n",
       "      <td>3.241111</td>\n",
       "      <td>132.198349</td>\n",
       "      <td>198.301656</td>\n",
       "      <td>0.587965</td>\n",
       "      <td>137.523459</td>\n",
       "      <td>0.867799</td>\n",
       "      <td>1.772801</td>\n",
       "      <td>0.629447</td>\n",
       "      <td>5.122661</td>\n",
       "      <td>0.966667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>59.900000</td>\n",
       "      <td>3.650000</td>\n",
       "      <td>136.900000</td>\n",
       "      <td>265.450000</td>\n",
       "      <td>1.350000</td>\n",
       "      <td>134.550000</td>\n",
       "      <td>1.865000</td>\n",
       "      <td>1.850000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.700000</td>\n",
       "      <td>2.300000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     age  chest_pain  rest_blood_press  cholesterol  rest_ecg  \\\n",
       "major_vessels                                                                   \n",
       "False          53.368889    3.241111        132.198349   198.301656  0.587965   \n",
       "True           59.900000    3.650000        136.900000   265.450000  1.350000   \n",
       "\n",
       "               max_heart_rate  ST_depression  Peak_ST_seg  major_vessels  \\\n",
       "major_vessels                                                              \n",
       "False              137.523459       0.867799     1.772801       0.629447   \n",
       "True               134.550000       1.865000     1.850000       3.000000   \n",
       "\n",
       "                   thal  has_heart_disease  \n",
       "major_vessels                               \n",
       "False          5.122661           0.966667  \n",
       "True           5.700000           2.300000  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed.groupby(by=df_imputed.major_vessels>2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One Hot Encoding of Categorical Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chest_1</th>\n",
       "      <th>chest_2</th>\n",
       "      <th>chest_3</th>\n",
       "      <th>chest_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   chest_1  chest_2  chest_3  chest_4\n",
       "0      1.0      0.0      0.0      0.0\n",
       "1      0.0      0.0      0.0      1.0\n",
       "2      0.0      0.0      0.0      1.0\n",
       "3      0.0      0.0      1.0      0.0\n",
       "4      0.0      1.0      0.0      0.0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one hot encoded variables can be created using the get_dummies variable\n",
    "tmpdf = pd.get_dummies(df_imputed['chest_pain'],prefix='chest')\n",
    "\n",
    "tmpdf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_male_0</th>\n",
       "      <th>is_male_1</th>\n",
       "      <th>high_blood_sugar_0</th>\n",
       "      <th>high_blood_sugar_1</th>\n",
       "      <th>exer_angina_0</th>\n",
       "      <th>exer_angina_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_male_0  is_male_1  high_blood_sugar_0  high_blood_sugar_1  \\\n",
       "0        0.0        1.0                 0.0                 1.0   \n",
       "1        0.0        1.0                 1.0                 0.0   \n",
       "2        0.0        1.0                 1.0                 0.0   \n",
       "3        0.0        1.0                 1.0                 0.0   \n",
       "4        1.0        0.0                 1.0                 0.0   \n",
       "\n",
       "   exer_angina_0  exer_angina_1  \n",
       "0            1.0            0.0  \n",
       "1            0.0            1.0  \n",
       "2            0.0            1.0  \n",
       "3            1.0            0.0  \n",
       "4            1.0            0.0  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#one hot encoding of ALL categorical variables\n",
    "# there is lot going on in this one line of code, so let's step through it\n",
    "\n",
    "# pd.concat([*]], axis=1) // this line of code concatenates all the data frames in the [*] list\n",
    "# [** for col in categ_features] // this steps through each feature in categ_features and \n",
    "#                                //   creates a new element in a list based on the output of **\n",
    "# pd.get_dummies(df_imputed[col],prefix=col) // this creates a one hot encoded dataframe of the variable=col (like code above)\n",
    "\n",
    "one_hot_df = pd.concat([pd.get_dummies(df_imputed[col],prefix=col) for col in categ_features], axis=1)\n",
    "\n",
    "one_hot_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [MLEnv]",
   "language": "python",
   "name": "Python [MLEnv]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}