t81_558_class_02_2_pandas_cat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_02_2_pandas_cat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 2: Python for Machine Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 2 Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* Part 2.1: Introduction to Pandas [[Video]](https://www.youtube.com/watch?v=bN4UuCBdpZc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_1_python_pandas.ipynb)\n",
    "* **Part 2.2: Categorical Values** [[Video]](https://www.youtube.com/watch?v=4a1odDpG0Ho&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_2_pandas_cat.ipynb)\n",
    "* Part 2.3: Grouping, Sorting, and Shuffling in Python Pandas [[Video]](https://www.youtube.com/watch?v=YS4wm5gD8DM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_3_pandas_grouping.ipynb)\n",
    "* Part 2.4: Using Apply and Map in Pandas for Keras [[Video]](https://www.youtube.com/watch?v=XNCEZ4WaPBY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_4_pandas_functional.ipynb)\n",
    "* Part 2.5: Feature Engineering in Pandas for Deep Learning in Keras [[Video]](https://www.youtube.com/watch?v=BWPTj4_Mi9E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_5_pandas_features.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2.2: Categorical and Continuous Values\n",
    "\n",
    "Neural networks require their input to be a fixed number of columns.  This is very similar to spreadsheet data.  This input must be completely numeric.  \n",
    "\n",
    "It is important to represent the data in a way that the neural network can train from it.  In class 6, we will see even more ways to preprocess data.  For now, we will look at several of the most basic ways to transform data for a neural network.\n",
    "\n",
    "Before we look at specific ways to preprocess data, it is important to consider four basic types of data, as defined by [Stanley Smith Stevens](https://en.wikipedia.org/wiki/Stanley_Smith_Stevens).  These are commonly referred to as the [levels of measure](https://en.wikipedia.org/wiki/Level_of_measurement):\n",
    "\n",
    "* Character Data (strings)\n",
    "    * **Nominal** - Individual discrete items, no order. For example: color, zip code, shape.\n",
    "    * **Ordinal** - Individual discrete items that can be ordered.  For example: grade level, job title, Starbucks(tm) coffee size (tall, vente, grande) \n",
    "* Numeric Data\n",
    "    * **Interval** - Numeric values, no defined start.  For example, temperature.  You would never say \"yesterday was twice as hot as today\".\n",
    "    * **Ratio** - Numeric values, clearly defined start.  For example, speed.  You would say that \"The first car is going twice as fast as the second.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Continuous Values\n",
    "\n",
    "One common transformation is to normalize the inputs.  It is sometimes valuable to normalization numeric inputs to be put in a standard form so that two values can easily be compared.  Consider if a friend told you that he received a $10 discount.  Is this a good deal?  Maybe.  But the value is not normalized.  If your friend purchased a car, then the discount is not that good.  If your friend purchased dinner, this is a very good discount!\n",
    "\n",
    "Percentages are a very common form of normalization.  If your friend tells you they got 10% off, we know that this is a better discount than 5%.  It does not matter how much the purchase price was.  One very common machine learning normalization is the Z-Score:\n",
    "\n",
    "$z = \\frac{x - \\mu}{\\sigma} $\n",
    "\n",
    "To calculate the Z-Score you need to also calculate the mean($\\mu$) and the standard deviation ($\\sigma$).  The mean is calculated as follows:\n",
    "\n",
    "$\\mu = \\bar{x} = \\frac{x_1+x_2+\\cdots +x_n}{n}$\n",
    "\n",
    "The standard deviation is calculated as follows:\n",
    "\n",
    "$\\sigma = \\sqrt{\\frac{1}{N} \\sum_{i=1}^N (x_i - \\mu)^2}, {\\rm \\ \\ where\\ \\ } \\mu = \\frac{1}{N} \\sum_{i=1}^N x_i$\n",
    "\n",
    "The following Python code replaces the mpg with a z-score.  Cars with average MPG will be near zero, above zero is above average, and below zero is below average.  Z-Scores above/below -3/3 are very rare, these are outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.962647</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.834543</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0 -0.706439          8         307.0       130.0    3504          12.0    70   \n",
       "1 -1.090751          8         350.0       165.0    3693          11.5    70   \n",
       "2 -0.706439          8         318.0       150.0    3436          11.0    70   \n",
       "3 -0.962647          8         304.0       150.0    3433          12.0    70   \n",
       "4 -0.834543          8         302.0       140.0    3449          10.5    70   \n",
       "\n",
       "   origin                       name  \n",
       "0       1  chevrolet chevelle malibu  \n",
       "1       1          buick skylark 320  \n",
       "2       1         plymouth satellite  \n",
       "3       1              amc rebel sst  \n",
       "4       1                ford torino  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "df['mpg'] = zscore(df['mpg'])\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Values as Dummies\n",
    "The classic means of encoding categorical values is to make them dummy variables.  This is also called one-hot-encoding.  Consider the following data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>subscriptions</th>\n",
       "      <th>dist_healthy</th>\n",
       "      <th>save_rate</th>\n",
       "      <th>dist_unhealthy</th>\n",
       "      <th>age</th>\n",
       "      <th>pop_dense</th>\n",
       "      <th>retail_dense</th>\n",
       "      <th>crime</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>1</td>\n",
       "      <td>9.017895</td>\n",
       "      <td>35</td>\n",
       "      <td>11.738935</td>\n",
       "      <td>49</td>\n",
       "      <td>0.885827</td>\n",
       "      <td>0.492126</td>\n",
       "      <td>0.071100</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>2</td>\n",
       "      <td>7.766643</td>\n",
       "      <td>59</td>\n",
       "      <td>6.805396</td>\n",
       "      <td>51</td>\n",
       "      <td>0.874016</td>\n",
       "      <td>0.342520</td>\n",
       "      <td>0.400809</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>34.766667</td>\n",
       "      <td>1</td>\n",
       "      <td>3.632069</td>\n",
       "      <td>6</td>\n",
       "      <td>13.671772</td>\n",
       "      <td>44</td>\n",
       "      <td>0.944882</td>\n",
       "      <td>0.724409</td>\n",
       "      <td>0.207723</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>15.808333</td>\n",
       "      <td>1</td>\n",
       "      <td>5.372942</td>\n",
       "      <td>16</td>\n",
       "      <td>4.333286</td>\n",
       "      <td>50</td>\n",
       "      <td>0.889764</td>\n",
       "      <td>0.444882</td>\n",
       "      <td>0.361216</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>40.941667</td>\n",
       "      <td>3</td>\n",
       "      <td>3.822477</td>\n",
       "      <td>20</td>\n",
       "      <td>5.967121</td>\n",
       "      <td>38</td>\n",
       "      <td>0.744094</td>\n",
       "      <td>0.661417</td>\n",
       "      <td>0.068033</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id job area   income     aspect  subscriptions  dist_healthy  save_rate  \\\n",
       "0   1  vv    c  50876.0  13.100000              1      9.017895         35   \n",
       "1   2  kd    c  60369.0  18.625000              2      7.766643         59   \n",
       "2   3  pe    c  55126.0  34.766667              1      3.632069          6   \n",
       "3   4  11    c  51690.0  15.808333              1      5.372942         16   \n",
       "4   5  kl    d  28347.0  40.941667              3      3.822477         20   \n",
       "\n",
       "   dist_unhealthy  age  pop_dense  retail_dense     crime product  \n",
       "0       11.738935   49   0.885827      0.492126  0.071100       b  \n",
       "1        6.805396   51   0.874016      0.342520  0.400809       c  \n",
       "2       13.671772   44   0.944882      0.724409  0.207723       b  \n",
       "3        4.333286   50   0.889764      0.444882  0.361216       b  \n",
       "4        5.967121   38   0.744094      0.661417  0.068033       a  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of areas: 4\n",
      "Areas: ['c', 'd', 'a', 'b']\n"
     ]
    }
   ],
   "source": [
    "areas = list(df['area'].unique())\n",
    "print(f'Number of areas: {len(areas)}')\n",
    "print(f'Areas: {areas}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four unique values in the areas column.  To encode these to dummy variables we would use four columns, each of which would represent one of the areas.  For each row, one column would have a value of one, the rest zeros.  This is why this type of encoding is sometimes called one-hot encoding.  The following code shows how you might encode the values \"a\" through \"d\".  The value A becomes [1,0,0,0] and the value B becomes [0,1,0,0]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   area_a  area_b  area_c  area_d\n",
      "0       1       0       0       0\n",
      "1       0       1       0       0\n",
      "2       0       0       1       0\n",
      "3       0       0       0       1\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(['a','b','c','d'],prefix='area')\n",
    "print(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To encode the \"area\" column, we use the following."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is necessary to merge these dummies back into the data frame.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   area_a  area_b  area_c  area_d\n",
      "0       0       0       1       0\n",
      "1       0       0       1       0\n",
      "2       0       0       1       0\n",
      "3       0       0       1       0\n",
      "4       0       0       0       1\n",
      "5       0       0       1       0\n",
      "6       0       0       0       1\n",
      "7       1       0       0       0\n",
      "8       0       0       1       0\n",
      "9       1       0       0       0\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(df['area'],prefix='area')\n",
    "print(dummies[0:10]) # Just show the first 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df,dummies],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Displaying select columns from the dataset we can see the dummy variables added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>e2</td>\n",
       "      <td>c</td>\n",
       "      <td>70854.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>38726.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>nb</td>\n",
       "      <td>a</td>\n",
       "      <td>55162.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>al</td>\n",
       "      <td>c</td>\n",
       "      <td>67311.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>pe</td>\n",
       "      <td>a</td>\n",
       "      <td>63344.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id job area   income  area_a  area_b  area_c  area_d\n",
       "0   1  vv    c  50876.0       0       0       1       0\n",
       "1   2  kd    c  60369.0       0       0       1       0\n",
       "2   3  pe    c  55126.0       0       0       1       0\n",
       "3   4  11    c  51690.0       0       0       1       0\n",
       "4   5  kl    d  28347.0       0       0       0       1\n",
       "5   6  e2    c  70854.0       0       0       1       0\n",
       "6   7  kl    d  38726.0       0       0       0       1\n",
       "7   8  nb    a  55162.0       1       0       0       0\n",
       "8   9  al    c  67311.0       0       0       1       0\n",
       "9  10  pe    a  63344.0       1       0       0       0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(df[0:10][['id','job','area','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, you will remove the original column ('area'), because it is the goal to get the dataframe to be entirely numeric for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>e2</td>\n",
       "      <td>70854.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>kl</td>\n",
       "      <td>38726.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>nb</td>\n",
       "      <td>55162.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>al</td>\n",
       "      <td>67311.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>pe</td>\n",
       "      <td>63344.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id job   income  area_a  area_b  area_c  area_d\n",
       "0   1  vv  50876.0       0       0       1       0\n",
       "1   2  kd  60369.0       0       0       1       0\n",
       "2   3  pe  55126.0       0       0       1       0\n",
       "3   4  11  51690.0       0       0       1       0\n",
       "4   5  kl  28347.0       0       0       0       1\n",
       "5   6  e2  70854.0       0       0       1       0\n",
       "6   7  kl  38726.0       0       0       0       1\n",
       "7   8  nb  55162.0       1       0       0       0\n",
       "8   9  al  67311.0       0       0       1       0\n",
       "9  10  pe  63344.0       1       0       0       0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.drop('area', axis=1, inplace=True)\n",
    "display(df[0:10][['id','job','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target Encoding for Categoricals\n",
    "\n",
    "Target encoding can sometimes increase the predictive power of a machine learning model.  However, it also greatly increases the risk of overfitting.  Because of this risk, care must be take if you are going to use this method.  It is a popular technique for Kaggle competitions.  \n",
    "\n",
    "Generally, target encoding can only be used on a categorical feature when the output of the machine learning model is numeric (regression).\n",
    "\n",
    "The concept of target encoding is actually very simple.  For each value "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y\n",
       "0  11.505457   dog   wolf  1\n",
       "1  60.906654   dog   wolf  0\n",
       "2  13.339096   dog   wolf  1\n",
       "3  24.058962   dog   wolf  1\n",
       "4  32.713906   dog   wolf  1\n",
       "5  85.913749   cat   wolf  1\n",
       "6  66.609021   cat   wolf  0\n",
       "7  54.116221   cat   wolf  0\n",
       "8   2.901382   cat   wolf  0\n",
       "9  73.374830   cat  tiger  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a small sample dataset\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(43)\n",
    "df = pd.DataFrame({\n",
    "    'cont_9': np.random.rand(10)*100,\n",
    "    'cat_0': ['dog'] * 5 + ['cat'] * 5,\n",
    "    'cat_1': ['wolf'] * 9 + ['tiger'] * 1,\n",
    "    'y': [1, 0, 1, 1, 1, 1, 0, 0, 0, 0]\n",
    "})\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than creating dummy variables for dog and cat, we would like to change it to a number. We could just use 0 for cat, 1 for dog.  However, we can encode more information than just that.  The simple 0 or 1 would also only work for one animal.  Consider what the mean target value is for cat and dog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cat': 0.2, 'dog': 0.8}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "means0 = df.groupby('cat_0')['y'].mean().to_dict()\n",
    "means0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The danger is that we are now using the target value for training.  This will potentially overfit.  The possibility of overfitting is even greater if there are a small number of a particular category.  To prevent this from happening, we use a weighting factor.  The stronger the weight the more than categories with a small number of values will tend towards the overall average of y, which is calculated as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['y'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complete function for target encoding is given here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Source: https://maxhalford.github.io/blog/target-encoding-done-the-right-way/\n",
    "def calc_smooth_mean(df1, df2, cat_name, target, weight):\n",
    "    # Compute the global mean\n",
    "    mean = df[target].mean()\n",
    "\n",
    "    # Compute the number of values and the mean of each group\n",
    "    agg = df.groupby(cat_name)[target].agg(['count', 'mean'])\n",
    "    counts = agg['count']\n",
    "    means = agg['mean']\n",
    "\n",
    "    # Compute the \"smoothed\" means\n",
    "    smooth = (counts * means + weight * mean) / (counts + weight)\n",
    "\n",
    "    # Replace each value by the according smoothed mean\n",
    "    if df2 is None:\n",
    "        return df1[cat_name].map(smooth)\n",
    "    else:\n",
    "        return df1[cat_name].map(smooth),df2[cat_name].map(smooth.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code encodes these two categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "WEIGHT = 5\n",
    "df['cat_0_enc'] = calc_smooth_mean(df1=df, df2=None, cat_name='cat_0', target='y', weight=WEIGHT)\n",
    "df['cat_1_enc'] = calc_smooth_mean(df1=df, df2=None, cat_name='cat_1', target='y', weight=WEIGHT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "      <th>cat_0_enc</th>\n",
       "      <th>cat_1_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y  cat_0_enc  cat_1_enc\n",
       "0  11.505457   dog   wolf  1       0.65   0.535714\n",
       "1  60.906654   dog   wolf  0       0.65   0.535714\n",
       "2  13.339096   dog   wolf  1       0.65   0.535714\n",
       "3  24.058962   dog   wolf  1       0.65   0.535714\n",
       "4  32.713906   dog   wolf  1       0.65   0.535714\n",
       "5  85.913749   cat   wolf  1       0.35   0.535714\n",
       "6  66.609021   cat   wolf  0       0.35   0.535714\n",
       "7  54.116221   cat   wolf  0       0.35   0.535714\n",
       "8   2.901382   cat   wolf  0       0.35   0.535714\n",
       "9  73.374830   cat  tiger  0       0.35   0.416667"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Values as Ordinal\n",
    "\n",
    "Typically categoricals will be encoded as dummy variables.  However, there might be other techniques to convert categoricals to numeric. Any time there is an order to the categoricals, a number should be used.  Consider if you had a categorical that described the current education level of an individual.   \n",
    "\n",
    "* Kindergarten (0)\n",
    "* First Grade (1)\n",
    "* Second Grade (2)\n",
    "* Third Grade (3)\n",
    "* Fourth Grade (4)\n",
    "* Fifth Grade (5)\n",
    "* Sixth Grade (6)\n",
    "* Seventh Grade (7)\n",
    "* Eighth Grade (8)\n",
    "* High School Freshman (9)\n",
    "* High School Sophomore (10)\n",
    "* High School Junior (11)\n",
    "* High School Senior (12)\n",
    "* College Freshman (13)\n",
    "* College Sophomore (14)\n",
    "* College Junior (15)\n",
    "* College Senior (16)\n",
    "* Graduate Student (17)\n",
    "* PhD Candidate (18)\n",
    "* Doctorate (19)\n",
    "* Post Doctorate (20)\n",
    "\n",
    "The above list has 21 levels.  This would take 21 dummy variables. However, simply encoding this to dummies would lose the order information.  Perhaps the easiest approach would be to assign simply number them and assign the category a single number that is equal to the value in parenthesis above.  However, we might be able to do even better.  Graduate student is likely more than a year, so you might increase more than just one value.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}