t81_558_class_03_3_feature_encode.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "o7HCt9m9nxMV"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_03_3_feature_encode.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NajxcrLpnxMX"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "\n",
    "**Module 3: Introduction to PyTorch**\n",
    "\n",
    "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L89kB5pcnxMX"
   },
   "source": [
    "# Module 3 Material\n",
    "\n",
    "- Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=d-rU5IuFqLs&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "- Part 3.2: Introduction to PyTorch [[Video]](https://www.youtube.com/watch?v=Pf-rrhMolm0&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_2_pytorch.ipynb)\n",
    "- **Part 3.3: Encoding a Feature Vector for PyTorch Deep Learning** [[Video]](https://www.youtube.com/watch?v=7SGPm2tIT58&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_3_feature_encode.ipynb)\n",
    "- Part 3.4: Early Stopping and Network Persistence [[Video]](https://www.youtube.com/watch?v=lS0vvIWiahU&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "- Part 3.5: Sequences vs Classes in PyTorch [[Video]](https://www.youtube.com/watch?v=NOu8jMZx3LY&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_5_pytorch_class_sequence.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fbni6lScnxMY"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "L_R_hZeHnxMZ",
    "outputId": "012b646d-7585-4094-9c5b-c2b2ee47d579"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_4W71JennxMa"
   },
   "source": [
    "# Part 3.3: Encoding a Feature Vector for PyTorch Deep Learning\n",
    "\n",
    "Neural networks can accept many types of data. We will begin with tabular data, where there are well-defined rows and columns. This data is what you would typically see in Microsoft Excel. Neural networks require numeric input. This numeric form is called a feature vector. Each input neurons receive one feature (or column) from this vector. Each row of training data typically becomes one vector. This section will see how to encode the following tabular data into a feature vector. You can see an example of tabular data below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 236
    },
    "id": "brD-4ORBnxMb",
    "outputId": "3f728454-4190-472b-faf2-e106b6b36563"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>...</th>\n",
       "      <th>pop_dense</th>\n",
       "      <th>retail_dense</th>\n",
       "      <th>crime</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.885827</td>\n",
       "      <td>0.492126</td>\n",
       "      <td>0.071100</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.874016</td>\n",
       "      <td>0.342520</td>\n",
       "      <td>0.400809</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.909449</td>\n",
       "      <td>0.598425</td>\n",
       "      <td>0.117803</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.925197</td>\n",
       "      <td>0.539370</td>\n",
       "      <td>0.451973</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area   income  ...  pop_dense  retail_dense     crime  product\n",
       "0        1  vv    c  50876.0  ...   0.885827      0.492126  0.071100        b\n",
       "1        2  kd    c  60369.0  ...   0.874016      0.342520  0.400809        c\n",
       "...    ...  ..  ...      ...  ...        ...           ...       ...      ...\n",
       "1998  1999  qp    c  67949.0  ...   0.909449      0.598425  0.117803        c\n",
       "1999  2000  pe    c  61467.0  ...   0.925197      0.539370  0.451973        c\n",
       "\n",
       "[2000 rows x 14 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 7)\n",
    "pd.set_option(\"display.max_rows\", 5)\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=[\"NA\", \"?\"],\n",
    ")\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 9)\n",
    "pd.set_option(\"display.max_rows\", 5)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zu6XWRr8nxMc"
   },
   "source": [
    "You can make the following observations from the above data:\n",
    "\n",
    "- The target column is the column that you seek to predict. There are several candidates here. However, we will initially use the column \"product\". This field specifies what product someone bought.\n",
    "- There is an ID column. You should exclude his column because it contains no information useful for prediction.\n",
    "- Many of these fields are numeric and might not require further processing.\n",
    "- The income column does have some missing values.\n",
    "- There are categorical values: job, area, and product.\n",
    "\n",
    "To begin with, we will convert the job code into dummy variables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "id": "uW4Xnc_JnxMd",
    "outputId": "c4efa8ff-4a6a-490d-aba5-d262cf315478"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 33)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>job_11</th>\n",
       "      <th>job_al</th>\n",
       "      <th>job_am</th>\n",
       "      <th>job_ax</th>\n",
       "      <th>...</th>\n",
       "      <th>job_rn</th>\n",
       "      <th>job_sa</th>\n",
       "      <th>job_vv</th>\n",
       "      <th>job_zz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      job_11  job_al  job_am  job_ax  ...  job_rn  job_sa  job_vv  job_zz\n",
       "0      False   False   False   False  ...   False   False    True   False\n",
       "1      False   False   False   False  ...   False   False   False   False\n",
       "2      False   False   False   False  ...   False   False   False   False\n",
       "3       True   False   False   False  ...   False   False   False   False\n",
       "4      False   False   False   False  ...   False   False   False   False\n",
       "...      ...     ...     ...     ...  ...     ...     ...     ...     ...\n",
       "1995   False   False   False   False  ...   False   False    True   False\n",
       "1996   False   False   False   False  ...   False   False   False   False\n",
       "1997   False   False   False   False  ...   False   False   False   False\n",
       "1998   False   False   False   False  ...   False   False   False   False\n",
       "1999   False   False   False   False  ...   False   False   False   False\n",
       "\n",
       "[2000 rows x 33 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option(\"display.max_columns\", 7)\n",
    "pd.set_option(\"display.max_rows\", 5)\n",
    "\n",
    "dummies = pd.get_dummies(df[\"job\"], prefix=\"job\")\n",
    "print(dummies.shape)\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 9)\n",
    "pd.set_option(\"display.max_rows\", 10)\n",
    "\n",
    "display(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dq9Nr5YAnxMe"
   },
   "source": [
    "Because there are 33 different job codes, there are 33 dummy variables. We also specified a prefix because the job codes (such as \"ax\") are not that meaningful by themselves. Something such as \"job_ax\" also tells us the origin of this field.\n",
    "\n",
    "Next, we must merge these dummies back into the main data frame. We also drop the original \"job\" field, as the dummies now represent it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "n854eX_SnxMf",
    "outputId": "367d7468-3151-4ef6-dd3b-bf8f9640fa13"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>...</th>\n",
       "      <th>job_rn</th>\n",
       "      <th>job_sa</th>\n",
       "      <th>job_vv</th>\n",
       "      <th>job_zz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>34.766667</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>15.808333</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>40.941667</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>c</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>38.233333</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>d</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>33.358333</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>d</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>39.425000</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>16.891667</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id area   income     aspect  ...  job_rn  job_sa  job_vv  job_zz\n",
       "0        1    c  50876.0  13.100000  ...   False   False    True   False\n",
       "1        2    c  60369.0  18.625000  ...   False   False   False   False\n",
       "2        3    c  55126.0  34.766667  ...   False   False   False   False\n",
       "3        4    c  51690.0  15.808333  ...   False   False   False   False\n",
       "4        5    d  28347.0  40.941667  ...   False   False   False   False\n",
       "...    ...  ...      ...        ...  ...     ...     ...     ...     ...\n",
       "1995  1996    c  51017.0  38.233333  ...   False   False    True   False\n",
       "1996  1997    d  26576.0  33.358333  ...   False   False   False   False\n",
       "1997  1998    d  28595.0  39.425000  ...   False   False   False   False\n",
       "1998  1999    c  67949.0   5.733333  ...   False   False   False   False\n",
       "1999  2000    c  61467.0  16.891667  ...   False   False   False   False\n",
       "\n",
       "[2000 rows x 46 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option(\"display.max_columns\", 7)\n",
    "pd.set_option(\"display.max_rows\", 5)\n",
    "\n",
    "df = pd.concat([df, dummies], axis=1)\n",
    "df.drop(\"job\", axis=1, inplace=True)\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 9)\n",
    "pd.set_option(\"display.max_rows\", 10)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IUoOeB7XnxMf"
   },
   "source": [
    "We also introduce dummy variables for the area column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "_SY51VwanxMg",
    "outputId": "c7768404-a100-4859-995a-a138a39ad964"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>subscriptions</th>\n",
       "      <th>...</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>34.766667</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>15.808333</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>40.941667</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>38.233333</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>33.358333</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>39.425000</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>16.891667</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id   income     aspect  subscriptions  ...  area_a  area_b  area_c  \\\n",
       "0        1  50876.0  13.100000              1  ...   False   False    True   \n",
       "1        2  60369.0  18.625000              2  ...   False   False    True   \n",
       "2        3  55126.0  34.766667              1  ...   False   False    True   \n",
       "3        4  51690.0  15.808333              1  ...   False   False    True   \n",
       "4        5  28347.0  40.941667              3  ...   False   False   False   \n",
       "...    ...      ...        ...            ...  ...     ...     ...     ...   \n",
       "1995  1996  51017.0  38.233333              1  ...   False   False    True   \n",
       "1996  1997  26576.0  33.358333              2  ...   False   False   False   \n",
       "1997  1998  28595.0  39.425000              3  ...   False   False   False   \n",
       "1998  1999  67949.0   5.733333              0  ...   False   False    True   \n",
       "1999  2000  61467.0  16.891667              0  ...   False   False    True   \n",
       "\n",
       "      area_d  \n",
       "0      False  \n",
       "1      False  \n",
       "2      False  \n",
       "3      False  \n",
       "4       True  \n",
       "...      ...  \n",
       "1995   False  \n",
       "1996    True  \n",
       "1997    True  \n",
       "1998   False  \n",
       "1999   False  \n",
       "\n",
       "[2000 rows x 49 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option(\"display.max_columns\", 7)\n",
    "pd.set_option(\"display.max_rows\", 5)\n",
    "\n",
    "df = pd.concat([df, pd.get_dummies(df[\"area\"], prefix=\"area\")], axis=1)\n",
    "df.drop(\"area\", axis=1, inplace=True)\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 9)\n",
    "pd.set_option(\"display.max_rows\", 10)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g5A89cPynxMg"
   },
   "source": [
    "The last remaining transformation is to fill in missing income values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "OGYmZTnynxMg"
   },
   "outputs": [],
   "source": [
    "med = df[\"income\"].median()\n",
    "df[\"income\"] = df[\"income\"].fillna(med)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rkuv9fU8nxMh"
   },
   "source": [
    "There are more advanced ways of filling in missing values, but they require more analysis. The idea would be to see if another field might hint at what the income was. For example, it might be beneficial to calculate a median income for each area or job category. This technique is something to keep in mind for the class Kaggle competition.\n",
    "\n",
    "At this point, the Pandas dataframe is ready to be converted to Numpy for neural network training. We need to know a list of the columns that will make up _x_ (the predictors or inputs) and _y_ (the target).\n",
    "\n",
    "The complete list of columns is:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tdoOh-gdnxMh",
    "outputId": "a0c8a060-8469-4593-eeb8-e9d3442e60ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'product', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
     ]
    }
   ],
   "source": [
    "print(list(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q41NxWnfnxMh"
   },
   "source": [
    "This data includes both the target and predictors. We need a list with the target removed. We also remove **id** because it is not useful for prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Oq51nXWYnxMh",
    "outputId": "4d28c4c0-7db6-48ad-fc54-63f4ff126a9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
     ]
    }
   ],
   "source": [
    "x_columns = df.columns.drop(\"product\").drop(\"id\")\n",
    "print(list(x_columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wt4QCkoGnxMi"
   },
   "source": [
    "## Generate X and Y for a Classification Neural Network\n",
    "\n",
    "We can now generate _x_ and _y_. Note that this is how we generate y for a classification problem. Regression would not use dummies and would encode the numeric value of the target.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "xme1eICenxMi"
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop(\"product\").drop(\"id\")\n",
    "x = df[x_columns].values\n",
    "le = preprocessing.LabelEncoder()\n",
    "y = le.fit_transform(df[\"product\"])\n",
    "products = le.classes_\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_wsFey-UnxMi"
   },
   "source": [
    "We can display the _x_ and _y_ matrices.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jM1yieHUnxMi",
    "outputId": "0bcfdf50-a4c3-4bda-f981-0aa167eb875e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[50876.0 13.1 1 ... False True False]\n",
      " [60369.0 18.625 2 ... False True False]\n",
      " [55126.0 34.766666666666666 1 ... False True False]\n",
      " ...\n",
      " [28595.0 39.425 3 ... False False True]\n",
      " [67949.0 5.733333333333333 0 ... False True False]\n",
      " [61467.0 16.891666666666666 0 ... False True False]]\n",
      "[[False False False ... False  True False]\n",
      " [False False False ... False False False]\n",
      " [False False False ... False False False]\n",
      " ...\n",
      " [False False False ... False False False]\n",
      " [False False False ... False False False]\n",
      " [False False False ... False False False]]\n"
     ]
    }
   ],
   "source": [
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RGAkkWisnxMj"
   },
   "source": [
    "The x and y values are now ready for a neural network. Make sure that you construct the neural network for a classification problem. Specifically,\n",
    "\n",
    "- Classification neural networks have an output neuron count equal to the number of classes.\n",
    "- Classification neural networks should use **categorical_crossentropy** and a **softmax** activation function on the output layer.\n",
    "\n",
    "## Generate X and Y for a Regression Neural Network\n",
    "\n",
    "The program generates the _x_ values the say way for a regression neural network. However, _y_ does not use dummies. Make sure to replace **income** with your actual target.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "LpgOJfYQnxMj"
   },
   "outputs": [],
   "source": [
    "y = df[\"income\"].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-sjQDu-qnxMj"
   },
   "source": [
    "# Module 3 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 3](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B598GNg3nxMj"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "t81_558_class_04_1_feature_encode.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}