Pytorch data loader & category embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Data-preprocessing\" data-toc-modified-id=\"Data-preprocessing-0.1\"><span class=\"toc-item-num\">0.1&nbsp;&nbsp;</span>Data preprocessing</a></span></li></ul></li><li><span><a href=\"#Get-category-embeddings\" data-toc-modified-id=\"Get-category-embeddings-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Get category embeddings</a></span><ul class=\"toc-item\"><li><span><a href=\"#Get-the-cardinality-of-the-categorical-features\" data-toc-modified-id=\"Get-the-cardinality-of-the-categorical-features-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Get the cardinality of the categorical features</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchvision import transforms, utils\n",
    "import feather"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai1/lib/python3.6/site-packages/dask/config.py:168: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  data = yaml.load(f.read()) or {}\n"
     ]
    }
   ],
   "source": [
    "from fastai.imports import *\n",
    "from fastai.torch_imports import *\n",
    "from fastai.dataset import *\n",
    "from fastai.learner import *\n",
    "from fastai.structured import *\n",
    "from fastai.column_data import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0. Data preprocessing\n",
    "1. Define dataset\n",
    "2. Initialize dataloader\n",
    "3. Categorize continuous & categorical features in the data\n",
    "4. Define and intialize embeddings for the categorical features\n",
    "5. Define the model architecture\n",
    "6. Pick loss function \n",
    "7. Train the model\n",
    "8. Room for improvements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the train & test data. We have already done lot of feature engineering & preprocessing on the data. Before feeding data to neural networks we need to\n",
    "\n",
    "1. Pick the most important features for training\n",
    "2. Categorize continuous & categorical data features\n",
    "3. Replace null or missing values with median\n",
    "4. Mark categorical features as `categorical`\n",
    "5. Scale/normalize continuous data as neural networks work better with normalized data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = 'data/elo/'\n",
    "dep = 'target'\n",
    "df_raw = feather.read_dataframe('train_df_alpha')\n",
    "df_test = feather.read_dataframe('test_df_alpha')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_raw_copy['target'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After loading our previously calculated aggregates to data frames, we exclude some unimportant columns which showed little improvement to the model's accuracy. We label features with <100 unique values as categorical columns and the rest as contiguous columns. We save the column names of categorical & contiguous columns to `cat_flds` and `cont_flds` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Convert date cols\n",
    "\n",
    "for df in [df_raw, df_test]:\n",
    "    for f in ['purchase_date_max','purchase_date_min','purchase_date_max_old',\\\n",
    "                     'purchase_date_min_old', 'observation_date_old']:\n",
    "        df[f] = df[f].astype(np.int64) * 1e-9\n",
    "\n",
    "### Remove some low importance cols\n",
    "\n",
    "\n",
    "cols_excluded = ['purchase_date_max', 'purchase_date_max_old', 'card_id', 'first_active_month',\n",
    "                 'target','outliers','card_id_size', 'card_id_size_old', \n",
    "                 'purchase_date_min', 'purchase_date_min_old','first_active_monthYear',\n",
    "                 'first_active_monthMonth',\n",
    "                 'first_active_monthWeek',\n",
    "                 'first_active_monthDay',\n",
    "                 'first_active_monthDayofweek',\n",
    "                 'first_active_monthDayofyear',\n",
    "                 'first_active_monthIs_month_end',\n",
    "                 'first_active_monthIs_month_start',\n",
    "                 'first_active_monthIs_quarter_end',\n",
    "                 'first_active_monthIs_quarter_start',\n",
    "                 'first_active_monthIs_year_end',\n",
    "                 'Black_Friday_2017_mean',\n",
    "                 'amount_month_ratio_max',\n",
    "                 'purchase_Month_mean_old',\n",
    "                 'purchase_amount_total_max',\n",
    "                 'first_active_monthIs_year_start']\n",
    "\n",
    "### Pick highly important features\n",
    "\n",
    "cols_included = ['feature_1','feature_2','feature_3','transactions_count','subsector_id_nunique','merchant_id_nunique','merchant_category_id_nunique','purchase_Month_mean','purchase_Month_min','purchase_Month_max','purchase_Week_nunique','purchase_Week_mean','purchase_Week_min','purchase_Week_max','purchase_Dayofweek_mean','purchase_Dayofweek_min','purchase_Dayofweek_max','purchase_Day_nunique','purchase_Day_mean','purchase_Day_min','purchase_Day_max','purchase_Hour_nunique','purchase_Hour_mean','purchase_Hour_min','purchase_Hour_max','purchase_amount_sum','purchase_amount_max','purchase_amount_min','purchase_amount_mean','purchase_amount_var','purchase_amount_skew','installments_sum','installments_max','installments_mean','installments_var','installments_skew','month_lag_max','month_lag_min','month_lag_mean','month_lag_var','month_lag_skew','month_diff_mean','month_diff_var','month_diff_skew','purchased_on_weekend_mean','category_1_mean','category_2_mean','category_3_mean','card_id_count','price_mean','price_max','price_min','price_var','Christmas_Day_2017_mean','Children_day_2017_mean','Black_Friday_2017_mean','Mothers_Day_2018_mean','duration_mean','duration_min','duration_max','duration_var','duration_skew','amount_month_ratio_mean','amount_month_ratio_min','amount_month_ratio_max','amount_month_ratio_var','amount_month_ratio_skew','category_2_mean_mean','category_3_mean_mean','purchase_date_diff','purchase_date_average','purchase_date_uptonow','purchase_date_uptomin','transactions_count_old','subsector_id_nunique_old','merchant_id_nunique_old','merchant_category_id_nunique_old','purchase_Month_nunique','purchase_Month_mean_old','purchase_Month_min_old','purchase_Month_max_old','purchase_Week_nunique_old','purchase_Week_mean_old','purchase_Week_min_old','purchase_Week_max_old','purchase_Dayofweek_mean_old','purchase_Day_nunique_old','purchase_Day_mean_old','purchase_Day_min_old','purchase_Hour_nunique_old','purchase_Hour_mean_old','purchase_Hour_min_old','purchase_Hour_max_old','purchase_amount_sum_old','purchase_amount_max_old','purchase_amount_min_old','purchase_amount_mean_old','purchase_amount_var_old','purchase_amount_skew_old','installments_sum_old','installments_max_old','installments_mean_old','installments_var_old','installments_skew_old','month_lag_max_old','month_lag_min_old','month_lag_mean_old','month_lag_var_old','month_lag_skew_old','month_diff_max','month_diff_min','month_diff_mean_old','month_diff_var_old','month_diff_skew_old','authorized_flag_mean','purchased_on_weekend_mean_old','category_1_mean_old','category_2_mean_old','category_3_mean_old','card_id_count_old','price_sum','price_mean_old','price_max_old','price_min_old','price_var_old','Christmas_Day_2017_mean_old','Mothers_Day_2017_mean','fathers_day_2017_mean','Children_day_2017_mean_old','Valentine_Day_2017_mean','Black_Friday_2017_mean_old','Mothers_Day_2018_mean_old','duration_mean_old','duration_min_old','duration_max_old','duration_var_old','duration_skew_old','amount_month_ratio_mean_old','amount_month_ratio_min_old','amount_month_ratio_max_old','amount_month_ratio_var_old','amount_month_ratio_skew_old','category_2_mean_mean_old','category_3_mean_mean_old','purchase_date_diff_old','purchase_date_average_old','purchase_date_uptonow_old','purchase_date_uptomin_old','quarter','observed_elapsed_time','days_feature1','days_feature2','days_feature3','days_feature1_ratio','days_feature2_ratio','days_feature3_ratio','feature_sum','feature_mean','feature_max','feature_min','feature_var','card_id_total','card_id_count_total','card_id_count_ratio','purchase_amount_total','purchase_amount_total_mean','purchase_amount_total_max','purchase_amount_total_min','purchase_amount_sum_ratio','hist_first_buy','new_first_buy','hist_last_buy','new_last_buy','month_diff_ratio','installments_total','installments_ratio','price_total','CLV','CLV_old','CLV_ratio']\n",
    "\n",
    "df_train_columns = [c for c in cols_included if c not in cols_excluded]\n",
    "\n",
    "exp_cols = ['merchant_address_id_nunique', 'merchant_rating_nunique']\n",
    "\n",
    "df_train_columns = df_train_columns + exp_cols\n",
    "\n",
    "len(df_train_columns)\n",
    "\n",
    "df_raw_copy = df_raw.copy()\n",
    "df_test_copy = df_test.copy()\n",
    "\n",
    "df_raw = df_raw[df_train_columns]\n",
    "df_test = df_test[df_train_columns]\n",
    "\n",
    "### Get validation idx\n",
    "\n",
    "n_valid = 12000\n",
    "n_trn = len(df_raw)-n_valid\n",
    "val_idx = list(range(n_trn, len(df_raw)))\n",
    "\n",
    "### Get categorical & continous fields\n",
    "\n",
    "cat_flds = [n for n in df_raw.columns.values if (df_raw[n].nunique()<100) & (n != 'outliers')]\n",
    "' '.join(cat_flds)\n",
    "\n",
    "[n for n in df_raw.drop(cat_flds,axis=1).columns if not is_numeric_dtype(df_raw[n])]\n",
    "\n",
    "for n in cat_flds: df_raw[n] = df_raw[n].astype('category').cat.as_ordered()\n",
    "\n",
    "cont_flds = [n for n in df_raw.columns if n not in cat_flds and n!= 'outliers']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some infinity values in our dataframe (came from ratios), boosting trees handle infinity very well. But it's not the case with neural nets. Let's impute them by replacing them with 0 and also replacing the missing values with median."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai1/lib/python3.6/site-packages/pandas/core/frame.py:4042: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  method=method)\n",
      "/home/chandrasekhar/anaconda3/envs/fastai1/lib/python3.6/site-packages/ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n"
     ]
    }
   ],
   "source": [
    "df_raw.replace(np.inf, 0, inplace=True)\n",
    "df_raw.replace(-np.inf, 0, inplace=True)\n",
    "\n",
    "df_test.replace(np.inf, 0, inplace=True)\n",
    "df_test.replace(-np.inf, 0, inplace=True)\n",
    "\n",
    "for n in cat_flds: df_raw[n] = df_raw[n].astype('category').cat.as_ordered()\n",
    "\n",
    "for n in cont_flds: df_raw[n] = df_raw[n].fillna(df_raw[n].median()).astype('float32')\n",
    "for n in cont_flds: df_test[n] = df_test[n].fillna(df_test[n].median()).astype('float32')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and mark all the categorical columns as categories. Pandas assigns them an integer for each category and stores the mapping of category to integer separately. For neural net embeddings we need our categories to be integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_cats(df, trn):\n",
    "    \"\"\"Changes any columns of strings in df into categorical variables using trn as\n",
    "    a template for the category codes.\n",
    "    Parameters:\n",
    "    -----------\n",
    "    df: A pandas dataframe. Any columns of strings will be changed to\n",
    "        categorical values. The category codes are determined by trn.\n",
    "    trn: A pandas dataframe. When creating a category for df, it looks up the\n",
    "        what the category's code were in trn and makes those the category codes\n",
    "        for df.\n",
    "    Examples:\n",
    "    ---------\n",
    "    >>> df = pd.DataFrame({'col1' : [1, 2, 3], 'col2' : ['a', 'b', 'a']})\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    note the type of col2 is string\n",
    "    >>> train_cats(df)\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    now the type of col2 is category {a : 1, b : 2}\n",
    "    >>> df2 = pd.DataFrame({'col1' : [1, 2, 3], 'col2' : ['b', 'a', 'a']})\n",
    "    >>> apply_cats(df2, df)\n",
    "           col1 col2\n",
    "        0     1    b\n",
    "        1     2    a\n",
    "        2     3    a\n",
    "    now the type of col is category {a : 1, b : 2}\n",
    "    \"\"\"\n",
    "    for n,c in df.items():\n",
    "        if (n in trn.columns) and (trn[n].dtype.name=='category'):\n",
    "            df[n] = c.astype('category').cat.as_ordered()\n",
    "            df[n].cat.set_categories(trn[n].cat.categories, ordered=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/private/elo/fastai/structured.py:175: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  df[n] = c.astype('category').cat.as_ordered()\n"
     ]
    }
   ],
   "source": [
    "apply_cats(df_test, df_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialise the `target` in test dataframe to be `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai1/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "df_raw['target'] = df_raw_copy['target']\n",
    "df_test['target'] = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write a function to convert our df into an entirely numeric dataframe for reason mentioned above. For each column of df which is not in skip_flds nor in ignore_flds, na values are replaced by the\n",
    "    median value of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def proc_df(df, y_fld=None, skip_flds=None, ignore_flds=None, do_scale=False, na_dict=None,\n",
    "            preproc_fn=None, max_n_cat=None, subset=None, mapper=None):\n",
    "    \"\"\" proc_df takes a data frame df and splits off the response variable, and\n",
    "    changes the df into an entirely numeric dataframe. For each column of df \n",
    "    which is not in skip_flds nor in ignore_flds, na values are replaced by the\n",
    "    median value of the column.\n",
    "    Parameters:\n",
    "    -----------\n",
    "    df: The data frame you wish to process.\n",
    "    y_fld: The name of the response variable\n",
    "    skip_flds: A list of fields that dropped from df.\n",
    "    ignore_flds: A list of fields that are ignored during processing.\n",
    "    do_scale: Standardizes each column in df. Takes Boolean Values(True,False)\n",
    "    na_dict: a dictionary of na columns to add. Na columns are also added if there\n",
    "        are any missing values.\n",
    "    preproc_fn: A function that gets applied to df.\n",
    "    max_n_cat: The maximum number of categories to break into dummy values, instead\n",
    "        of integer codes.\n",
    "    subset: Takes a random subset of size subset from df.\n",
    "    mapper: If do_scale is set as True, the mapper variable\n",
    "        calculates the values used for scaling of variables during training time (mean and standard deviation).\n",
    "    Returns:\n",
    "    --------\n",
    "    [x, y, nas, mapper(optional)]:\n",
    "        x: x is the transformed version of df. x will not have the response variable\n",
    "            and is entirely numeric.\n",
    "        y: y is the response variable\n",
    "        nas: returns a dictionary of which nas it created, and the associated median.\n",
    "        mapper: A DataFrameMapper which stores the mean and standard deviation of the corresponding continuous\n",
    "        variables which is then used for scaling of during test-time.\n",
    "    Examples:\n",
    "    ---------\n",
    "    >>> df = pd.DataFrame({'col1' : [1, 2, 3], 'col2' : ['a', 'b', 'a']})\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    note the type of col2 is string\n",
    "    >>> train_cats(df)\n",
    "    >>> df\n",
    "       col1 col2\n",
    "    0     1    a\n",
    "    1     2    b\n",
    "    2     3    a\n",
    "    now the type of col2 is category { a : 1, b : 2}\n",
    "    >>> x, y, nas = proc_df(df, 'col1')\n",
    "    >>> x\n",
    "       col2\n",
    "    0     1\n",
    "    1     2\n",
    "    2     1\n",
    "    >>> data = DataFrame(pet=[\"cat\", \"dog\", \"dog\", \"fish\", \"cat\", \"dog\", \"cat\", \"fish\"],\n",
    "                 children=[4., 6, 3, 3, 2, 3, 5, 4],\n",
    "                 salary=[90, 24, 44, 27, 32, 59, 36, 27])\n",
    "    >>> mapper = DataFrameMapper([(:pet, LabelBinarizer()),\n",
    "                          ([:children], StandardScaler())])\n",
    "    >>>round(fit_transform!(mapper, copy(data)), 2)\n",
    "    8x4 Array{Float64,2}:\n",
    "    1.0  0.0  0.0   0.21\n",
    "    0.0  1.0  0.0   1.88\n",
    "    0.0  1.0  0.0  -0.63\n",
    "    0.0  0.0  1.0  -0.63\n",
    "    1.0  0.0  0.0  -1.46\n",
    "    0.0  1.0  0.0  -0.63\n",
    "    1.0  0.0  0.0   1.04\n",
    "    0.0  0.0  1.0   0.21\n",
    "    \"\"\"\n",
    "    if not ignore_flds: ignore_flds=[]\n",
    "    if not skip_flds: skip_flds=[]\n",
    "    if subset: df = get_sample(df,subset)\n",
    "    else: df = df.copy()\n",
    "    ignored_flds = df.loc[:, ignore_flds]\n",
    "    df.drop(ignore_flds, axis=1, inplace=True)\n",
    "    if preproc_fn: preproc_fn(df)\n",
    "    if y_fld is None: y = None\n",
    "    else:\n",
    "        if not is_numeric_dtype(df[y_fld]): df[y_fld] = pd.Categorical(df[y_fld]).codes\n",
    "        y = df[y_fld].values\n",
    "        skip_flds += [y_fld]\n",
    "    df.drop(skip_flds, axis=1, inplace=True)\n",
    "\n",
    "    if na_dict is None: na_dict = {}\n",
    "    else: na_dict = na_dict.copy()\n",
    "    na_dict_initial = na_dict.copy()\n",
    "    for n,c in df.items(): na_dict = fix_missing(df, c, n, na_dict)\n",
    "    if len(na_dict_initial.keys()) > 0:\n",
    "        df.drop([a + '_na' for a in list(set(na_dict.keys()) - set(na_dict_initial.keys()))], axis=1, inplace=True)\n",
    "    if do_scale: mapper = scale_vars(df, mapper)\n",
    "    for n,c in df.items(): numericalize(df, c, n, max_n_cat)\n",
    "    df = pd.get_dummies(df, dummy_na=True)\n",
    "    df = pd.concat([ignored_flds, df], axis=1)\n",
    "    res = [df, y, na_dict]\n",
    "    if do_scale: res = res + [mapper]\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai1/lib/python3.6/site-packages/numpy/lib/nanfunctions.py:1508: RuntimeWarning: overflow encountered in multiply\n",
      "  sqr = np.multiply(arr, arr, out=arr)\n"
     ]
    }
   ],
   "source": [
    "df_raw_copy = df_raw[cat_flds+cont_flds+[dep]].copy()\n",
    "df, y, nas, mapper_t = proc_df(df_raw_copy, 'target', do_scale=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For validation set, we pick 10000 rows from the dataframe and copy their indices to `val_idx`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_idx = list(range(n_trn, len(df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(201917, 178)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the same transformation we did for training data set. We will be passing the mapping of categorical columns so that our test dataset has the same mapping as our training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.86 s, sys: 379 ms, total: 6.24 s\n",
      "Wall time: 5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_test_copy = df_test[cat_flds+cont_flds+[dep]].copy()\n",
    "df_t, _, nas, mapper = proc_df(df_test_copy,'target', do_scale=True, mapper=mapper_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 178), (123623, 178))"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape, df_t.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the Root Mean Square Error function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse(x,y): return math.sqrt(((x-y)**2).mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = feather.read_dataframe('df_torch_train')\n",
    "df_t = feather.read_dataframe('df_torch_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get category embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the cardinality of the categorical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_cardinality = {n: len(c.cat.categories)+1 for n,c in df_raw[cat_flds].items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the embeddings, we get the unique categories in each of the categorical fields. We create embeddings with half the caridanality of the categorical features with minimum being 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_sizes = [(size, max(5, size//2)) for item, size in embedding_cardinality.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the target variable range to be max & min of the existing target range. We define the range of our output value to be within 20% of our maximum value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_range=(np.min(y)*1,np.max(y)*1.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we have 178 features of which 53 are categorical and 125 are contiguous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(53, 125, 178)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cat_flds), len(cont_flds), len(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's prepare our dataset using pytorch Dataset class for feeding our model. Our dataset simply returns an array of [categorical + contiguous] feature values as x & target as y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelDataset(Dataset):\n",
    "    def __init__(self, df, cat_fields, cont_fields, y):\n",
    "        self.df = df\n",
    "        self.y = y.astype(np.float32)\n",
    "        cat_values = [c.values for n,c in df[cat_fields].items()]\n",
    "        cont_values = [c.values for n,c in df[cont_fields].items()]\n",
    "        self.cat_features = np.stack(cat_values, 1).astype(np.int64)\n",
    "        self.cont_features = np.stack(cont_values, 1).astype(np.float32)\n",
    "#         self.cats = np.stack(cat_fields,  1).astype(np.int64)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.df)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        cat_val = self.cat_features[idx]\n",
    "        cont_val = self.cont_features[idx]\n",
    "        y = self.y[idx]\n",
    "        return [cat_val, cont_val, y]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialise our training dataset with the `ModelDataset` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ds = ModelDataset(df, cat_flds, cont_flds, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the dataloader we use pytorch provided `DataLoader`, and will initialise with our training dataset and a batch size of 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl = torch.utils.data.DataLoader(train_ds, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at how each mini-batch of data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1,l2,l3 = next(iter(train_dl))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([20, 53]), torch.Size([20, 125]), torch.Size([20]))"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1.shape, l2.shape, l3.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a function which uses pytorch ModuleList to create embeddings given the embedding sizes. We will initialise the embeddings inplace with uniformly distributed data between `(-0.01,0.01)`. Kaiming He initialization also proved to work good for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embeddings(emb_sizes):\n",
    "    embeddings = nn.ModuleList([nn.Embedding(car, siz) for car,siz in emb_sizes])\n",
    "    for emb in embeddings:\n",
    "        emb.weight.data.uniform_(-0.01,0.01)\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the pytorch model which will take embeddings for categorical features and add linear layers with batch normalization & dropouts and output a single value which is our target. This is a deep neural net as it has more than one layers with non linear layers woven in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TabularModel(nn.Module):\n",
    "    def __init__(self, emb_sizes, emb_dropout, lin_layers, lin_layers_dropout, n_cat_fields, n_cont_fields, y_range):\n",
    "        super().__init__()\n",
    "        # get embeddings\n",
    "        self.embeddings = get_embeddings(emb_sizes)\n",
    "        # embedding dropout\n",
    "        self.emb_dropout = nn.Dropout(emb_dropout)\n",
    "        # calculate linear layer sizes accounting embeddings\n",
    "        emb_vectors_sum = sum([e.embedding_dim for e in self.embeddings])\n",
    "        \n",
    "        # Linear layer sizes are sum of embeddings size + contiguous fields' size + linear layers we wish to have\n",
    "        linear_szs = [emb_vectors_sum + n_cont_fields] + lin_layers\n",
    "        \n",
    "        self.n_cont_fields = n_cont_fields\n",
    "        # initialize linear layers\n",
    "        self.lin_layers = nn.ModuleList([nn.Linear(linear_szs[i], linear_szs[i+1]) \n",
    "                                         for i in range(len(linear_szs)-1)])\n",
    "        # Define output layer\n",
    "        self.output_layer = nn.Linear(linear_szs[-1], 1)\n",
    "        \n",
    "        # Initialize batch normalisation for linear layers\n",
    "        self.batch_norms_lin = nn.ModuleList([nn.BatchNorm1d(s) for s in linear_szs[1:]])\n",
    "        # Initialize batch normalisation for continous fields\n",
    "        self.batch_norm_cont = nn.BatchNorm1d(n_cont_fields)\n",
    "        \n",
    "        # dropout for linear layers\n",
    "        self.linear_drops = nn.ModuleList([nn.Dropout(p) for p in lin_layers_dropout])\n",
    "        \n",
    "        self.y_range = y_range\n",
    "        \n",
    "    def forward(self, cat_fields, cont_fields):\n",
    "        # Initialize embeddings for respective categorical fields\n",
    "        x1 = [e(cat_fields[:,i]) for i,e in enumerate(self.embeddings)]\n",
    "        # concatenate all the embeddings on axis 1\n",
    "        x1 = torch.cat(x1, 1)\n",
    "        # apply dropout on embeddings\n",
    "        x1 = self.emb_dropout(x1)\n",
    "        \n",
    "        # apply batch normalization on continous fields\n",
    "        x2 = self.batch_norm_cont(cont_fields)\n",
    "        \n",
    "        # concatenate along axis 1\n",
    "        x1 = torch.cat([x1,x2], 1)\n",
    "        \n",
    "        # apply linear layers and respective batch norms followed by dropouts \n",
    "        for lin, drop, bn in zip(self.lin_layers, self.linear_drops, self.batch_norms_lin):\n",
    "            # Non linear activation function relu will give only the non-negative values, negatives zeroed.\n",
    "            x1 = F.relu(lin(x1))\n",
    "            x1 = bn(x1)\n",
    "            x1 = drop(x1)\n",
    "        x1 = self.output_layer(x1)\n",
    "        # pass the final layer through sigmoid which gives a value between 0 & 1\n",
    "        x1 = torch.sigmoid(x1)\n",
    "        y_min = self.y_range[0]\n",
    "        y_max = self.y_range[1]\n",
    "        # Mulitply/scale the output from sigmoid with the range of target to get our required y value.\n",
    "        x1 = x1*(y_max-y_min)\n",
    "        x1 = y_min + x1\n",
    "        \n",
    "        return x1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize our pytorch model with 2 linear layers of sizes 1000, 500 with 1e-3 & 1e-2 dropouts. We will apply a dropout of 0.1 on our embeddings to ensure they're optimally regularized to prevent over fitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = TabularModel(emb_sizes, 0.1, [1000,500], [0.001,0.01], len(cat_flds), len(cont_flds), y_range)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look of the model parameters here. We can see the embedding layers followed by dropout, linear layers, batch normalization layers and finally dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Module.parameters of TabularModel(\n",
       "  (embeddings): ModuleList(\n",
       "    (0): Embedding(6, 5)\n",
       "    (1): Embedding(4, 5)\n",
       "    (2): Embedding(3, 5)\n",
       "    (3): Embedding(88, 44)\n",
       "    (4): Embedding(25, 12)\n",
       "    (5): Embedding(87, 43)\n",
       "    (6): Embedding(40, 20)\n",
       "    (7): Embedding(13, 6)\n",
       "    (8): Embedding(13, 6)\n",
       "    (9): Embedding(11, 5)\n",
       "    (10): Embedding(53, 26)\n",
       "    (11): Embedding(53, 26)\n",
       "    (12): Embedding(8, 5)\n",
       "    (13): Embedding(8, 5)\n",
       "    (14): Embedding(31, 15)\n",
       "    (15): Embedding(32, 16)\n",
       "    (16): Embedding(32, 16)\n",
       "    (17): Embedding(24, 12)\n",
       "    (18): Embedding(25, 12)\n",
       "    (19): Embedding(25, 12)\n",
       "    (20): Embedding(14, 7)\n",
       "    (21): Embedding(3, 5)\n",
       "    (22): Embedding(3, 5)\n",
       "    (23): Embedding(88, 44)\n",
       "    (24): Embedding(63, 31)\n",
       "    (25): Embedding(64, 32)\n",
       "    (26): Embedding(64, 32)\n",
       "    (27): Embedding(35, 17)\n",
       "    (28): Embedding(93, 46)\n",
       "    (29): Embedding(12, 6)\n",
       "    (30): Embedding(12, 6)\n",
       "    (31): Embedding(12, 6)\n",
       "    (32): Embedding(53, 26)\n",
       "    (33): Embedding(49, 24)\n",
       "    (34): Embedding(49, 24)\n",
       "    (35): Embedding(32, 16)\n",
       "    (36): Embedding(31, 15)\n",
       "    (37): Embedding(25, 12)\n",
       "    (38): Embedding(24, 12)\n",
       "    (39): Embedding(23, 11)\n",
       "    (40): Embedding(14, 7)\n",
       "    (41): Embedding(13, 6)\n",
       "    (42): Embedding(14, 7)\n",
       "    (43): Embedding(15, 7)\n",
       "    (44): Embedding(15, 7)\n",
       "    (45): Embedding(5, 5)\n",
       "    (46): Embedding(8, 5)\n",
       "    (47): Embedding(8, 5)\n",
       "    (48): Embedding(6, 5)\n",
       "    (49): Embedding(3, 5)\n",
       "    (50): Embedding(10, 5)\n",
       "    (51): Embedding(87, 43)\n",
       "    (52): Embedding(15, 7)\n",
       "  )\n",
       "  (emb_dropout): Dropout(p=0.1)\n",
       "  (lin_layers): ModuleList(\n",
       "    (0): Linear(in_features=912, out_features=1000, bias=True)\n",
       "    (1): Linear(in_features=1000, out_features=500, bias=True)\n",
       "  )\n",
       "  (output_layer): Linear(in_features=500, out_features=1, bias=True)\n",
       "  (batch_norms_lin): ModuleList(\n",
       "    (0): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (1): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  )\n",
       "  (batch_norm_cont): BatchNorm1d(125, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (linear_drops): ModuleList(\n",
       "    (0): Dropout(p=0.001)\n",
       "    (1): Dropout(p=0.01)\n",
       "  )\n",
       ")>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Stochastic Gradient Descent as our optimiser function. let's go ahead and intialize the optimizer function with model parameters and learning rate of 1e-4. I found that 1e-3 works better in converging faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(), 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will pick Mean Square Error as our loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train our models for 2 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  2000] loss: 15.206\n",
      "[1,  4000] loss: 14.758\n",
      "[1,  6000] loss: 14.831\n",
      "[1,  8000] loss: 14.637\n",
      "[1, 10000] loss: 15.437\n",
      "[2,  2000] loss: 14.602\n",
      "[2,  4000] loss: 14.734\n",
      "[2,  6000] loss: 14.821\n",
      "[2,  8000] loss: 14.631\n",
      "[2, 10000] loss: 15.434\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(2):  # loop over the dataset multiple times\n",
    "\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(train_dl, 0):\n",
    "        # get the inputs; data is a list of [inputs, labels]\n",
    "        in1, in2, labels = data\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward + backward + optimize\n",
    "        outputs = model(in1, in2)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # print statistics\n",
    "        running_loss += loss.item()\n",
    "        if i % 2000 == 1999:    # print every 2000 mini-batches\n",
    "            print('[%d, %5d] loss: %.3f' %\n",
    "                  (epoch + 1, i + 1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the accuracy on our training data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl_all = torch.utils.data.DataLoader(train_ds, 201917)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_al1, x_al2, y_all = next(iter(train_dl_all))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = model(x_al1, x_al2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([201917, 1]), (201917,))"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917,), (201917,))"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.detach().numpy()[:, 0].shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.8548685360753963"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rmse(preds.detach().numpy()[:, 0], y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.85 is our training rmse. There is lot of scope for improvement here. We can use one-cycle policy to determine to train faster on larger epochs to converge better. We can finetune the hyperparameters like batchsize, linear layers size, embeddings size, dropout etc. I was able to get an rmse of 3.61 on the test dataset for the finetuned model on kaggle competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}