Elo feature engineering alpha.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Load-dependencies\" data-toc-modified-id=\"Load-dependencies-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Load dependencies</a></span></li><li><span><a href=\"#Read-csv-files-to-DFs\" data-toc-modified-id=\"Read-csv-files-to-DFs-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Read csv files to DFs</a></span></li><li><span><a href=\"#Fill-missing-values\" data-toc-modified-id=\"Fill-missing-values-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Fill missing values</a></span></li><li><span><a href=\"#Add-date-part\" data-toc-modified-id=\"Add-date-part-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Add date part</a></span></li><li><span><a href=\"#Add-extra-columns-(purchased-on-weekend,-monthend,-month_diff-etc.\" data-toc-modified-id=\"Add-extra-columns-(purchased-on-weekend,-monthend,-month_diff-etc.-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Add extra columns (purchased on weekend, monthend, month_diff etc.</a></span></li><li><span><a href=\"#De-normalizing-purchase-amount-column\" data-toc-modified-id=\"De-normalizing-purchase-amount-column-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>De normalizing purchase amount column</a></span></li><li><span><a href=\"#Rolling-aggregate-on-month\" data-toc-modified-id=\"Rolling-aggregate-on-month-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Rolling aggregate on month</a></span></li><li><span><a href=\"#Time-between-successive-transactions\" data-toc-modified-id=\"Time-between-successive-transactions-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Time between successive transactions</a></span></li><li><span><a href=\"#Additional-features\" data-toc-modified-id=\"Additional-features-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>Additional features</a></span></li><li><span><a href=\"#Latest-&amp;-First-5-col-aggregates\" data-toc-modified-id=\"Latest-&amp;-First-5-col-aggregates-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>Latest &amp; First 5 col aggregates</a></span></li><li><span><a href=\"#Merge-merchants-df\" data-toc-modified-id=\"Merge-merchants-df-11\"><span class=\"toc-item-num\">11&nbsp;&nbsp;</span>Merge merchants df</a></span></li><li><span><a href=\"#Aggregate-by-card_id\" data-toc-modified-id=\"Aggregate-by-card_id-12\"><span class=\"toc-item-num\">12&nbsp;&nbsp;</span>Aggregate by card_id</a></span></li><li><span><a href=\"#Add-exta-interpreted-columns-on-aggregates\" data-toc-modified-id=\"Add-exta-interpreted-columns-on-aggregates-13\"><span class=\"toc-item-num\">13&nbsp;&nbsp;</span>Add exta interpreted columns on aggregates</a></span></li><li><span><a href=\"#Aggregate-on-categories\" data-toc-modified-id=\"Aggregate-on-categories-14\"><span class=\"toc-item-num\">14&nbsp;&nbsp;</span>Aggregate on categories</a></span></li><li><span><a href=\"#Aggregate-on-month\" data-toc-modified-id=\"Aggregate-on-month-15\"><span class=\"toc-item-num\">15&nbsp;&nbsp;</span>Aggregate on month</a></span></li><li><span><a href=\"#Load-test-&amp;-train-DFs\" data-toc-modified-id=\"Load-test-&amp;-train-DFs-16\"><span class=\"toc-item-num\">16&nbsp;&nbsp;</span>Load test &amp; train DFs</a></span></li><li><span><a href=\"#Add-observed-date\" data-toc-modified-id=\"Add-observed-date-17\"><span class=\"toc-item-num\">17&nbsp;&nbsp;</span>Add observed date</a></span></li><li><span><a href=\"#Merge-train-&amp;-test-with-new-&amp;-old-transactions-history\" data-toc-modified-id=\"Merge-train-&amp;-test-with-new-&amp;-old-transactions-history-18\"><span class=\"toc-item-num\">18&nbsp;&nbsp;</span>Merge train &amp; test with new &amp; old transactions history</a></span></li><li><span><a href=\"#Adding-train-days\" data-toc-modified-id=\"Adding-train-days-19\"><span class=\"toc-item-num\">19&nbsp;&nbsp;</span>Adding train days</a></span></li><li><span><a href=\"#Add-extra-columns-like-age,-total-transactions,-installments,-purchase-amount,-first-buy-etc\" data-toc-modified-id=\"Add-extra-columns-like-age,-total-transactions,-installments,-purchase-amount,-first-buy-etc-20\"><span class=\"toc-item-num\">20&nbsp;&nbsp;</span>Add extra columns like age, total transactions, installments, purchase amount, first buy etc</a></span></li><li><span><a href=\"#Redo-some-date-features-with-observed-time\" data-toc-modified-id=\"Redo-some-date-features-with-observed-time-21\"><span class=\"toc-item-num\">21&nbsp;&nbsp;</span>Redo some date features with observed time</a></span></li><li><span><a href=\"#Mark-the-outliers\" data-toc-modified-id=\"Mark-the-outliers-22\"><span class=\"toc-item-num\">22&nbsp;&nbsp;</span>Mark the outliers</a></span></li><li><span><a href=\"#Target-encode-the-outliers\" data-toc-modified-id=\"Target-encode-the-outliers-23\"><span class=\"toc-item-num\">23&nbsp;&nbsp;</span>Target encode the outliers</a></span></li><li><span><a href=\"#Scrap-pad\" data-toc-modified-id=\"Scrap-pad-24\"><span class=\"toc-item-num\">24&nbsp;&nbsp;</span>Scrap pad</a></span></li><li><span><a href=\"#Transform-purchase_value-to-actual-amount\" data-toc-modified-id=\"Transform-purchase_value-to-actual-amount-25\"><span class=\"toc-item-num\">25&nbsp;&nbsp;</span>Transform purchase_value to actual amount</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Load dependencies\n",
    "2. Read csv files to dataframes\n",
    "3. Fill missing values\n",
    "4. Add date part \n",
    "5. Add extra columns (purchased on weekend, monthend, month_diff etc.\n",
    "6. Aggregate by card_id\n",
    "7. Aggregate by categories\n",
    "8. Mark categorical columns\n",
    "9. Add exta interpreted columns on aggregates\n",
    "10. Load test & train csvs to dfs\n",
    "11. Add date part to test & train dfs\n",
    "12. Merge train & test with new & old transactions history\n",
    "13. Add extra columns like age, total transactions, installments, purchase amount, first buy etc\n",
    "14. Mark the outliers\n",
    "15. Target encode the outliers \n",
    "16. Save to feather"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "IPython.notebook.set_autosave_interval(120000)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autosaving every 120 seconds\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 4\n",
    "%autosave 120"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.io import *\n",
    "from fastai.structured import *\n",
    "from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier\n",
    "from pandas_summary import DataFrameSummary\n",
    "from IPython.display import display\n",
    "from sklearn import metrics\n",
    "import feather\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read csv files to DFs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = '../data/elo/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = ['historical_transactions', 'new_merchant_transactions']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans, new_hist_trans = [pd.read_csv(f'{PATH}{c}.csv') for c in files]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the transactions data available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_4e6213e9bc</td>\n",
       "      <td>C_ID_4e6213e9bc</td>\n",
       "      <td>C_ID_4e6213e9bc</td>\n",
       "      <td>C_ID_4e6213e9bc</td>\n",
       "      <td>C_ID_4e6213e9bc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>88</td>\n",
       "      <td>88</td>\n",
       "      <td>88</td>\n",
       "      <td>88</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>A</td>\n",
       "      <td>A</td>\n",
       "      <td>A</td>\n",
       "      <td>A</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>80</td>\n",
       "      <td>367</td>\n",
       "      <td>80</td>\n",
       "      <td>560</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_e020e9b302</td>\n",
       "      <td>M_ID_86ec983688</td>\n",
       "      <td>M_ID_979ed661fc</td>\n",
       "      <td>M_ID_e6d5ae8ea6</td>\n",
       "      <td>M_ID_e020e9b302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>-8</td>\n",
       "      <td>-7</td>\n",
       "      <td>-6</td>\n",
       "      <td>-5</td>\n",
       "      <td>-11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>-0.703331</td>\n",
       "      <td>-0.733128</td>\n",
       "      <td>-0.720386</td>\n",
       "      <td>-0.735352</td>\n",
       "      <td>-0.722865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2017-06-25 15:33:07</td>\n",
       "      <td>2017-07-15 12:10:45</td>\n",
       "      <td>2017-08-09 22:04:29</td>\n",
       "      <td>2017-09-02 10:06:26</td>\n",
       "      <td>2017-03-10 01:14:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>37</td>\n",
       "      <td>16</td>\n",
       "      <td>37</td>\n",
       "      <td>34</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        0                    1  \\\n",
       "authorized_flag                         Y                    Y   \n",
       "card_id                   C_ID_4e6213e9bc      C_ID_4e6213e9bc   \n",
       "city_id                                88                   88   \n",
       "category_1                              N                    N   \n",
       "installments                            0                    0   \n",
       "category_3                              A                    A   \n",
       "merchant_category_id                   80                  367   \n",
       "merchant_id               M_ID_e020e9b302      M_ID_86ec983688   \n",
       "month_lag                              -8                   -7   \n",
       "purchase_amount                 -0.703331            -0.733128   \n",
       "purchase_date         2017-06-25 15:33:07  2017-07-15 12:10:45   \n",
       "category_2                              1                    1   \n",
       "state_id                               16                   16   \n",
       "subsector_id                           37                   16   \n",
       "\n",
       "                                        2                    3  \\\n",
       "authorized_flag                         Y                    Y   \n",
       "card_id                   C_ID_4e6213e9bc      C_ID_4e6213e9bc   \n",
       "city_id                                88                   88   \n",
       "category_1                              N                    N   \n",
       "installments                            0                    0   \n",
       "category_3                              A                    A   \n",
       "merchant_category_id                   80                  560   \n",
       "merchant_id               M_ID_979ed661fc      M_ID_e6d5ae8ea6   \n",
       "month_lag                              -6                   -5   \n",
       "purchase_amount                 -0.720386            -0.735352   \n",
       "purchase_date         2017-08-09 22:04:29  2017-09-02 10:06:26   \n",
       "category_2                              1                    1   \n",
       "state_id                               16                   16   \n",
       "subsector_id                           37                   34   \n",
       "\n",
       "                                        4  \n",
       "authorized_flag                         Y  \n",
       "card_id                   C_ID_4e6213e9bc  \n",
       "city_id                                88  \n",
       "category_1                              N  \n",
       "installments                            0  \n",
       "category_3                              A  \n",
       "merchant_category_id                   80  \n",
       "merchant_id               M_ID_e020e9b302  \n",
       "month_lag                             -11  \n",
       "purchase_amount                 -0.722865  \n",
       "purchase_date         2017-03-10 01:14:19  \n",
       "category_2                              1  \n",
       "state_id                               16  \n",
       "subsector_id                           37  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_ef55cf8d4b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>107</td>\n",
       "      <td>140</td>\n",
       "      <td>330</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>B</td>\n",
       "      <td>B</td>\n",
       "      <td>B</td>\n",
       "      <td>B</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>307</td>\n",
       "      <td>307</td>\n",
       "      <td>507</td>\n",
       "      <td>661</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_b0c793002c</td>\n",
       "      <td>M_ID_88920c89e8</td>\n",
       "      <td>M_ID_ad5237ef6b</td>\n",
       "      <td>M_ID_9e84cda3b1</td>\n",
       "      <td>M_ID_3c86fa3831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>-0.557574</td>\n",
       "      <td>-0.56958</td>\n",
       "      <td>-0.551037</td>\n",
       "      <td>-0.671925</td>\n",
       "      <td>-0.659904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2018-03-11 14:57:36</td>\n",
       "      <td>2018-03-19 18:53:37</td>\n",
       "      <td>2018-04-26 14:08:44</td>\n",
       "      <td>2018-03-07 09:43:21</td>\n",
       "      <td>2018-03-22 21:07:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        0                    1  \\\n",
       "authorized_flag                         Y                    Y   \n",
       "card_id                   C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                               107                  140   \n",
       "category_1                              N                    N   \n",
       "installments                            1                    1   \n",
       "category_3                              B                    B   \n",
       "merchant_category_id                  307                  307   \n",
       "merchant_id               M_ID_b0c793002c      M_ID_88920c89e8   \n",
       "month_lag                               1                    1   \n",
       "purchase_amount                 -0.557574             -0.56958   \n",
       "purchase_date         2018-03-11 14:57:36  2018-03-19 18:53:37   \n",
       "category_2                              1                    1   \n",
       "state_id                                9                    9   \n",
       "subsector_id                           19                   19   \n",
       "\n",
       "                                        2                    3  \\\n",
       "authorized_flag                         Y                    Y   \n",
       "card_id                   C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                               330                   -1   \n",
       "category_1                              N                    Y   \n",
       "installments                            1                    1   \n",
       "category_3                              B                    B   \n",
       "merchant_category_id                  507                  661   \n",
       "merchant_id               M_ID_ad5237ef6b      M_ID_9e84cda3b1   \n",
       "month_lag                               2                    1   \n",
       "purchase_amount                 -0.551037            -0.671925   \n",
       "purchase_date         2018-04-26 14:08:44  2018-03-07 09:43:21   \n",
       "category_2                              1                  NaN   \n",
       "state_id                                9                   -1   \n",
       "subsector_id                           14                    8   \n",
       "\n",
       "                                        4  \n",
       "authorized_flag                         Y  \n",
       "card_id                   C_ID_ef55cf8d4b  \n",
       "city_id                                -1  \n",
       "category_1                              Y  \n",
       "installments                            1  \n",
       "category_3                              B  \n",
       "merchant_category_id                  166  \n",
       "merchant_id               M_ID_3c86fa3831  \n",
       "month_lag                               1  \n",
       "purchase_amount                 -0.659904  \n",
       "purchase_date         2018-03-22 21:07:53  \n",
       "category_2                            NaN  \n",
       "state_id                               -1  \n",
       "subsector_id                           29  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see data dictionary to find out what each of these fields mean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| train.csv          |                                                                                    |\n",
    "|--------------------|------------------------------------------------------------------------------------|\n",
    "| card_id            | Unique card identifier                                                             |\n",
    "| first_active_month | YYYY-MM', month of first purchase                                                  |\n",
    "| feature_1          | Anonymized card categorical feature                                                |\n",
    "| feature_2          | Anonymized card categorical feature                                                |\n",
    "| feature_3          | Anonymized card categorical feature                                                |\n",
    "| target             | Loyalty numerical score calculated 2 months after historical and evaluation period |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| historical_transactions.csv |                                                  |\n",
    "|-----------------------------|--------------------------------------------------|\n",
    "| card_id                     | Card identifier                                  |\n",
    "| month_lag                   | month lag to reference date                      |\n",
    "| purchase_date               | Purchase date                                    |\n",
    "| authorized_flag             | Y' if approved, 'N' if denied                    |\n",
    "| category_3                  | anonymized category                              |\n",
    "| installments                | number of installments of purchase               |\n",
    "| category_1                  | anonymized category                              |\n",
    "| merchant_category_id        | Merchant category identifier (anonymized )       |\n",
    "| subsector_id                | Merchant category group identifier (anonymized ) |\n",
    "| merchant_id                 | Merchant identifier (anonymized)                 |\n",
    "| purchase_amount             | Normalized purchase amount                       |\n",
    "| city_id                     | City identifier (anonymized )                    |\n",
    "| state_id                    | State identifier (anonymized )                   |\n",
    "| category_2                  | anonymized category                              |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| new_merchant_period.csv |                                                  |\n",
    "|-------------------------|--------------------------------------------------|\n",
    "| card_id                 | Card identifier                                  |\n",
    "| month_lag               | month lag to reference date                      |\n",
    "| purchase_date           | Purchase date                                    |\n",
    "| authorized_flag         | Y' if approved, 'N' if denied                    |\n",
    "| category_3              | anonymized category                              |\n",
    "| installments            | number of installments of purchase               |\n",
    "| category_1              | anonymized category                              |\n",
    "| merchant_category_id    | Merchant category identifier (anonymized )       |\n",
    "| subsector_id            | Merchant category group identifier (anonymized ) |\n",
    "| merchant_id             | Merchant identifier (anonymized)                 |\n",
    "| purchase_amount         | Normalized purchase amount                       |\n",
    "| city_id                 | City identifier (anonymized )                    |\n",
    "| state_id                | State identifier (anonymized )                   |\n",
    "| category_2              | anonymized category                              |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| merchants.csv               |                                                                                                |\n",
    "|-----------------------------|------------------------------------------------------------------------------------------------|\n",
    "| merchant_id                 | Unique merchant identifier                                                                     |\n",
    "| merchant_group_id           | Merchant group (anonymized )                                                                   |\n",
    "| merchant_category_id        | Unique identifier for merchant category (anonymized )                                          |\n",
    "| subsector_id                | Merchant category group (anonymized )                                                          |\n",
    "| numerical_1                 | anonymized measure                                                                             |\n",
    "| numerical_2                 | anonymized measure                                                                             |\n",
    "| category_1                  | anonymized category                                                                            |\n",
    "| most_recent_sales_range     | Range of revenue (monetary units) in last active month --> A > B > C > D > E                   |\n",
    "| most_recent_purchases_range | Range of quantity of transactions in last active month --> A > B > C > D > E                   |\n",
    "| avg_sales_lag3              | Monthly average of revenue in last 3 months divided by revenue in last active month            |\n",
    "| avg_purchases_lag3          | Monthly average of transactions in last 3 months divided by transactions in last active month  |\n",
    "| active_months_lag3          | Quantity of active months within last 3 months                                                 |\n",
    "| avg_sales_lag6              | Monthly average of revenue in last 6 months divided by revenue in last active month            |\n",
    "| avg_purchases_lag6          | Monthly average of transactions in last 6 months divided by transactions in last active month  |\n",
    "| active_months_lag6          | Quantity of active months within last 6 months                                                 |\n",
    "| avg_sales_lag12             | Monthly average of revenue in last 12 months divided by revenue in last active month           |\n",
    "| avg_purchases_lag12         | Monthly average of transactions in last 12 months divided by transactions in last active month |\n",
    "| active_months_lag12         | Quantity of active months within last 12 months                                                |\n",
    "| category_4                  | anonymized category                                                                            |\n",
    "| city_id                     | City identifier (anonymized )                                                                  |\n",
    "| state_id                    | State identifier (anonymized )                                                                 |\n",
    "| category_2                  | anonymized category                                                                            |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with transactions data. We will come back to merchants & cards data later. We have ~1.9m new & ~2.9m historical transactions data. This can be joined with cards & merchants data on `card_id` and `merchant_id` respectively. But should we? We will find out later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1963031, 14), (29112361, 14))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.shape, hist_trans.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the missing values percent for each of the columns in the transactions data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "authorized_flag         0.000000\n",
       "card_id                 0.000000\n",
       "city_id                 0.000000\n",
       "category_1              0.000000\n",
       "installments            0.000000\n",
       "category_3              0.090757\n",
       "merchant_category_id    0.000000\n",
       "merchant_id             0.070544\n",
       "month_lag               0.000000\n",
       "purchase_amount         0.000000\n",
       "purchase_date           0.000000\n",
       "category_2              1.351412\n",
       "state_id                0.000000\n",
       "subsector_id            0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans.isnull().sum()/len(new_hist_trans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "authorized_flag         0.000000\n",
       "card_id                 0.000000\n",
       "city_id                 0.000000\n",
       "category_1              0.000000\n",
       "installments            0.000000\n",
       "category_3              0.028488\n",
       "merchant_category_id    0.000000\n",
       "merchant_id             0.013355\n",
       "month_lag               0.000000\n",
       "purchase_amount         0.000000\n",
       "purchase_date           0.000000\n",
       "category_2              0.056925\n",
       "state_id                0.000000\n",
       "subsector_id            0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.isnull().sum()/len(new_hist_trans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a standard practice to remove the rows which have missing data. Doing that had negative impact in the leaderboard score for me & others in the competition. So I decided to fill in the missing values with mode for categorical columns & for installments with -1 & 999. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's map `category_1` & `category_3` as categorical columns with key & value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fill_nas_for_transactions_df(df):\n",
    "    # Fill nas for category_3 with mode\n",
    "    df['category_2'].fillna(1.0,inplace=True)\n",
    "    df['category_3'].fillna('A',inplace=True)\n",
    "    df['merchant_id'].fillna('M_ID_00a6ca8a8a',inplace=True)\n",
    "    df['installments'].replace(-1, np.nan,inplace=True)\n",
    "    df['installments'].replace(999, np.nan,inplace=True)\n",
    "    df['category_1'] = df['category_1'].map({'Y':1, 'N':0}) \n",
    "    df['category_3'] = df['category_3'].map({'A':0, 'B':1, 'C':2})\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = [hist_trans, new_hist_trans]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans, new_hist_trans = [fill_nas_for_transactions_df(df) for df in dfs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add date part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fast.ai library provides excellent utility functions. We will be leveraging functions from fast.ai throughout for data manipulation & feature engineering starting with date field. `add_datepart` function reads a date and generates additional interpreted fields which are very useful for feature engineering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_datepart(hist_trans, 'purchase_date', drop=False, time=True)\n",
    "add_datepart(new_hist_trans, 'purchase_date', drop=False, time=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_ef55cf8d4b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>107</td>\n",
       "      <td>140</td>\n",
       "      <td>330</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>307</td>\n",
       "      <td>307</td>\n",
       "      <td>507</td>\n",
       "      <td>661</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_b0c793002c</td>\n",
       "      <td>M_ID_88920c89e8</td>\n",
       "      <td>M_ID_ad5237ef6b</td>\n",
       "      <td>M_ID_9e84cda3b1</td>\n",
       "      <td>M_ID_3c86fa3831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>-0.557574</td>\n",
       "      <td>-0.56958</td>\n",
       "      <td>-0.551037</td>\n",
       "      <td>-0.671925</td>\n",
       "      <td>-0.659904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2018-03-11 14:57:36</td>\n",
       "      <td>2018-03-19 18:53:37</td>\n",
       "      <td>2018-04-26 14:08:44</td>\n",
       "      <td>2018-03-07 09:43:21</td>\n",
       "      <td>2018-03-22 21:07:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Year</th>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Month</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Week</th>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Day</th>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>7</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofweek</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofyear</th>\n",
       "      <td>70</td>\n",
       "      <td>78</td>\n",
       "      <td>116</td>\n",
       "      <td>66</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Hour</th>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Minute</th>\n",
       "      <td>57</td>\n",
       "      <td>53</td>\n",
       "      <td>8</td>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Second</th>\n",
       "      <td>36</td>\n",
       "      <td>37</td>\n",
       "      <td>44</td>\n",
       "      <td>21</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Elapsed</th>\n",
       "      <td>1520780256</td>\n",
       "      <td>1521485617</td>\n",
       "      <td>1524751724</td>\n",
       "      <td>1520415801</td>\n",
       "      <td>1521752873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             0                    1  \\\n",
       "authorized_flag                              Y                    Y   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    107                  140   \n",
       "category_1                                   0                    0   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       307                  307   \n",
       "merchant_id                    M_ID_b0c793002c      M_ID_88920c89e8   \n",
       "month_lag                                    1                    1   \n",
       "purchase_amount                      -0.557574             -0.56958   \n",
       "purchase_date              2018-03-11 14:57:36  2018-03-19 18:53:37   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                    9   \n",
       "subsector_id                                19                   19   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               3                    3   \n",
       "purchase_Week                               10                   12   \n",
       "purchase_Day                                11                   19   \n",
       "purchase_Dayofweek                           6                    0   \n",
       "purchase_Dayofyear                          70                   78   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                   18   \n",
       "purchase_Minute                             57                   53   \n",
       "purchase_Second                             36                   37   \n",
       "purchase_Elapsed                    1520780256           1521485617   \n",
       "\n",
       "                                             2                    3  \\\n",
       "authorized_flag                              Y                    Y   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    330                   -1   \n",
       "category_1                                   0                    1   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       507                  661   \n",
       "merchant_id                    M_ID_ad5237ef6b      M_ID_9e84cda3b1   \n",
       "month_lag                                    2                    1   \n",
       "purchase_amount                      -0.551037            -0.671925   \n",
       "purchase_date              2018-04-26 14:08:44  2018-03-07 09:43:21   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                   -1   \n",
       "subsector_id                                14                    8   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               4                    3   \n",
       "purchase_Week                               17                   10   \n",
       "purchase_Day                                26                    7   \n",
       "purchase_Dayofweek                           3                    2   \n",
       "purchase_Dayofyear                         116                   66   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                    9   \n",
       "purchase_Minute                              8                   43   \n",
       "purchase_Second                             44                   21   \n",
       "purchase_Elapsed                    1524751724           1520415801   \n",
       "\n",
       "                                             4  \n",
       "authorized_flag                              Y  \n",
       "card_id                        C_ID_ef55cf8d4b  \n",
       "city_id                                     -1  \n",
       "category_1                                   1  \n",
       "installments                                 1  \n",
       "category_3                                   1  \n",
       "merchant_category_id                       166  \n",
       "merchant_id                    M_ID_3c86fa3831  \n",
       "month_lag                                    1  \n",
       "purchase_amount                      -0.659904  \n",
       "purchase_date              2018-03-22 21:07:53  \n",
       "category_2                                   1  \n",
       "state_id                                    -1  \n",
       "subsector_id                                29  \n",
       "purchase_Year                             2018  \n",
       "purchase_Month                               3  \n",
       "purchase_Week                               12  \n",
       "purchase_Day                                22  \n",
       "purchase_Dayofweek                           3  \n",
       "purchase_Dayofyear                          81  \n",
       "purchase_Is_month_end                    False  \n",
       "purchase_Is_month_start                  False  \n",
       "purchase_Is_quarter_end                  False  \n",
       "purchase_Is_quarter_start                False  \n",
       "purchase_Is_year_end                     False  \n",
       "purchase_Is_year_start                   False  \n",
       "purchase_Hour                               21  \n",
       "purchase_Minute                              7  \n",
       "purchase_Second                             53  \n",
       "purchase_Elapsed                    1521752873  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add extra columns (purchased on weekend, monthend, month_diff etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`add_datepart` already added columns like `'Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',\n",
    "            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start'` to our dataframe. Let's add some more columns like if the transaction has been done on weekday or weekend, `month_diff` which says the months since the reference date. We will find out later that these columns turned out to be pretty strong features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai/lib/python3.6/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n",
      "  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAELCAYAAADQsFGkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHvRJREFUeJzt3X+cVXW97/HXe++BGVREBbSOQIMP8Qckao5wyx9ZpGE3JQ0TtSum5gO9lHUfZnoqU8vT0es9Po6mcbhqIZpi/jhNhg/KH2SaBwF/oIhcR6Sc8BgKDWLMwMx87h97DWez28PsYTbuPa738/GYx17ru75r789abPZ7/dh7LUUEZmaWPplKF2BmZpXhADAzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYpVVPpAgoNGzYs6uvrK12GmVm/snTp0rcjYnhv5qm6AKivr2fJkiWVLsPMrF+R9MfezuNDQGZmKeUAMDNLKQeAmVlKVd05gGK2bNlCc3Mzra2tlS7F3id1dXWMGDGCAQMGVLoUsw+sfhEAzc3NDB48mPr6eiRVuhzbySKCd955h+bmZkaPHl3pcsw+sPrFIaDW1laGDh3qD/+UkMTQoUO9x2e2k/WLAAD84Z8y/vc22/n6TQCYmVl5OQDMzFKqX5wE/iDr+uXzsGHDKl3K+27hwoUMHDiQT3ziE5UuZVtLflrpCmxHNXyl0hX0K94DeB+0t7dXuoSqtHDhQv7whz9Uugyz1Op3ewBX/Wo5L6/ZUNbnHPsPu/P9k8Ztt8/q1auZPHkyEydO5LnnnuOAAw7gjjvuYOzYsVu34JcsWcIll1zCwoULufLKK1mzZg2rV69m2LBhzJ07l29/+9ssWLAASXz1q1/la1/7GgA33XQTv/rVr9iyZQu/+MUvOOigg3jmmWf4xje+waZNmxg0aBA//elPOfDAA1m+fDlf+cpX2Lx5M52dndx///2MGTOGO++8kxtvvJHNmzczceJEbrnlFrLZbNFlufDCC1m8eDGbNm1i6tSpXHXVVUBub+TMM8/k8ccfZ8uWLcyePZvLL7+cpqYmvvWtbzFjxgwigksvvZSHH34YSXz3u9/l9NNPZ+HChVx//fU89NBDAMycOZOGhgbOOecc6uvrmT59+jbLWFdXx6xZs8hms9x5553cdNNNHHPMMWX8VzWznvS7AKiklStXctttt3HUUUdx7rnncsstt2y3/9KlS3nyyScZNGgQP/nJT3j99dd57rnnqKmpYd26dVv7DRs2jGeffZZbbrmF66+/nltvvZWDDjqIJ554gpqaGh555BH+8R//kfvvv59Zs2Zx8cUXc9ZZZ7F582Y6OjpYsWIF8+bN46mnnmLAgAFcdNFF3HXXXZx99tlF67rmmmvYa6+96OjoYNKkSSxbtozx48cDMHLkSJ5++mm++c1vcs455/DUU0/R2trKuHHjmDFjBg888ADPP/88L7zwAm+//TZHHnkkxx57bI/rrtgyzpgxg912241LLrmkF/8KZlYu/S4AetpS35lGjhzJUUcdBcCXv/xlbrzxxu32P/nkkxk0aBAAjzzyCDNmzKCmJrfK99prr639Tj31VACOOOIIHnjgAQBaWlqYPn06r776KpLYsmULAB//+Me55ppraG5u5tRTT2XMmDE8+uijLF26lCOPPBKATZs2sffee3db17333svs2bNpb2/nzTff5OWXX94aACeffDIAhxxyCBs3bmTw4MEMHjyYuro6/vrXv/Lkk09yxhlnkM1m2WefffjkJz/J4sWL2X333be7Looto5lVVr8LgEoq/G66JGpqaujs7AT4ux8u7brrrluHI6Lb77bX1tYCkM1mt54v+N73vsenPvUpHnzwQVavXs1xxx0HwJlnnsnEiRP59a9/zWc/+1luvfVWIoLp06fzox/9qMdleP3117n++utZvHgxe+65J+ecc842dXfVkslktg53jbe3txMRRZ83fz0UWxfFltHMKssngXvhT3/6E08//TQAd999N0cffTT19fUsXboUgPvvv7/beU844QRmzZq19cMv/xBQMS0tLey7774A/OxnP9vavmrVKvbbbz++/vWvc/LJJ7Ns2TImTZrEfffdx1/+8petz/3HPxa/NPiGDRvYddddGTJkCG+99RYPP/xwaQufOPbYY5k3bx4dHR2sXbuWJ554ggkTJvCRj3yEl19+mba2NlpaWnj00Ud7fK7Bgwfz7rvv9ur1zax8HAC9cPDBBzNnzhzGjx/PunXruPDCC/n+97/PxRdfzDHHHNPtSVeA888/n1GjRjF+/HgOPfRQfv7zn2/3tS699FIuv/xyjjrqKDo6Ora2z5s3j49+9KMcdthhvPLKK5x99tmMHTuWH/7wh5xwwgmMHz+e448/njfffLPo8x566KEcfvjhjBs3jnPPPXfrIa1SnXLKKVuX4dOf/jTXXXcdH/rQhxg5ciRf+tKXGD9+PGeddRaHH354j8910kkn8eCDD3LYYYfx+9//vld1mFnfqbtd+kppaGiIwjuCrVixgoMPPrhCFeWsXr2az3/+87z00ksVrSNNKvbv7t8B9F8p/h2ApKUR0dCbebwHYGaWUj4JXKL6+vp+t/U/ceJE2tratmmbO3cuhxxySIUqMrNq0m8CYHvforHiFi1aVOkSdli1HZo0+yAq6RCQpMmSVkpqknRZkem1kuYl0xdJqk/az5L0fN5fp6TDeltkXV0d77zzjj8UUqLrhjB1dXWVLsXsA63HPQBJWeBm4HigGVgsqTEiXs7rdh6wPiL2lzQNuBY4PSLuAu5KnucQ4JcR8XxvixwxYgTNzc2sXbu2t7NaP9V1S0gz23lKOQQ0AWiKiFUAku4BpgD5ATAFuDIZvg/4sSTFtpvsZwB370iRAwYM8K0BzczKrJRDQPsCb+SNNydtRftERDvQAgwt6HM6OxgAZmZWfqUEQLEzr4UH47fbR9JE4G8RUfRrNJIukLRE0hIf5jEze3+UEgDNwMi88RHAmu76SKoBhgD51zqYxna2/iNidkQ0RETD8OHDS6nbzMz6qJQAWAyMkTRa0kByH+aNBX0agenJ8FTgsa7j/5IywGnAPeUp2czMyqHHk8AR0S5pJrAAyAK3R8RySVcDSyKiEbgNmCupidyW/7S8pzgWaO46iWxmZtWhpB+CRcR8YH5B2xV5w63ktvKLzbsQ+G87XqKZme0MvhaQmVlKOQDMzFLKAWBmllIOADOzlHIAmJmllAPAzCylHABmZinlADAzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYp5QAwM0spB4CZWUo5AMzMUqqkAJA0WdJKSU2SLisyvVbSvGT6Ikn1edPGS3pa0nJJL0qqK1/5Zma2o3oMAElZ4GbgRGAscIaksQXdzgPWR8T+wA3Atcm8NcCdwIyIGAccB2wpW/VmZrbDStkDmAA0RcSqiNgM3ANMKegzBZiTDN8HTJIk4ARgWUS8ABAR70RER3lKNzOzviglAPYF3sgbb07aivaJiHagBRgKHACEpAWSnpV0ad9LNjOzcqgpoY+KtEWJfWqAo4Ejgb8Bj0paGhGPbjOzdAFwAcCoUaNKKMnMzPqqlD2AZmBk3vgIYE13fZLj/kOAdUn77yLi7Yj4GzAf+FjhC0TE7IhoiIiG4cOH934pzMys10oJgMXAGEmjJQ0EpgGNBX0agenJ8FTgsYgIYAEwXtIuSTB8Eni5PKWbmVlf9HgIKCLaJc0k92GeBW6PiOWSrgaWREQjcBswV1ITuS3/acm86yX9C7kQCWB+RPx6Jy2LmZn1QinnAIiI+eQO3+S3XZE33Aqc1s28d5L7KqiZmVUR/xLYzCylHABmZinlADAzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYp5QAwM0spB4CZWUo5AMzMUsoBYGaWUg4AM7OUcgCYmaWUA8DMLKUcAGZmKVVSAEiaLGmlpCZJlxWZXitpXjJ9kaT6pL1e0iZJzyd/s8pbvpmZ7age7wksKQvcDBwPNAOLJTVGxMt53c4D1kfE/pKmAdcCpyfTXouIw8pct5mZ9VEpewATgKaIWBURm4F7gCkFfaYAc5Lh+4BJklS+Ms3MrNxKCYB9gTfyxpuTtqJ9IqIdaAGGJtNGS3pO0u8kHdPHes3MrEx6PAQEFNuSjxL7vAmMioh3JB0B/LukcRGxYZuZpQuACwBGjRpVQklmZtZXpewBNAMj88ZHAGu66yOpBhgCrIuItoh4ByAilgKvAQcUvkBEzI6IhohoGD58eO+XwszMeq2UAFgMjJE0WtJAYBrQWNCnEZieDE8FHouIkDQ8OYmMpP2AMcCq8pRuZmZ90eMhoIholzQTWABkgdsjYrmkq4ElEdEI3AbMldQErCMXEgDHAldLagc6gBkRsW5nLIiZmfVOKecAiIj5wPyCtivyhluB04rMdz9wfx9rNDOzncC/BDYzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYp5QAwM0spB4CZWUo5AMzMUsoBYGaWUg4AM7OUcgCYmaWUA8DMLKUcAGZmKeUAMDNLKQeAmVlKOQDMzFKqpACQNFnSSklNki4rMr1W0rxk+iJJ9QXTR0naKOmS8pRtZmZ91WMASMoCNwMnAmOBMySNLeh2HrA+IvYHbgCuLZh+A/Bw38s1M7NyKWUPYALQFBGrImIzcA8wpaDPFGBOMnwfMEmSACR9AVgFLC9PyWZmVg6lBMC+wBt5481JW9E+EdEOtABDJe0KfBu4qu+lmplZOZUSACrSFiX2uQq4ISI2bvcFpAskLZG0ZO3atSWUZGZmfVVTQp9mYGTe+AhgTTd9miXVAEOAdcBEYKqk64A9gE5JrRHx4/yZI2I2MBugoaGhMFzMzGwnKCUAFgNjJI0G/gxMA84s6NMITAeeBqYCj0VEAMd0dZB0JbCx8MPfzMwqo8cAiIh2STOBBUAWuD0ilku6GlgSEY3AbcBcSU3ktvyn7cyizcys70rZAyAi5gPzC9quyBtuBU7r4Tmu3IH6zMxsJ/Evgc3MUsoBYGaWUg4AM7OUcgCYmaWUA8DMLKUcAGZmKeUAMDNLKQeAmVlKOQDMzFLKAWBmllIOADOzlHIAmJmllAPAzCylHABmZinlADAzSykHgJlZSjkAzMxSygFgZpZSJQWApMmSVkpqknRZkem1kuYl0xdJqk/aJ0h6Pvl7QdIp5S3fzMx2VI8BICkL3AycCIwFzpA0tqDbecD6iNgfuAG4Nml/CWiIiMOAycC/SSrpPsRmZrZzlbIHMAFoiohVEbEZuAeYUtBnCjAnGb4PmCRJEfG3iGhP2uuAKEfRZmbWd6UEwL7AG3njzUlb0T7JB34LMBRA0kRJy4EXgRl5gWBmZhVUSgCoSFvhlny3fSJiUUSMA44ELpdU93cvIF0gaYmkJWvXri2hJDMz66tSAqAZGJk3PgJY012f5Bj/EGBdfoeIWAG8B3y08AUiYnZENEREw/Dhw0uv3szMdlgpAbAYGCNptKSBwDSgsaBPIzA9GZ4KPBYRkcxTAyDpI8CBwOqyVG5mZn3S4zdyIqJd0kxgAZAFbo+I5ZKuBpZERCNwGzBXUhO5Lf9pyexHA5dJ2gJ0AhdFxNs7Y0HMzKx3SvpKZkTMB+YXtF2RN9wKnFZkvrnA3D7WaGZmO4F/CWxmllIOADOzlHIAmJmllAPAzCylHABmZinlADAzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYp5QAwM0spB4CZWUo5AMzMUsoBYGaWUg4AM7OUKikAJE2WtFJSk6TLikyvlTQvmb5IUn3SfrykpZJeTB4/Xd7yzcxsR/UYAJKywM3AicBY4AxJYwu6nQesj4j9gRuAa5P2t4GTIuIQYDq+P7CZWdUoZQ9gAtAUEasiYjNwDzCloM8UYE4yfB8wSZIi4rmIWJO0LwfqJNWWo3AzM+ubUgJgX+CNvPHmpK1on4hoB1qAoQV9vgg8FxFtO1aqmZmVU00JfVSkLXrTR9I4coeFTij6AtIFwAUAo0aNKqEkMzPrq1L2AJqBkXnjI4A13fWRVAMMAdYl4yOAB4GzI+K1Yi8QEbMjoiEiGoYPH967JTAzsx1SSgAsBsZIGi1pIDANaCzo00juJC/AVOCxiAhJewC/Bi6PiKfKVbSZmfVdjwGQHNOfCSwAVgD3RsRySVdLOjnpdhswVFIT8L+Arq+KzgT2B74n6fnkb++yL4WZmfVaKecAiIj5wPyCtivyhluB04rM90Pgh32s0czMdgL/EtjMLKUcAGZmKeUAMDNLKQeAmVlKOQDMzFLKAWBmllIOADOzlHIAmJmllAPAzCylHABmZinlADAzSykHgJlZSjkAzMxSygFgZpZSDgAzs5RyAJiZpZQDwMwspRwAZmYp5QAwM0upkgJA0mRJKyU1SbqsyPRaSfOS6Ysk1SftQyU9LmmjpB+Xt3QzM+uLHgNAUha4GTgRGAucIWlsQbfzgPURsT9wA3Bt0t4KfA+4pGwVm5lZWZSyBzABaIqIVRGxGbgHmFLQZwowJxm+D5gkSRHxXkQ8SS4IzMysipQSAPsCb+SNNydtRftERDvQAgwtR4FmZrZzlBIAKtIWO9Cn+xeQLpC0RNKStWvXljqbmZn1QSkB0AyMzBsfAazpro+kGmAIsK7UIiJidkQ0RETD8OHDS53NzMz6oJQAWAyMkTRa0kBgGtBY0KcRmJ4MTwUei4iS9wDMzOz9V9NTh4holzQTWABkgdsjYrmkq4ElEdEI3AbMldREbst/Wtf8klYDuwMDJX0BOCEiXi7/opiZWW/0GAAAETEfmF/QdkXecCtwWjfz1vehPjMz20n8S2Azs5RyAJiZpZQDwMwspRwAZmYp5QAwM0spB4CZWUo5AMzMUsoBYGaWUg4AM7OUcgCYmaWUA8DMLKUcAGZmKeUAMDNLKQeAmVlKOQDMzFLKAWBmllIOADOzlHIAmJmlVEkBIGmypJWSmiRdVmR6raR5yfRFkurzpl2etK+U9NnylW5mZn3R4z2BJWWBm4HjgWZgsaTGghu7nwesj4j9JU0DrgVOlzSW3A3ixwH/ADwi6YCI6Cj3gpj1Vx2dnazf8B7rN2xg48YNbH5vA52tLdS0tZDtbCWUpVM1hLK5v0wNKJM8ZlEmC8mfMjUok03+MmQyWTLZGmoGDKR24EBqa2sZVFvLoNqB7FI7gEzGBwHSrJSbwk8AmiJiFYCke4ApQH4ATAGuTIbvA34sSUn7PRHRBrwuqSl5vqfLU75ZebVueo+29zaRzWTIZkRGucdsJoMy6vXzbdzUxjstG3h347ts2riB9k0tZFpbGLj5r+za/lf2iPXsFS0MUyfD8uZrjwzvaA82aRDZ6CBLOzV0UBMd1NDOgGR8gPq2LfVe1PE31dFKLW2qo011bMnW0Z6poyNbR2RriZpalK1F2QGQyeTCRZmtAZN7zJBRFmVzj5msyGayZDJZspkMmWwm95jJIAGIrrUpKfeXG0laY5v2rt7K5Obtmm/r/F2PG1uStkxuevIc+W25v8y2r53SICwlAPYF3sgbbwYmdtcnItoltQBDk/b/KJh33x2u1mwne+GXNzLxlX8uOq0zRAcZOskkj8obztCp5JEMgdg93mU3tbJbwfNsiF1Yl9mTdzN78McBH+a12iFkBu3BwF0Gs9tuu7Pn7ruz5+67sk8JH0qdnZ1sae+kvbOT9o52trR30N7RSUdHB+0dHbnH9g42b9lM++Y22re00dHeRmxpI9rbUHsbmY5Wsh1t1HS2MqCzjd3a/0pdtFIXrexCK7uorQxr9n3yyKV9forOEAEEIpKwCYBkPJJ+207ftm9ISf/Cvt3P35NS+vRWKQFQ7FWjxD6lzIukC4ALktE2SS+VUFelDQPernQRJXCd5VWGOjcA/1mOWranP6zP/lAj9J86D+ztDKUEQDMwMm98BLCmmz7NkmqAIcC6EuclImYDswEkLYmIhlIXoFJcZ3m5zvLqD3X2hxqhf9XZ23lKOfC1GBgjabSkgeRO6jYW9GkEpifDU4HHIiKS9mnJt4RGA2OAZ3pbpJmZlV+PewDJMf2ZwAIgC9weEcslXQ0siYhG4DZgbnKSdx25kCDpdy+5E8btwP/0N4DMzKpDKYeAiIj5wPyCtivyhluB07qZ9xrgml7UNLsXfSvJdZaX6yyv/lBnf6gRPsB1KnekxszM0iadX341M7PqCQBJ/1vSK5KWSXpQ0h5506rmchKSTpO0XFKnpIa89npJmyQ9n/zNqsY6k2lVsz7zSbpS0p/z1uHnKl1Tl54uh1ItJK2W9GKy/nr9rZCdRdLtkv6S/xVvSXtJ+q2kV5PHPStZY1JTsTqr7n0paaSkxyWtSP6fX5y0926dRkRV/AEnADXJ8LXAtcnwWOAFoBYYDbwGZCtY58Hkvm+7EGjIa68HXqr0eiyhzqpanwU1XwlcUuk6itSVTdbTfsDAZP2NrXRd3dS6GhhW6TqK1HUs8LH8/yPAdcBlyfBlXf/nq7DOqntfAh8GPpYMDwb+X/J/u1frtGr2ACLiNxHRnoz+B7nfDEDe5SQi4nWg63ISFRERKyJiZaVev1TbqbOq1mc/sfVyKBGxGei6HIqVKCKeIPcNwXxTgDnJ8BzgC+9rUUV0U2fViYg3I+LZZPhdYAW5qyz0ap1WTQAUOBd4OBkudimKar2cxGhJz0n6naRjKl1MN6p9fc5MDgPeXg2HBBLVvs7yBfAbSUuTX9hXs30i4k3IfaABe1e4nu2pxvclkDv8DBwOLKKX67Skr4GWi6RHgA8VmfSdiPhl0uc75H4zcFfXbEX679SvLpVSZxFvAqMi4h1JRwD/LmlcRGyosjrf9/W5zYtvp2bgJ8APknp+APwfchsDlVbRddZLR0XEGkl7A7+V9EqyVWs7rlrfl0jaDbgf+EZEbJB6d72g9zUAIuIz25suaTrweWBSJAexKPFyEuXUU53dzNMGtCXDSyW9BhwA7LQTcTtSJxVYn/lKrVnS/wUe2snllKqi66w3ImJN8vgXSQ+SO3xVrQHwlqQPR8Sbkj4M/KXSBRUTEW91DVfT+1LSAHIf/ndFxANJc6/WadUcApI0Gfg2cHJE/C1vUr+4nISk4crdOwFJ+5Grc1Vlqyqqatdn8obtcgpQLRcFLOVyKBUnaVdJg7uGyX2xolrWYTH5l5CZDnS311pR1fi+VG5T/zZgRUT8S96k3q3TSp/Nzjur3UTuOOvzyd+svGnfIfctjJXAiRWu8xRyW4RtwFvAgqT9i8Byct8QeRY4qRrrrLb1WVDzXOBFYFnyRv5wpWvKq+1z5L5p8Rq5Q2wVr6lIjfsl778Xkvdi1dQJ3E3uMOmW5H15HrlLxj8KvJo87lWldVbd+xI4mtwhqWV5n5mf6+069S+BzcxSqmoOAZmZ2fvLAWBmllIOADOzlHIAmJmllAPAzCylHABmZinlALAPpOSyyMMqXUclSDpO0icqXYdVPweA9VuS3tdLmfQjxwEOAOuRA8AqKrmRziuS5iRXW7xP0i75W/CSGiQtTIavlDRb0m+AOyRlJV2f3ARlmaSv5T391yQ9m0w7KJl/gqQ/JFdt/YOkA5P2cZKeSW74sUzSmKT9y3nt/9Z1uY9uluUnkpYkN+i4Kq99taR/kvR0Mv1jkhZIek3SjKSPlLsp0ktJvacn7cdJeijvuX4s6Zy8570qfxmTK0POAL6Z1FytV6W1KuAtKKsGBwLnRcRTkm4HLuqh/xHA0RGxSdKF5G5sc3hEtEvaK6/f2xHxMUkXAZcA5wOvAMcmfT8D/BO5y3jMAP41Iu5KrveTlXQwcDq5K2xukXQLcBZwRzd1fSci1iUh8aik8RGxLJn2RkR8XNINwM+Ao4A6cpdsmAWcChwGHAoMAxZLKuUibtssY0Scr9zd6DZGxPUlzG8p5gCwavBGRDyVDN8JfL2H/o0RsSkZ/gy560a1A0RE/s08uq6QuJTcByzAEGBOsoUfwICk/WngO5JGAA9ExKuSJpELm8XJZXYHsf2rK35JuWvw15C7Y9NYctdqgf+6eNyLwG6Ru4nHu5Jalbv96dHA3RHRQe6Kjr8DjgR6upx4sWU0K4kDwKpB4QWpgtw9IboOUdYVTH8vb1hF5u/Sljx28F/v9R8Aj0fEKcnhkoUAEfFzSYuA/w4skHR+8txzIuLynhYgubLqJcCREbFe0s8K6u6qpTNvuGu8huL3HIBt1wP8/bootoxmJfE5AKsGoyR9PBk+A3iS3L1tj0javrideX8DzOg6IVxwCKiYIcCfk+FzuhqTS3iviogbyW2tjyd3NcWpyt1cpeuG2x/p5nl3JxdMLZL2AU7soY5CTwCnJ+c0hpO7N+0zwB+Bscnlu4cAk0p4rnfJ3SfWbLscAFYNVgDTJS0D9iJ3B6argH+V9HtyW7fduRX4E7BM0gvAmT281nXAjyQ9Re5m711OB16S9DxwEHBHRLwMfJfcLRaXAb8ld2jn70TEC8Bz5I7p3w48VazfdjxI7nDRC8BjwKUR8Z8R8QZwbzLtruQ1evIr4BSfBLae+HLQVlHJYZiHIuKjFS7FLHW8B2BmllLeAzDrpeRkcW1B8/+IiBcrUY/ZjnIAmJmllA8BmZmllAPAzCylHABmZinlADAzSykHgJlZSv1/Y55iMKFboKoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.kdeplot(new_hist_trans['purchase_amount'])\n",
    "# sns.kdeplot(new_hist_trans['purchase_amount'], bw=.2, label=\"bw: 0.2\")\n",
    "# sns.kdeplot(new_hist_trans['purchase_amount'])\n",
    "plt.legend();\n",
    "plt.xlim(-20,20)\n",
    "sns.distplot(new_hist_trans['purchase_amount'], bins=20, kde=True, rug=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_extra_cols(df):\n",
    "    df['purchased_on_weekend'] = (df.purchase_Dayofweek >=5).astype(int)\n",
    "    df['purchased_on_weekday'] = (df.purchase_Dayofweek <5).astype(int)\n",
    "    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days)//30\n",
    "    df['month_diff'] += df['month_lag']\n",
    "    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0})\n",
    "#     Trim the purchase_amount\n",
    "#     df['purchase_amount'] = df['purchase_amount'].apply(lambda x: min(x, 0.8)) \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans, new_hist_trans = [add_extra_cols(df) for df in dfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((29112361, 33), (1963031, 33))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans.shape, new_hist_trans.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_ef55cf8d4b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>107</td>\n",
       "      <td>140</td>\n",
       "      <td>330</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>307</td>\n",
       "      <td>307</td>\n",
       "      <td>507</td>\n",
       "      <td>661</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_b0c793002c</td>\n",
       "      <td>M_ID_88920c89e8</td>\n",
       "      <td>M_ID_ad5237ef6b</td>\n",
       "      <td>M_ID_9e84cda3b1</td>\n",
       "      <td>M_ID_3c86fa3831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>-0.557574</td>\n",
       "      <td>-0.56958</td>\n",
       "      <td>-0.551037</td>\n",
       "      <td>-0.671925</td>\n",
       "      <td>-0.659904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2018-03-11 14:57:36</td>\n",
       "      <td>2018-03-19 18:53:37</td>\n",
       "      <td>2018-04-26 14:08:44</td>\n",
       "      <td>2018-03-07 09:43:21</td>\n",
       "      <td>2018-03-22 21:07:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Year</th>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Month</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Week</th>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Day</th>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>7</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofweek</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofyear</th>\n",
       "      <td>70</td>\n",
       "      <td>78</td>\n",
       "      <td>116</td>\n",
       "      <td>66</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Hour</th>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Minute</th>\n",
       "      <td>57</td>\n",
       "      <td>53</td>\n",
       "      <td>8</td>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Second</th>\n",
       "      <td>36</td>\n",
       "      <td>37</td>\n",
       "      <td>44</td>\n",
       "      <td>21</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Elapsed</th>\n",
       "      <td>1520780256</td>\n",
       "      <td>1521485617</td>\n",
       "      <td>1524751724</td>\n",
       "      <td>1520415801</td>\n",
       "      <td>1521752873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekend</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekday</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_diff</th>\n",
       "      <td>19</td>\n",
       "      <td>18</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             0                    1  \\\n",
       "authorized_flag                              1                    1   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    107                  140   \n",
       "category_1                                   0                    0   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       307                  307   \n",
       "merchant_id                    M_ID_b0c793002c      M_ID_88920c89e8   \n",
       "month_lag                                    1                    1   \n",
       "purchase_amount                      -0.557574             -0.56958   \n",
       "purchase_date              2018-03-11 14:57:36  2018-03-19 18:53:37   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                    9   \n",
       "subsector_id                                19                   19   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               3                    3   \n",
       "purchase_Week                               10                   12   \n",
       "purchase_Day                                11                   19   \n",
       "purchase_Dayofweek                           6                    0   \n",
       "purchase_Dayofyear                          70                   78   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                   18   \n",
       "purchase_Minute                             57                   53   \n",
       "purchase_Second                             36                   37   \n",
       "purchase_Elapsed                    1520780256           1521485617   \n",
       "purchased_on_weekend                         1                    0   \n",
       "purchased_on_weekday                         0                    1   \n",
       "month_diff                                  19                   18   \n",
       "\n",
       "                                             2                    3  \\\n",
       "authorized_flag                              1                    1   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    330                   -1   \n",
       "category_1                                   0                    1   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       507                  661   \n",
       "merchant_id                    M_ID_ad5237ef6b      M_ID_9e84cda3b1   \n",
       "month_lag                                    2                    1   \n",
       "purchase_amount                      -0.551037            -0.671925   \n",
       "purchase_date              2018-04-26 14:08:44  2018-03-07 09:43:21   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                   -1   \n",
       "subsector_id                                14                    8   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               4                    3   \n",
       "purchase_Week                               17                   10   \n",
       "purchase_Day                                26                    7   \n",
       "purchase_Dayofweek                           3                    2   \n",
       "purchase_Dayofyear                         116                   66   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                    9   \n",
       "purchase_Minute                              8                   43   \n",
       "purchase_Second                             44                   21   \n",
       "purchase_Elapsed                    1524751724           1520415801   \n",
       "purchased_on_weekend                         0                    0   \n",
       "purchased_on_weekday                         1                    1   \n",
       "month_diff                                  18                   19   \n",
       "\n",
       "                                             4  \n",
       "authorized_flag                              1  \n",
       "card_id                        C_ID_ef55cf8d4b  \n",
       "city_id                                     -1  \n",
       "category_1                                   1  \n",
       "installments                                 1  \n",
       "category_3                                   1  \n",
       "merchant_category_id                       166  \n",
       "merchant_id                    M_ID_3c86fa3831  \n",
       "month_lag                                    1  \n",
       "purchase_amount                      -0.659904  \n",
       "purchase_date              2018-03-22 21:07:53  \n",
       "category_2                                   1  \n",
       "state_id                                    -1  \n",
       "subsector_id                                29  \n",
       "purchase_Year                             2018  \n",
       "purchase_Month                               3  \n",
       "purchase_Week                               12  \n",
       "purchase_Day                                22  \n",
       "purchase_Dayofweek                           3  \n",
       "purchase_Dayofyear                          81  \n",
       "purchase_Is_month_end                    False  \n",
       "purchase_Is_month_start                  False  \n",
       "purchase_Is_quarter_end                  False  \n",
       "purchase_Is_quarter_start                False  \n",
       "purchase_Is_year_end                     False  \n",
       "purchase_Is_year_start                   False  \n",
       "purchase_Hour                               21  \n",
       "purchase_Minute                              7  \n",
       "purchase_Second                             53  \n",
       "purchase_Elapsed                    1521752873  \n",
       "purchased_on_weekend                         0  \n",
       "purchased_on_weekday                         1  \n",
       "month_diff                                  18  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### De normalizing purchase amount column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `purchase_amount` column is normalized and heavily transformed. Let's try to reverse engineer it to get the actual amount. We will solve a simple optimisation function to make the least value of purchase amount to be non-negative and get the regression co-efficients. We will round them to 2 decimals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans['purchase_amount'] = np.round(new_hist_trans['purchase_amount'] / 0.00150265118 + 497.06,2)\n",
    "hist_trans['purchase_amount'] = np.round(hist_trans['purchase_amount'] / 0.00150265118 + 497.06,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_415bb3a509</td>\n",
       "      <td>C_ID_ef55cf8d4b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>107</td>\n",
       "      <td>140</td>\n",
       "      <td>330</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>307</td>\n",
       "      <td>307</td>\n",
       "      <td>507</td>\n",
       "      <td>661</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_b0c793002c</td>\n",
       "      <td>M_ID_88920c89e8</td>\n",
       "      <td>M_ID_ad5237ef6b</td>\n",
       "      <td>M_ID_9e84cda3b1</td>\n",
       "      <td>M_ID_3c86fa3831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>84348.9</td>\n",
       "      <td>79031.6</td>\n",
       "      <td>87243.7</td>\n",
       "      <td>33705</td>\n",
       "      <td>39029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2018-03-11 14:57:36</td>\n",
       "      <td>2018-03-19 18:53:37</td>\n",
       "      <td>2018-04-26 14:08:44</td>\n",
       "      <td>2018-03-07 09:43:21</td>\n",
       "      <td>2018-03-22 21:07:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Year</th>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Month</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Week</th>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Day</th>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>7</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofweek</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofyear</th>\n",
       "      <td>70</td>\n",
       "      <td>78</td>\n",
       "      <td>116</td>\n",
       "      <td>66</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Hour</th>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Minute</th>\n",
       "      <td>57</td>\n",
       "      <td>53</td>\n",
       "      <td>8</td>\n",
       "      <td>43</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Second</th>\n",
       "      <td>36</td>\n",
       "      <td>37</td>\n",
       "      <td>44</td>\n",
       "      <td>21</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Elapsed</th>\n",
       "      <td>1520780256</td>\n",
       "      <td>1521485617</td>\n",
       "      <td>1524751724</td>\n",
       "      <td>1520415801</td>\n",
       "      <td>1521752873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekend</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekday</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_diff</th>\n",
       "      <td>19</td>\n",
       "      <td>18</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             0                    1  \\\n",
       "authorized_flag                              1                    1   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    107                  140   \n",
       "category_1                                   0                    0   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       307                  307   \n",
       "merchant_id                    M_ID_b0c793002c      M_ID_88920c89e8   \n",
       "month_lag                                    1                    1   \n",
       "purchase_amount                        84348.9              79031.6   \n",
       "purchase_date              2018-03-11 14:57:36  2018-03-19 18:53:37   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                    9   \n",
       "subsector_id                                19                   19   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               3                    3   \n",
       "purchase_Week                               10                   12   \n",
       "purchase_Day                                11                   19   \n",
       "purchase_Dayofweek                           6                    0   \n",
       "purchase_Dayofyear                          70                   78   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                   18   \n",
       "purchase_Minute                             57                   53   \n",
       "purchase_Second                             36                   37   \n",
       "purchase_Elapsed                    1520780256           1521485617   \n",
       "purchased_on_weekend                         1                    0   \n",
       "purchased_on_weekday                         0                    1   \n",
       "month_diff                                  19                   18   \n",
       "\n",
       "                                             2                    3  \\\n",
       "authorized_flag                              1                    1   \n",
       "card_id                        C_ID_415bb3a509      C_ID_415bb3a509   \n",
       "city_id                                    330                   -1   \n",
       "category_1                                   0                    1   \n",
       "installments                                 1                    1   \n",
       "category_3                                   1                    1   \n",
       "merchant_category_id                       507                  661   \n",
       "merchant_id                    M_ID_ad5237ef6b      M_ID_9e84cda3b1   \n",
       "month_lag                                    2                    1   \n",
       "purchase_amount                        87243.7                33705   \n",
       "purchase_date              2018-04-26 14:08:44  2018-03-07 09:43:21   \n",
       "category_2                                   1                    1   \n",
       "state_id                                     9                   -1   \n",
       "subsector_id                                14                    8   \n",
       "purchase_Year                             2018                 2018   \n",
       "purchase_Month                               4                    3   \n",
       "purchase_Week                               17                   10   \n",
       "purchase_Day                                26                    7   \n",
       "purchase_Dayofweek                           3                    2   \n",
       "purchase_Dayofyear                         116                   66   \n",
       "purchase_Is_month_end                    False                False   \n",
       "purchase_Is_month_start                  False                False   \n",
       "purchase_Is_quarter_end                  False                False   \n",
       "purchase_Is_quarter_start                False                False   \n",
       "purchase_Is_year_end                     False                False   \n",
       "purchase_Is_year_start                   False                False   \n",
       "purchase_Hour                               14                    9   \n",
       "purchase_Minute                              8                   43   \n",
       "purchase_Second                             44                   21   \n",
       "purchase_Elapsed                    1524751724           1520415801   \n",
       "purchased_on_weekend                         0                    0   \n",
       "purchased_on_weekday                         1                    1   \n",
       "month_diff                                  18                   19   \n",
       "\n",
       "                                             4  \n",
       "authorized_flag                              1  \n",
       "card_id                        C_ID_ef55cf8d4b  \n",
       "city_id                                     -1  \n",
       "category_1                                   1  \n",
       "installments                                 1  \n",
       "category_3                                   1  \n",
       "merchant_category_id                       166  \n",
       "merchant_id                    M_ID_3c86fa3831  \n",
       "month_lag                                    1  \n",
       "purchase_amount                          39029  \n",
       "purchase_date              2018-03-22 21:07:53  \n",
       "category_2                                   1  \n",
       "state_id                                    -1  \n",
       "subsector_id                                29  \n",
       "purchase_Year                             2018  \n",
       "purchase_Month                               3  \n",
       "purchase_Week                               12  \n",
       "purchase_Day                                22  \n",
       "purchase_Dayofweek                           3  \n",
       "purchase_Dayofyear                          81  \n",
       "purchase_Is_month_end                    False  \n",
       "purchase_Is_month_start                  False  \n",
       "purchase_Is_quarter_end                  False  \n",
       "purchase_Is_quarter_start                False  \n",
       "purchase_Is_year_end                     False  \n",
       "purchase_Is_year_start                   False  \n",
       "purchase_Hour                               21  \n",
       "purchase_Minute                              7  \n",
       "purchase_Second                             53  \n",
       "purchase_Elapsed                    1521752873  \n",
       "purchased_on_weekend                         0  \n",
       "purchased_on_weekday                         1  \n",
       "month_diff                                  18  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rolling aggregate on month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our purchase amounts, lets' calculate `mean`, `sum`, `max`, `peak to peak` aggregates per month grouped by `card_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_per_month(history):\n",
    "    grouped = history.groupby(['card_id', 'month_diff'])['purchase_amount']\n",
    "\n",
    "    agg_func = {\n",
    "            'purchase_amount': ['count', 'sum', 'max', 'mean'],\n",
    "            }\n",
    "\n",
    "    intermediate_group = grouped.agg(agg_func)\n",
    "    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n",
    "    intermediate_group.reset_index(inplace=True)\n",
    "\n",
    "    final_group = intermediate_group.groupby('card_id').agg(['mean', 'sum', np.ptp, 'max'])\n",
    "    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n",
    "    final_group.reset_index(inplace=True)\n",
    "    \n",
    "    return final_group"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time between successive transactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding another new feature - time between successive transactions. The time taken by a customer between his transactions. For this we group our transactions by `card_id` and find the time diff between successive transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def time_diff(df):\n",
    "    df['purchase_date_successive_diff'] = df.sort_values('purchase_date').groupby('card_id')['purchase_date'].diff().dt.total_seconds()\n",
    "    df['purchase_date_successive_diff'].fillna(0, inplace=True)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1330"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans, new_hist_trans = [time_diff(df) for df in dfs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the data frame to disk in feather format. This is our first milestone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans.to_feather('hist_trans_beta')\n",
    "new_hist_trans.to_feather('new_hist_trans_beta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans = feather.read_dataframe('hist_trans_beta')\n",
    "new_hist_trans = feather.read_dataframe('new_hist_trans_beta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = [hist_trans, new_hist_trans]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding more features like holidays in Brazil. Typically on the holidays merchants see a spike in sales and have very good offers. We will also add other features like what was the EMI paid every month which is `purchase_amount` divided by `installments`. Another feature `duration` which is a feature interaction between `purchase_amount` and `month_diff` (months from reference date). We also have `amount_month_ratio` which is also a feature interaction `purchase_amount` between `month_diff` which signifies amount spent by a customer from the reference date. This one adds more weight to the spendings close to the reference date. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def additional_feats(hist_df):\n",
    "    hist_df['price'] = hist_df['purchase_amount'] / hist_df['installments']\n",
    "    #Christmas : December 25 2017\n",
    "    hist_df['Christmas_Day_2017']=(pd.to_datetime('2017-12-25')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #Mothers Day: May 14 2017\n",
    "    hist_df['Mothers_Day_2017']=(pd.to_datetime('2017-06-04')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #fathers day: August 13 2017\n",
    "    hist_df['fathers_day_2017']=(pd.to_datetime('2017-08-13')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #Childrens day: October 12 2017\n",
    "    hist_df['Children_day_2017']=(pd.to_datetime('2017-10-12')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #Valentine's Day : 12th June, 2017\n",
    "    hist_df['Valentine_Day_2017']=(pd.to_datetime('2017-06-12')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #Black Friday : 24th November 2017\n",
    "    hist_df['Black_Friday_2017']=(pd.to_datetime('2017-11-24') - hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    #2018\n",
    "    #Mothers Day: May 13 2018\n",
    "    hist_df['Mothers_Day_2018']=(pd.to_datetime('2018-05-13')-hist_df['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 100 else 0)\n",
    "    hist_df['duration'] = hist_df['purchase_amount']*hist_df['month_diff']\n",
    "    hist_df['amount_month_ratio'] = hist_df['purchase_amount']/hist_df['month_diff']\n",
    "    return hist_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After adding the additional features we copy them to existing dataframes to save space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans, new_hist_trans = [additional_feats(df) for df in dfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((29112361, 52), (1963031, 52))"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans.shape, new_hist_trans.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latest & First 5 col aggregates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen in previous additional feature `amount_month_ratio` which adds more weight to purchases close to reference date, let's add features like sum/max of last 5 purchases etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def head_sum(x):\n",
    "    return x.head().sum()\n",
    "\n",
    "def head_max(x):\n",
    "    return x.head().max()\n",
    "\n",
    "def tail_sum(x):\n",
    "    return x.tail().sum()\n",
    "\n",
    "def tail_max(x):\n",
    "    return x.tail().max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will sort all our transactions by purchase date chronologically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 864 ms, sys: 681 ms, total: 1.54 s\n",
      "Wall time: 358 ms\n"
     ]
    }
   ],
   "source": [
    "%time new_hist_trans = new_hist_trans.sort_values('purchase_date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.94 s, sys: 3.11 s, total: 8.05 s\n",
      "Wall time: 6.06 s\n"
     ]
    }
   ],
   "source": [
    "%time hist_trans = hist_trans.sort_values('purchase_date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans = new_hist_trans.reset_index().drop('index', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans.drop('index', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans.drop('level_0', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merge merchants df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the merchants data! We have already pre-processed and added some features to the merchants dataframe. Let's load it in and merge with our transactions data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "merchants = feather.read_dataframe('merchants_df')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans = new_hist_trans.merge(merchants, on='merchant_id', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans = hist_trans.merge(merchants, on='merchant_id', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'hist_trans' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-6e8dfbee3be1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mhist_trans\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_hist_trans\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'hist_trans' is not defined"
     ]
    }
   ],
   "source": [
    "hist_trans.shape, new_hist_trans.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans.to_feather('hist_trans_beta')\n",
    "new_hist_trans.to_feather('new_hist_trans_beta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai/lib/python3.6/site-packages/pyarrow/pandas_compat.py:752: FutureWarning: .labels was deprecated in version 0.24.0. Use .codes instead.\n",
      "  labels, = index.labels\n"
     ]
    }
   ],
   "source": [
    "hist_trans = feather.read_dataframe('hist_trans_beta')\n",
    "new_hist_trans = feather.read_dataframe('new_hist_trans_beta')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After merging the merchants data to our transactions data, let's see the final list of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>authorized_flag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_7c628841cb</td>\n",
       "      <td>C_ID_25d399500c</td>\n",
       "      <td>C_ID_e3542c52f1</td>\n",
       "      <td>C_ID_fabd47ca44</td>\n",
       "      <td>C_ID_6f9a771d17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city_id</th>\n",
       "      <td>69</td>\n",
       "      <td>19</td>\n",
       "      <td>199</td>\n",
       "      <td>69</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>installments</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_category_id</th>\n",
       "      <td>80</td>\n",
       "      <td>307</td>\n",
       "      <td>307</td>\n",
       "      <td>80</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_id</th>\n",
       "      <td>M_ID_c03b62d83d</td>\n",
       "      <td>M_ID_2445d76702</td>\n",
       "      <td>M_ID_b16ae63c45</td>\n",
       "      <td>M_ID_b35d0757d1</td>\n",
       "      <td>M_ID_b6b9b8ed67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_lag</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount</th>\n",
       "      <td>46.4</td>\n",
       "      <td>174.01</td>\n",
       "      <td>50</td>\n",
       "      <td>5.5</td>\n",
       "      <td>550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date</th>\n",
       "      <td>2017-03-01 03:24:51</td>\n",
       "      <td>2017-03-01 11:01:06</td>\n",
       "      <td>2017-03-01 11:27:39</td>\n",
       "      <td>2017-03-01 11:55:11</td>\n",
       "      <td>2017-03-01 12:37:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state_id</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subsector_id</th>\n",
       "      <td>37</td>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>37</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Year</th>\n",
       "      <td>2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Month</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Week</th>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Day</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofweek</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Dayofyear</th>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_month_start</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_quarter_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_end</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Is_year_start</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Hour</th>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Minute</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>55</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Second</th>\n",
       "      <td>51</td>\n",
       "      <td>6</td>\n",
       "      <td>39</td>\n",
       "      <td>11</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_Elapsed</th>\n",
       "      <td>1488338691</td>\n",
       "      <td>1488366066</td>\n",
       "      <td>1488367659</td>\n",
       "      <td>1488369311</td>\n",
       "      <td>1488371846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekend</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchased_on_weekday</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_diff</th>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_date_successive_diff</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>price</th>\n",
       "      <td>46.4</td>\n",
       "      <td>174.01</td>\n",
       "      <td>inf</td>\n",
       "      <td>inf</td>\n",
       "      <td>550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Christmas_Day_2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mothers_Day_2017</th>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fathers_day_2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Children_day_2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Valentine_Day_2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black_Friday_2017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mothers_Day_2018</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration</th>\n",
       "      <td>1113.6</td>\n",
       "      <td>4176.24</td>\n",
       "      <td>1200</td>\n",
       "      <td>132</td>\n",
       "      <td>13200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amount_month_ratio</th>\n",
       "      <td>1.93333</td>\n",
       "      <td>7.25042</td>\n",
       "      <td>2.08333</td>\n",
       "      <td>0.229167</td>\n",
       "      <td>22.9167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_mean</th>\n",
       "      <td>130.934</td>\n",
       "      <td>130.934</td>\n",
       "      <td>125.997</td>\n",
       "      <td>130.934</td>\n",
       "      <td>127.287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_min</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_max</th>\n",
       "      <td>175626</td>\n",
       "      <td>175626</td>\n",
       "      <td>40000</td>\n",
       "      <td>175626</td>\n",
       "      <td>22610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_sum</th>\n",
       "      <td>1.53191e+08</td>\n",
       "      <td>1.53191e+08</td>\n",
       "      <td>2.25018e+07</td>\n",
       "      <td>1.53191e+08</td>\n",
       "      <td>8.35804e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_mean</th>\n",
       "      <td>93.4493</td>\n",
       "      <td>93.4493</td>\n",
       "      <td>102.429</td>\n",
       "      <td>102.429</td>\n",
       "      <td>93.4493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_min</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_max</th>\n",
       "      <td>48713.6</td>\n",
       "      <td>48713.6</td>\n",
       "      <td>65056</td>\n",
       "      <td>65056</td>\n",
       "      <td>48713.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_sum</th>\n",
       "      <td>7.81402e+07</td>\n",
       "      <td>7.81402e+07</td>\n",
       "      <td>1.00192e+08</td>\n",
       "      <td>1.00192e+08</td>\n",
       "      <td>7.81402e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_address_id</th>\n",
       "      <td>M_ID_c03b62d83d_80_37_69_9_1</td>\n",
       "      <td>M_ID_2445d76702_307_19_0_9_1</td>\n",
       "      <td>M_ID_b16ae63c45_307_19_0_14_4</td>\n",
       "      <td>M_ID_b35d0757d1_80_37_69_9_1</td>\n",
       "      <td>M_ID_b6b9b8ed67_178_29_0_24_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>numerical_range</th>\n",
       "      <td>173</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>merchant_rating</th>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                          0  \\\n",
       "authorized_flag                                           1   \n",
       "card_id                                     C_ID_7c628841cb   \n",
       "city_id                                                  69   \n",
       "category_1                                                0   \n",
       "installments                                              1   \n",
       "category_3                                                1   \n",
       "merchant_category_id                                     80   \n",
       "merchant_id                                 M_ID_c03b62d83d   \n",
       "month_lag                                                 1   \n",
       "purchase_amount                                        46.4   \n",
       "purchase_date                           2017-03-01 03:24:51   \n",
       "category_2                                                1   \n",
       "state_id                                                  9   \n",
       "subsector_id                                             37   \n",
       "purchase_Year                                          2017   \n",
       "purchase_Month                                            3   \n",
       "purchase_Week                                             9   \n",
       "purchase_Day                                              1   \n",
       "purchase_Dayofweek                                        2   \n",
       "purchase_Dayofyear                                       60   \n",
       "purchase_Is_month_end                                 False   \n",
       "purchase_Is_month_start                                True   \n",
       "purchase_Is_quarter_end                               False   \n",
       "purchase_Is_quarter_start                             False   \n",
       "purchase_Is_year_end                                  False   \n",
       "purchase_Is_year_start                                False   \n",
       "purchase_Hour                                             3   \n",
       "purchase_Minute                                          24   \n",
       "purchase_Second                                          51   \n",
       "purchase_Elapsed                                 1488338691   \n",
       "purchased_on_weekend                                      0   \n",
       "purchased_on_weekday                                      1   \n",
       "month_diff                                               24   \n",
       "purchase_date_successive_diff                             0   \n",
       "price                                                  46.4   \n",
       "Christmas_Day_2017                                        0   \n",
       "Mothers_Day_2017                                         94   \n",
       "fathers_day_2017                                          0   \n",
       "Children_day_2017                                         0   \n",
       "Valentine_Day_2017                                        0   \n",
       "Black_Friday_2017                                         0   \n",
       "Mothers_Day_2018                                          0   \n",
       "duration                                             1113.6   \n",
       "amount_month_ratio                                  1.93333   \n",
       "category_2_mean                                     130.934   \n",
       "category_2_min                                         0.01   \n",
       "category_2_max                                       175626   \n",
       "category_2_sum                                  1.53191e+08   \n",
       "category_3_mean                                     93.4493   \n",
       "category_3_min                                         0.01   \n",
       "category_3_max                                      48713.6   \n",
       "category_3_sum                                  7.81402e+07   \n",
       "category_4                                                1   \n",
       "merchant_address_id            M_ID_c03b62d83d_80_37_69_9_1   \n",
       "numerical_range                                         173   \n",
       "merchant_rating                                          10   \n",
       "\n",
       "                                                          1  \\\n",
       "authorized_flag                                           1   \n",
       "card_id                                     C_ID_25d399500c   \n",
       "city_id                                                  19   \n",
       "category_1                                                0   \n",
       "installments                                              1   \n",
       "category_3                                                1   \n",
       "merchant_category_id                                    307   \n",
       "merchant_id                                 M_ID_2445d76702   \n",
       "month_lag                                                 1   \n",
       "purchase_amount                                      174.01   \n",
       "purchase_date                           2017-03-01 11:01:06   \n",
       "category_2                                                1   \n",
       "state_id                                                  9   \n",
       "subsector_id                                             19   \n",
       "purchase_Year                                          2017   \n",
       "purchase_Month                                            3   \n",
       "purchase_Week                                             9   \n",
       "purchase_Day                                              1   \n",
       "purchase_Dayofweek                                        2   \n",
       "purchase_Dayofyear                                       60   \n",
       "purchase_Is_month_end                                 False   \n",
       "purchase_Is_month_start                                True   \n",
       "purchase_Is_quarter_end                               False   \n",
       "purchase_Is_quarter_start                             False   \n",
       "purchase_Is_year_end                                  False   \n",
       "purchase_Is_year_start                                False   \n",
       "purchase_Hour                                            11   \n",
       "purchase_Minute                                           1   \n",
       "purchase_Second                                           6   \n",
       "purchase_Elapsed                                 1488366066   \n",
       "purchased_on_weekend                                      0   \n",
       "purchased_on_weekday                                      1   \n",
       "month_diff                                               24   \n",
       "purchase_date_successive_diff                             0   \n",
       "price                                                174.01   \n",
       "Christmas_Day_2017                                        0   \n",
       "Mothers_Day_2017                                         94   \n",
       "fathers_day_2017                                          0   \n",
       "Children_day_2017                                         0   \n",
       "Valentine_Day_2017                                        0   \n",
       "Black_Friday_2017                                         0   \n",
       "Mothers_Day_2018                                          0   \n",
       "duration                                            4176.24   \n",
       "amount_month_ratio                                  7.25042   \n",
       "category_2_mean                                     130.934   \n",
       "category_2_min                                         0.01   \n",
       "category_2_max                                       175626   \n",
       "category_2_sum                                  1.53191e+08   \n",
       "category_3_mean                                     93.4493   \n",
       "category_3_min                                         0.01   \n",
       "category_3_max                                      48713.6   \n",
       "category_3_sum                                  7.81402e+07   \n",
       "category_4                                                0   \n",
       "merchant_address_id            M_ID_2445d76702_307_19_0_9_1   \n",
       "numerical_range                                          22   \n",
       "merchant_rating                                          12   \n",
       "\n",
       "                                                           2  \\\n",
       "authorized_flag                                            1   \n",
       "card_id                                      C_ID_e3542c52f1   \n",
       "city_id                                                  199   \n",
       "category_1                                                 0   \n",
       "installments                                               0   \n",
       "category_3                                                 0   \n",
       "merchant_category_id                                     307   \n",
       "merchant_id                                  M_ID_b16ae63c45   \n",
       "month_lag                                                  1   \n",
       "purchase_amount                                           50   \n",
       "purchase_date                            2017-03-01 11:27:39   \n",
       "category_2                                                 4   \n",
       "state_id                                                  14   \n",
       "subsector_id                                              19   \n",
       "purchase_Year                                           2017   \n",
       "purchase_Month                                             3   \n",
       "purchase_Week                                              9   \n",
       "purchase_Day                                               1   \n",
       "purchase_Dayofweek                                         2   \n",
       "purchase_Dayofyear                                        60   \n",
       "purchase_Is_month_end                                  False   \n",
       "purchase_Is_month_start                                 True   \n",
       "purchase_Is_quarter_end                                False   \n",
       "purchase_Is_quarter_start                              False   \n",
       "purchase_Is_year_end                                   False   \n",
       "purchase_Is_year_start                                 False   \n",
       "purchase_Hour                                             11   \n",
       "purchase_Minute                                           27   \n",
       "purchase_Second                                           39   \n",
       "purchase_Elapsed                                  1488367659   \n",
       "purchased_on_weekend                                       0   \n",
       "purchased_on_weekday                                       1   \n",
       "month_diff                                                24   \n",
       "purchase_date_successive_diff                              0   \n",
       "price                                                    inf   \n",
       "Christmas_Day_2017                                         0   \n",
       "Mothers_Day_2017                                          94   \n",
       "fathers_day_2017                                           0   \n",
       "Children_day_2017                                          0   \n",
       "Valentine_Day_2017                                         0   \n",
       "Black_Friday_2017                                          0   \n",
       "Mothers_Day_2018                                           0   \n",
       "duration                                                1200   \n",
       "amount_month_ratio                                   2.08333   \n",
       "category_2_mean                                      125.997   \n",
       "category_2_min                                          0.01   \n",
       "category_2_max                                         40000   \n",
       "category_2_sum                                   2.25018e+07   \n",
       "category_3_mean                                      102.429   \n",
       "category_3_min                                          0.01   \n",
       "category_3_max                                         65056   \n",
       "category_3_sum                                   1.00192e+08   \n",
       "category_4                                                 0   \n",
       "merchant_address_id            M_ID_b16ae63c45_307_19_0_14_4   \n",
       "numerical_range                                            0   \n",
       "merchant_rating                                           16   \n",
       "\n",
       "                                                          3  \\\n",
       "authorized_flag                                           1   \n",
       "card_id                                     C_ID_fabd47ca44   \n",
       "city_id                                                  69   \n",
       "category_1                                                0   \n",
       "installments                                              0   \n",
       "category_3                                                0   \n",
       "merchant_category_id                                     80   \n",
       "merchant_id                                 M_ID_b35d0757d1   \n",
       "month_lag                                                 1   \n",
       "purchase_amount                                         5.5   \n",
       "purchase_date                           2017-03-01 11:55:11   \n",
       "category_2                                                1   \n",
       "state_id                                                  9   \n",
       "subsector_id                                             37   \n",
       "purchase_Year                                          2017   \n",
       "purchase_Month                                            3   \n",
       "purchase_Week                                             9   \n",
       "purchase_Day                                              1   \n",
       "purchase_Dayofweek                                        2   \n",
       "purchase_Dayofyear                                       60   \n",
       "purchase_Is_month_end                                 False   \n",
       "purchase_Is_month_start                                True   \n",
       "purchase_Is_quarter_end                               False   \n",
       "purchase_Is_quarter_start                             False   \n",
       "purchase_Is_year_end                                  False   \n",
       "purchase_Is_year_start                                False   \n",
       "purchase_Hour                                            11   \n",
       "purchase_Minute                                          55   \n",
       "purchase_Second                                          11   \n",
       "purchase_Elapsed                                 1488369311   \n",
       "purchased_on_weekend                                      0   \n",
       "purchased_on_weekday                                      1   \n",
       "month_diff                                               24   \n",
       "purchase_date_successive_diff                             0   \n",
       "price                                                   inf   \n",
       "Christmas_Day_2017                                        0   \n",
       "Mothers_Day_2017                                         94   \n",
       "fathers_day_2017                                          0   \n",
       "Children_day_2017                                         0   \n",
       "Valentine_Day_2017                                        0   \n",
       "Black_Friday_2017                                         0   \n",
       "Mothers_Day_2018                                          0   \n",
       "duration                                                132   \n",
       "amount_month_ratio                                 0.229167   \n",
       "category_2_mean                                     130.934   \n",
       "category_2_min                                         0.01   \n",
       "category_2_max                                       175626   \n",
       "category_2_sum                                  1.53191e+08   \n",
       "category_3_mean                                     102.429   \n",
       "category_3_min                                         0.01   \n",
       "category_3_max                                        65056   \n",
       "category_3_sum                                  1.00192e+08   \n",
       "category_4                                                1   \n",
       "merchant_address_id            M_ID_b35d0757d1_80_37_69_9_1   \n",
       "numerical_range                                           0   \n",
       "merchant_rating                                          15   \n",
       "\n",
       "                                                           4  \n",
       "authorized_flag                                            1  \n",
       "card_id                                      C_ID_6f9a771d17  \n",
       "city_id                                                   96  \n",
       "category_1                                                 0  \n",
       "installments                                               1  \n",
       "category_3                                                 1  \n",
       "merchant_category_id                                     178  \n",
       "merchant_id                                  M_ID_b6b9b8ed67  \n",
       "month_lag                                                  1  \n",
       "purchase_amount                                          550  \n",
       "purchase_date                            2017-03-01 12:37:26  \n",
       "category_2                                                 2  \n",
       "state_id                                                  24  \n",
       "subsector_id                                              29  \n",
       "purchase_Year                                           2017  \n",
       "purchase_Month                                             3  \n",
       "purchase_Week                                              9  \n",
       "purchase_Day                                               1  \n",
       "purchase_Dayofweek                                         2  \n",
       "purchase_Dayofyear                                        60  \n",
       "purchase_Is_month_end                                  False  \n",
       "purchase_Is_month_start                                 True  \n",
       "purchase_Is_quarter_end                                False  \n",
       "purchase_Is_quarter_start                              False  \n",
       "purchase_Is_year_end                                   False  \n",
       "purchase_Is_year_start                                 False  \n",
       "purchase_Hour                                             12  \n",
       "purchase_Minute                                           37  \n",
       "purchase_Second                                           26  \n",
       "purchase_Elapsed                                  1488371846  \n",
       "purchased_on_weekend                                       0  \n",
       "purchased_on_weekday                                       1  \n",
       "month_diff                                                24  \n",
       "purchase_date_successive_diff                              0  \n",
       "price                                                    550  \n",
       "Christmas_Day_2017                                         0  \n",
       "Mothers_Day_2017                                          94  \n",
       "fathers_day_2017                                           0  \n",
       "Children_day_2017                                          0  \n",
       "Valentine_Day_2017                                         0  \n",
       "Black_Friday_2017                                          0  \n",
       "Mothers_Day_2018                                           0  \n",
       "duration                                               13200  \n",
       "amount_month_ratio                                   22.9167  \n",
       "category_2_mean                                      127.287  \n",
       "category_2_min                                          0.01  \n",
       "category_2_max                                         22610  \n",
       "category_2_sum                                   8.35804e+06  \n",
       "category_3_mean                                      93.4493  \n",
       "category_3_min                                          0.01  \n",
       "category_3_max                                       48713.6  \n",
       "category_3_sum                                   7.81402e+07  \n",
       "category_4                                                 0  \n",
       "merchant_address_id            M_ID_b6b9b8ed67_178_29_0_24_2  \n",
       "numerical_range                                          737  \n",
       "merchant_rating                                           12  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the features we compute earlier for merchants like `merchant_address_id`, `numerical_range` & `merchant_rating` appended to our transactions data. Now our transactions data has 56 features in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((29112368, 56), (1963031, 56))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans.shape, new_hist_trans.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregate by card_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's replace the `0` in our transactions data with `0.0001` to avoid numerical inconsistencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans.loc[hist_trans['purchase_amount'] == 0, 'purchase_amount'] = 0.0001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the aggregates on the transactions data grouped by `card_id` as we have to calculate the loyalty score for each `card_id`. The following code is pruned after some experimentation and removal of aggregates which didn't have so much feature importance. Most of the aggregates are self-explanatory. We have some special aggregates grouped by category & we calculate `purchase_amount` aggregates on the grouped by category data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_hist_trans(df):\n",
    "    unique_cols = ['subsector_id', 'merchant_id', 'merchant_category_id', 'merchant_address_id']\n",
    "\n",
    "    col_seas = ['purchase_Month','purchase_Week', 'purchase_Dayofweek', 'purchase_Day', 'purchase_Hour',\n",
    "               'merchant_rating']\n",
    "\n",
    "    aggs = {}\n",
    "    for c in unique_cols:\n",
    "        aggs[c] = ['nunique'] \n",
    "    for c in col_seas:\n",
    "        aggs[c] = ['nunique', 'mean', 'min', 'max']\n",
    "   \n",
    "    aggs['purchase_amount'] = [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                               ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                               ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                               ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                               ('tail_sum', tail_sum), ('tail_max', tail_max), ('gmean', scipy.stats.gmean ),\n",
    "                                ('hmean', scipy.stats.hmean)]\n",
    "    aggs['installments'] = [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                               ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                               ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                               ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                               ('tail_sum', tail_sum), ('tail_max', tail_max)]\n",
    "    aggs['purchase_date'] = ['max','min']\n",
    "    aggs['month_lag'] = ['max','min','mean','var','skew']\n",
    "    aggs['month_diff'] = ['max','min','mean','var','skew']\n",
    "    aggs['authorized_flag'] = ['mean']\n",
    "    aggs['purchased_on_weekend'] = ['mean'] # overwrite\n",
    "    aggs['purchase_Dayofweek'] = ['mean'] # overwrite\n",
    "    aggs['purchase_Day'] = ['nunique', 'mean', 'min'] # overwrite\n",
    "    aggs['category_1'] = ['mean']\n",
    "    aggs['category_2'] = ['mean']\n",
    "    aggs['category_3'] = ['mean']\n",
    "    aggs['category_4'] = ['mean']\n",
    "    aggs['numerical_range'] = ['mean', 'min', 'max', 'skew']\n",
    "    aggs['card_id'] = ['size','count']\n",
    "    aggs['price'] = ['sum','mean','max','min','var', 'skew']\n",
    "    aggs['Christmas_Day_2017'] = ['mean']\n",
    "    aggs['Mothers_Day_2017'] = ['mean']\n",
    "    aggs['fathers_day_2017'] = ['mean']\n",
    "    aggs['Children_day_2017'] = ['mean']\n",
    "    aggs['Valentine_Day_2017'] = ['mean']\n",
    "    aggs['Black_Friday_2017'] = ['mean']\n",
    "    aggs['Mothers_Day_2018'] = ['mean']\n",
    "    aggs['duration']=['mean','min','max','var','skew']\n",
    "    aggs['amount_month_ratio']=['mean','min','max','var','skew']\n",
    "    \n",
    "    #exta\n",
    "    aggs['purchase_date_successive_diff'] = ['mean']\n",
    "\n",
    "#     aggs['purchase_date_successive_diff'] = ['mean', 'median', 'max', 'min', 'var', 'skew']\n",
    "    for col in ['category_2','category_3']:\n",
    "        df[col+'_mean'] = df.groupby([col])['purchase_amount'].transform('mean')\n",
    "        df[col+'_min'] = df.groupby([col])['purchase_amount'].transform('min')\n",
    "        df[col+'_max'] = df.groupby([col])['purchase_amount'].transform('max')\n",
    "        df[col+'_sum'] = df.groupby([col])['purchase_amount'].transform('sum')\n",
    "        aggs[col+'_mean'] = ['mean']\n",
    "\n",
    "    new_df = df.groupby(['card_id']).agg(aggs)\n",
    "    new_df.columns = ['_'.join(col).strip() for col in new_df.columns.values]\n",
    "    new_df.reset_index(inplace=True)\n",
    "    other_df = (df.groupby('card_id')\n",
    "          .size()\n",
    "          .reset_index(name='transactions_count'))\n",
    "    \n",
    "    new_df = pd.merge(other_df, new_df, on='card_id', how='left')\n",
    "\n",
    "    new_df['purchase_date_diff'] = (new_df['purchase_date_max'] - new_df['purchase_date_min']).dt.days\n",
    "    new_df['purchase_date_average'] = new_df['purchase_date_diff']/new_df['card_id_size']\n",
    "    new_df['purchase_date_uptonow'] = (datetime.datetime.today() - new_df['purchase_date_max']).dt.days\n",
    "    new_df['purchase_date_uptomin'] = (datetime.datetime.today() - new_df['purchase_date_min']).dt.days\n",
    "    return new_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not calculating the same aggregates for both historical & new transactional data, as some of the features add little value and will be removed during feature selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_new_trans(df):\n",
    "    unique_cols = ['subsector_id', 'merchant_id', 'merchant_category_id', 'merchant_address_id']\n",
    "\n",
    "    col_seas = ['purchase_Month', 'purchase_Week', 'purchase_Dayofweek','purchase_Day', 'purchase_Hour', \n",
    "                'merchant_rating']\n",
    "\n",
    "    aggs = {}\n",
    "    for c in unique_cols:\n",
    "        aggs[c] = ['nunique'] \n",
    "    for c in col_seas:\n",
    "        aggs[c] = ['nunique', 'mean', 'min', 'max']\n",
    "        \n",
    "    aggs['purchase_amount'] = [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                               ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                               ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                               ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                               ('tail_sum', tail_sum), ('tail_max', tail_max), ('gmean', scipy.stats.gmean ),\n",
    "                                ('hmean', scipy.stats.hmean)]\n",
    "    aggs['installments'] = [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                               ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                               ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                               ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                               ('tail_sum', tail_sum), ('tail_max', tail_max)]\n",
    "    aggs['purchase_date'] = ['max','min']\n",
    "    aggs['month_lag'] = ['max','min','mean','var','skew']\n",
    "    aggs['month_diff'] = ['mean','var','skew']\n",
    "    aggs['purchased_on_weekend'] = ['mean']\n",
    "    aggs['purchase_Month'] = ['mean', 'min', 'max']\n",
    "    aggs['purchase_Dayofweek'] = ['mean', 'min', 'max']\n",
    "    aggs['category_1'] = ['mean']\n",
    "    aggs['category_2'] = ['mean']\n",
    "    aggs['category_3'] = ['mean']\n",
    "    aggs['category_4'] = ['mean']\n",
    "    aggs['numerical_range'] = ['mean', 'min', 'max', 'skew']\n",
    "    aggs['card_id'] = ['size','count']\n",
    "    aggs['price'] = ['mean','max','min','var']\n",
    "    aggs['Christmas_Day_2017'] = ['mean']\n",
    "    aggs['Children_day_2017'] = ['mean']\n",
    "    aggs['Black_Friday_2017'] = ['mean']\n",
    "    aggs['Mothers_Day_2018'] = ['mean']\n",
    "    aggs['duration']=['mean','min','max','var','skew']\n",
    "    aggs['amount_month_ratio']=['mean','min','max','var','skew']\n",
    "    \n",
    "    #extra\n",
    "    aggs['purchase_date_successive_diff'] = ['mean']\n",
    "    \n",
    "#     aggs['purchase_date_successive_diff'] = ['mean', 'median', 'max', 'min', 'var', 'skew']\n",
    "    for col in ['category_2','category_3']:\n",
    "        df[col+'_mean'] = df.groupby([col])['purchase_amount'].transform('mean')\n",
    "        df[col+'_min'] = df.groupby([col])['purchase_amount'].transform('min')\n",
    "        df[col+'_max'] = df.groupby([col])['purchase_amount'].transform('max')\n",
    "        df[col+'_sum'] = df.groupby([col])['purchase_amount'].transform('sum')\n",
    "        aggs[col+'_mean'] = ['mean']\n",
    "\n",
    "    new_df = df.groupby(['card_id']).agg(aggs)\n",
    "    new_df.columns = ['_'.join(col).strip() for col in new_df.columns.values]\n",
    "    new_df.reset_index(inplace=True)\n",
    "    other_df = (df.groupby('card_id')\n",
    "          .size()\n",
    "          .reset_index(name='transactions_count'))\n",
    "    \n",
    "    new_df = pd.merge(other_df, new_df, on='card_id', how='left')\n",
    "\n",
    "    new_df['purchase_date_diff'] = (new_df['purchase_date_max'] - new_df['purchase_date_min']).dt.days\n",
    "    new_df['purchase_date_average'] = new_df['purchase_date_diff']/new_df['card_id_size']\n",
    "    new_df['purchase_date_uptonow'] = (datetime.datetime.today() - new_df['purchase_date_max']).dt.days\n",
    "    new_df['purchase_date_uptomin'] = (datetime.datetime.today() - new_df['purchase_date_min']).dt.days\n",
    "    return new_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gc\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 14min 28s, sys: 53.6 s, total: 15min 22s\n",
      "Wall time: 14min 13s\n"
     ]
    }
   ],
   "source": [
    "%time hist_trans_agg = aggregate_hist_trans(hist_trans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai/lib/python3.6/site-packages/numpy/lib/function_base.py:3652: RuntimeWarning: Invalid value encountered in percentile\n",
      "  interpolation=interpolation)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8min 44s, sys: 1.25 s, total: 8min 46s\n",
      "Wall time: 8min 16s\n"
     ]
    }
   ],
   "source": [
    "%time new_hist_trans_agg = aggregate_new_trans(new_hist_trans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add exta interpreted columns on aggregates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding extra features on top of the aggregates. I know it's overwhelming but these are important. `card_id_size` is the number of transactions done by the card. Most of the features are self-explanatory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_extra_cols_on_agg(df):\n",
    "    df['inverse_avg_transactions_per_day'] = df['purchase_date_diff']/df['card_id_size']\n",
    "    df['repurchase_merchant_rate'] = df['transactions_count']/df['merchant_id_nunique']\n",
    "    df['merchant_category_repurchase'] = df['merchant_category_id_nunique']/df['merchant_id_nunique']\n",
    "    df['avg_spend_per_merchant'] = df['purchase_amount_sum']/df['merchant_id_nunique']\n",
    "    df['avg_trans_per_merchant'] = df['transactions_count']/df['merchant_id_nunique']\n",
    "    df['avg_spend_per_transaction'] = df['purchase_amount_sum']/df['transactions_count']\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "[hist_trans_agg, new_hist_trans_agg] = [add_extra_cols_on_agg(df) for df in [hist_trans_agg, \n",
    "                                                                             new_hist_trans_agg]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_agg.to_feather('hist_trans_agg_beta')\n",
    "new_hist_trans_agg.to_feather('new_hist_trans_agg_beta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_agg = feather.read_dataframe('hist_trans_agg_beta')\n",
    "new_hist_trans_agg = feather.read_dataframe('new_hist_trans_agg_beta')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have 114, 108 features for old & new transactional aggregates data. We will adding some more features on top of this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((325540, 114), (290001, 108))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans_agg.shape, new_hist_trans_agg.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregate on categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some more aggregates on the categories. Pivot tables are a common approach when calculating aggregates grouped by more than one conditions. Here we are delving deep in to the finer categorical spends aggregates. Our `category_1` had a cardinality of 2 whereas `category_2` had a cardinality of 6. We will calculate aggregates grouped by each of those possible values for the category. We will later see that these aggregates don't add much feature importance, but this was a good exercise for me to flex my pandas skills :D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def agg_on_cat(df, category, feature):\n",
    "    temp_df = df.pivot_table(index='card_id', columns=category, aggfunc={feature: ['sum', 'mean']})\n",
    "    cols = [category + '_{0[2]}_{0[0]}_{0[1]}'.format(col) for col in temp_df.columns.tolist()]\n",
    "    temp_df.columns = cols\n",
    "    return temp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cat_agg(df):\n",
    "    agg_df = agg_on_cat(df, 'category_1', 'purchase_amount')\n",
    "    agg_df = pd.merge(agg_df, agg_on_cat(df, 'category_2', 'purchase_amount'), on='card_id', how='left')\n",
    "    agg_df = pd.merge(agg_df, agg_on_cat(df, 'category_3', 'purchase_amount'), on='card_id', how='left')\n",
    "    agg_df = pd.merge(agg_df, agg_on_cat(df, 'authorized_flag', 'purchase_amount'), on='card_id', how='left')\n",
    "    return agg_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 30.4 s, sys: 9.67 s, total: 40.1 s\n",
      "Wall time: 27.9 s\n"
     ]
    }
   ],
   "source": [
    "%time hist_trans_agg_cat, new_hist_trans_agg_cat = [get_cat_agg(df) for df in [hist_trans, new_hist_trans]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((325540, 24), (290001, 22))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans_agg_cat.shape, new_hist_trans_agg_cat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at our newly calculated aggregates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>card_id</th>\n",
       "      <th>C_ID_00007093c1</th>\n",
       "      <th>C_ID_0001238066</th>\n",
       "      <th>C_ID_0001506ef0</th>\n",
       "      <th>C_ID_0001793786</th>\n",
       "      <th>C_ID_000183fdda</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>category_1_0_purchase_amount_mean</th>\n",
       "      <td>-0.664262</td>\n",
       "      <td>-0.564558</td>\n",
       "      <td>-0.723677</td>\n",
       "      <td>-0.007407</td>\n",
       "      <td>-0.599162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1_1_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.650332</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1_0_purchase_amount_sum</th>\n",
       "      <td>-1.328524</td>\n",
       "      <td>-13.549391</td>\n",
       "      <td>-1.447354</td>\n",
       "      <td>-0.229620</td>\n",
       "      <td>-6.590778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_1_1_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.300665</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_1.0_purchase_amount_mean</th>\n",
       "      <td>-0.656749</td>\n",
       "      <td>-0.580966</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.139747</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_2.0_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.344766</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_3.0_purchase_amount_mean</th>\n",
       "      <td>-0.671775</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.723677</td>\n",
       "      <td>0.102887</td>\n",
       "      <td>-0.599162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_4.0_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_5.0_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.495945</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.361628</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_1.0_purchase_amount_sum</th>\n",
       "      <td>-0.656749</td>\n",
       "      <td>-13.362220</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.375707</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_2.0_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-2.758131</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_3.0_purchase_amount_sum</th>\n",
       "      <td>-0.671775</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.447354</td>\n",
       "      <td>0.514433</td>\n",
       "      <td>-6.590778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_4.0_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_2_5.0_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.487835</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.361628</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_0_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.152008</td>\n",
       "      <td>-0.723677</td>\n",
       "      <td>-0.007407</td>\n",
       "      <td>-0.107680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_1_purchase_amount_mean</th>\n",
       "      <td>-0.664262</td>\n",
       "      <td>-0.625781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.696173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_2_purchase_amount_mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.389160</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.576515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_0_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.152008</td>\n",
       "      <td>-1.447354</td>\n",
       "      <td>-0.229620</td>\n",
       "      <td>-0.107680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_1_purchase_amount_sum</th>\n",
       "      <td>-1.328524</td>\n",
       "      <td>-13.141406</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-4.177040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category_3_2_purchase_amount_sum</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.556641</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-2.306059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>authorized_flag_1_purchase_amount_mean</th>\n",
       "      <td>-0.664262</td>\n",
       "      <td>-0.571156</td>\n",
       "      <td>-0.723677</td>\n",
       "      <td>-0.007407</td>\n",
       "      <td>-0.599162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>authorized_flag_1_purchase_amount_sum</th>\n",
       "      <td>-1.328524</td>\n",
       "      <td>-14.850055</td>\n",
       "      <td>-1.447354</td>\n",
       "      <td>-0.229620</td>\n",
       "      <td>-6.590778</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "card_id                                 C_ID_00007093c1  C_ID_0001238066  \\\n",
       "category_1_0_purchase_amount_mean             -0.664262        -0.564558   \n",
       "category_1_1_purchase_amount_mean                   NaN        -0.650332   \n",
       "category_1_0_purchase_amount_sum              -1.328524       -13.549391   \n",
       "category_1_1_purchase_amount_sum                    NaN        -1.300665   \n",
       "category_2_1.0_purchase_amount_mean           -0.656749        -0.580966   \n",
       "category_2_2.0_purchase_amount_mean                 NaN              NaN   \n",
       "category_2_3.0_purchase_amount_mean           -0.671775              NaN   \n",
       "category_2_4.0_purchase_amount_mean                 NaN              NaN   \n",
       "category_2_5.0_purchase_amount_mean                 NaN        -0.495945   \n",
       "category_2_1.0_purchase_amount_sum            -0.656749       -13.362220   \n",
       "category_2_2.0_purchase_amount_sum                  NaN              NaN   \n",
       "category_2_3.0_purchase_amount_sum            -0.671775              NaN   \n",
       "category_2_4.0_purchase_amount_sum                  NaN              NaN   \n",
       "category_2_5.0_purchase_amount_sum                  NaN        -1.487835   \n",
       "category_3_0_purchase_amount_mean                   NaN        -0.152008   \n",
       "category_3_1_purchase_amount_mean             -0.664262        -0.625781   \n",
       "category_3_2_purchase_amount_mean                   NaN        -0.389160   \n",
       "category_3_0_purchase_amount_sum                    NaN        -0.152008   \n",
       "category_3_1_purchase_amount_sum              -1.328524       -13.141406   \n",
       "category_3_2_purchase_amount_sum                    NaN        -1.556641   \n",
       "authorized_flag_1_purchase_amount_mean        -0.664262        -0.571156   \n",
       "authorized_flag_1_purchase_amount_sum         -1.328524       -14.850055   \n",
       "\n",
       "card_id                                 C_ID_0001506ef0  C_ID_0001793786  \\\n",
       "category_1_0_purchase_amount_mean             -0.723677        -0.007407   \n",
       "category_1_1_purchase_amount_mean                   NaN              NaN   \n",
       "category_1_0_purchase_amount_sum              -1.447354        -0.229620   \n",
       "category_1_1_purchase_amount_sum                    NaN              NaN   \n",
       "category_2_1.0_purchase_amount_mean                 NaN         0.139747   \n",
       "category_2_2.0_purchase_amount_mean                 NaN        -0.344766   \n",
       "category_2_3.0_purchase_amount_mean           -0.723677         0.102887   \n",
       "category_2_4.0_purchase_amount_mean                 NaN              NaN   \n",
       "category_2_5.0_purchase_amount_mean                 NaN        -0.361628   \n",
       "category_2_1.0_purchase_amount_sum                  NaN         2.375707   \n",
       "category_2_2.0_purchase_amount_sum                  NaN        -2.758131   \n",
       "category_2_3.0_purchase_amount_sum            -1.447354         0.514433   \n",
       "category_2_4.0_purchase_amount_sum                  NaN              NaN   \n",
       "category_2_5.0_purchase_amount_sum                  NaN        -0.361628   \n",
       "category_3_0_purchase_amount_mean             -0.723677        -0.007407   \n",
       "category_3_1_purchase_amount_mean                   NaN              NaN   \n",
       "category_3_2_purchase_amount_mean                   NaN              NaN   \n",
       "category_3_0_purchase_amount_sum              -1.447354        -0.229620   \n",
       "category_3_1_purchase_amount_sum                    NaN              NaN   \n",
       "category_3_2_purchase_amount_sum                    NaN              NaN   \n",
       "authorized_flag_1_purchase_amount_mean        -0.723677        -0.007407   \n",
       "authorized_flag_1_purchase_amount_sum         -1.447354        -0.229620   \n",
       "\n",
       "card_id                                 C_ID_000183fdda  \n",
       "category_1_0_purchase_amount_mean             -0.599162  \n",
       "category_1_1_purchase_amount_mean                   NaN  \n",
       "category_1_0_purchase_amount_sum              -6.590778  \n",
       "category_1_1_purchase_amount_sum                    NaN  \n",
       "category_2_1.0_purchase_amount_mean                 NaN  \n",
       "category_2_2.0_purchase_amount_mean                 NaN  \n",
       "category_2_3.0_purchase_amount_mean           -0.599162  \n",
       "category_2_4.0_purchase_amount_mean                 NaN  \n",
       "category_2_5.0_purchase_amount_mean                 NaN  \n",
       "category_2_1.0_purchase_amount_sum                  NaN  \n",
       "category_2_2.0_purchase_amount_sum                  NaN  \n",
       "category_2_3.0_purchase_amount_sum            -6.590778  \n",
       "category_2_4.0_purchase_amount_sum                  NaN  \n",
       "category_2_5.0_purchase_amount_sum                  NaN  \n",
       "category_3_0_purchase_amount_mean             -0.107680  \n",
       "category_3_1_purchase_amount_mean             -0.696173  \n",
       "category_3_2_purchase_amount_mean             -0.576515  \n",
       "category_3_0_purchase_amount_sum              -0.107680  \n",
       "category_3_1_purchase_amount_sum              -4.177040  \n",
       "category_3_2_purchase_amount_sum              -2.306059  \n",
       "authorized_flag_1_purchase_amount_mean        -0.599162  \n",
       "authorized_flag_1_purchase_amount_sum         -6.590778  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans_agg_cat.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_agg_cat.reset_index().to_feather('hist_trans_agg_cat')\n",
    "new_hist_trans_agg_cat.reset_index().to_feather('new_hist_trans_agg_cat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_agg_cat = feather.read_dataframe('hist_trans_agg_cat')\n",
    "new_hist_trans_agg_cat = feather.read_dataframe('new_hist_trans_agg_cat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregate on month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More aggregates grouped by month, (`month_diff` is the months since the reference date, we want to add more weight to our recent transactions (i.e., new customers)) & calculate aggregates over the customer spending."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_per_month(history):\n",
    "    grouped = history.groupby(['card_id', 'month_diff'])['purchase_amount']\n",
    "\n",
    "    agg_func = {\n",
    "            'purchase_amount': ['count', 'sum', 'max', 'mean'],\n",
    "            }\n",
    "\n",
    "    intermediate_group = grouped.agg(agg_func)\n",
    "    intermediate_group.columns = ['_month_diff_'.join(col).strip() for col in intermediate_group.columns.values]\n",
    "    intermediate_group.reset_index(inplace=True)\n",
    "    \n",
    "    return intermediate_group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aggregate_per_month(new_hist_trans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai/lib/python3.6/site-packages/ipykernel_launcher.py:8: FutureWarning: using a dict on a Series for aggregation\n",
      "is deprecated and will be removed in a future version\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 36s, sys: 12.5 s, total: 3min 49s\n",
      "Wall time: 3min 29s\n"
     ]
    }
   ],
   "source": [
    "%time hist_trans_agg_month, new_hist_trans_agg_month = [aggregate_per_month(df) for df in [hist_trans, new_hist_trans]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((618851, 6), (431268, 6))"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans_agg_month.shape, new_hist_trans_agg_month.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new aggregates grouped by month & purchase amount for each card are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>C_ID_0001793786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>month_diff</th>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount_month_diff_count</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount_month_diff_sum</th>\n",
       "      <td>110</td>\n",
       "      <td>857.42</td>\n",
       "      <td>2183.57</td>\n",
       "      <td>30.92</td>\n",
       "      <td>12340.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount_month_diff_max</th>\n",
       "      <td>60</td>\n",
       "      <td>250</td>\n",
       "      <td>444.94</td>\n",
       "      <td>21</td>\n",
       "      <td>2580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purchase_amount_month_diff_mean</th>\n",
       "      <td>55</td>\n",
       "      <td>171.484</td>\n",
       "      <td>103.98</td>\n",
       "      <td>15.46</td>\n",
       "      <td>685.599</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                0                1  \\\n",
       "card_id                           C_ID_00007093c1  C_ID_0001238066   \n",
       "month_diff                                     12               11   \n",
       "purchase_amount_month_diff_count                2                5   \n",
       "purchase_amount_month_diff_sum                110           857.42   \n",
       "purchase_amount_month_diff_max                 60              250   \n",
       "purchase_amount_month_diff_mean                55          171.484   \n",
       "\n",
       "                                                2                3  \\\n",
       "card_id                           C_ID_0001238066  C_ID_0001506ef0   \n",
       "month_diff                                     12               12   \n",
       "purchase_amount_month_diff_count               21                2   \n",
       "purchase_amount_month_diff_sum            2183.57            30.92   \n",
       "purchase_amount_month_diff_max             444.94               21   \n",
       "purchase_amount_month_diff_mean            103.98            15.46   \n",
       "\n",
       "                                                4  \n",
       "card_id                           C_ID_0001793786  \n",
       "month_diff                                     15  \n",
       "purchase_amount_month_diff_count               18  \n",
       "purchase_amount_month_diff_sum            12340.8  \n",
       "purchase_amount_month_diff_max               2580  \n",
       "purchase_amount_month_diff_mean           685.599  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans_agg_month.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for df in [hist_trans_agg_month, new_hist_trans_agg_month]:\n",
    "#     df.drop(['month_diff_mean', 'month_diff_sum', 'month_diff_ptp', 'month_diff_max'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_agg_month.to_feather('hist_trans_agg_month')\n",
    "new_hist_trans_agg_month.to_feather('new_hist_trans_agg_month')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chandrasekhar/anaconda3/envs/fastai/lib/python3.6/site-packages/pyarrow/pandas_compat.py:752: FutureWarning: .labels was deprecated in version 0.24.0. Use .codes instead.\n",
      "  labels, = index.labels\n"
     ]
    }
   ],
   "source": [
    "new_hist_trans_agg_month = feather.read_dataframe('new_hist_trans_agg_month')\n",
    "hist_trans_agg_month = feather.read_dataframe('hist_trans_agg_month')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load test & train DFs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = 'data/elo/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = [pd.read_csv(f'{PATH}{c}') for c in ['train.csv', 'test.csv']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add observed date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check kaggle discussion for reverse engineering the observation date aka reference date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>first_active_month</th>\n",
       "      <td>2017-06</td>\n",
       "      <td>2017-01</td>\n",
       "      <td>2016-08</td>\n",
       "      <td>2017-09</td>\n",
       "      <td>2017-11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_92a2005557</td>\n",
       "      <td>C_ID_3d0044924f</td>\n",
       "      <td>C_ID_d639edf6cd</td>\n",
       "      <td>C_ID_186d6a6901</td>\n",
       "      <td>C_ID_cdbd2c0db2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>target</th>\n",
       "      <td>-0.820283</td>\n",
       "      <td>0.392913</td>\n",
       "      <td>0.688056</td>\n",
       "      <td>0.142495</td>\n",
       "      <td>-0.159749</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  0                1                2  \\\n",
       "first_active_month          2017-06          2017-01          2016-08   \n",
       "card_id             C_ID_92a2005557  C_ID_3d0044924f  C_ID_d639edf6cd   \n",
       "feature_1                         5                4                2   \n",
       "feature_2                         2                1                2   \n",
       "feature_3                         1                0                0   \n",
       "target                    -0.820283         0.392913         0.688056   \n",
       "\n",
       "                                  3                4  \n",
       "first_active_month          2017-09          2017-11  \n",
       "card_id             C_ID_186d6a6901  C_ID_cdbd2c0db2  \n",
       "feature_1                         4                1  \n",
       "feature_2                         3                3  \n",
       "feature_3                         0                0  \n",
       "target                     0.142495        -0.159749  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>first_active_month</th>\n",
       "      <td>2017-04</td>\n",
       "      <td>2017-01</td>\n",
       "      <td>2017-08</td>\n",
       "      <td>2017-12</td>\n",
       "      <td>2015-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>card_id</th>\n",
       "      <td>C_ID_0ab67a22ab</td>\n",
       "      <td>C_ID_130fd0cbdd</td>\n",
       "      <td>C_ID_b709037bc5</td>\n",
       "      <td>C_ID_d27d835a9f</td>\n",
       "      <td>C_ID_2b5e3df5c2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_1</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_2</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  0                1                2  \\\n",
       "first_active_month          2017-04          2017-01          2017-08   \n",
       "card_id             C_ID_0ab67a22ab  C_ID_130fd0cbdd  C_ID_b709037bc5   \n",
       "feature_1                         3                2                5   \n",
       "feature_2                         3                3                1   \n",
       "feature_3                         1                0                1   \n",
       "\n",
       "                                  3                4  \n",
       "first_active_month          2017-12          2015-12  \n",
       "card_id             C_ID_d27d835a9f  C_ID_2b5e3df5c2  \n",
       "feature_1                         2                5  \n",
       "feature_2                         1                1  \n",
       "feature_3                         0                1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We had some success by taking ratios of purchase amounts before and after reference date. Let's go ahead and add more such features. We will calculate the latest max spending of the customer on historical transactions , the first month's max spending in the new transactions and take their ratios later.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_hist_transaction = hist_trans.groupby('card_id').agg({'month_lag' : 'max', 'purchase_date' : 'max'}).reset_index()\n",
    "last_hist_transaction.columns = ['card_id', 'hist_month_lag', 'hist_purchase_date']\n",
    "first_new_transaction = new_hist_trans.groupby('card_id').agg({'month_lag' : 'min', 'purchase_date' : 'min'}).reset_index()\n",
    "first_new_transaction.columns = ['card_id', 'new_month_lag', 'new_purchase_date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_hist_transaction['hist_purchase_date'] = pd.to_datetime(last_hist_transaction['hist_purchase_date']) \n",
    "first_new_transaction['new_purchase_date'] = pd.to_datetime(first_new_transaction['new_purchase_date']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>hist_month_lag</th>\n",
       "      <th>hist_purchase_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 05:14:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 16:18:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-17 12:33:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-10-31 20:20:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-25 20:57:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  hist_month_lag  hist_purchase_date\n",
       "0  C_ID_00007093c1               0 2018-02-27 05:14:57\n",
       "1  C_ID_0001238066               0 2018-02-27 16:18:59\n",
       "2  C_ID_0001506ef0               0 2018-02-17 12:33:56\n",
       "3  C_ID_0001793786               0 2017-10-31 20:20:18\n",
       "4  C_ID_000183fdda               0 2018-02-25 20:57:08"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_hist_transaction.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>new_month_lag</th>\n",
       "      <th>new_purchase_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>2</td>\n",
       "      <td>2018-04-03 11:13:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-01 16:48:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-16 22:21:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-15 15:44:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-02 12:26:26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  new_month_lag   new_purchase_date\n",
       "0  C_ID_00007093c1              2 2018-04-03 11:13:35\n",
       "1  C_ID_0001238066              1 2018-03-01 16:48:27\n",
       "2  C_ID_0001506ef0              1 2018-03-16 22:21:58\n",
       "3  C_ID_0001793786              1 2017-11-15 15:44:20\n",
       "4  C_ID_000183fdda              1 2018-03-02 12:26:26"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_new_transaction.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_hist_transaction['observation_date'] = \\\n",
    "    last_hist_transaction.apply(lambda x: x['hist_purchase_date']  - pd.DateOffset(months=x['hist_month_lag']), axis=1)\n",
    "\n",
    "first_new_transaction['observation_date'] = \\\n",
    "    first_new_transaction.apply(lambda x: x['new_purchase_date']  - pd.DateOffset(months=x['new_month_lag']-1), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>hist_month_lag</th>\n",
       "      <th>hist_purchase_date</th>\n",
       "      <th>observation_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 05:14:57</td>\n",
       "      <td>2018-02-27 05:14:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 16:18:59</td>\n",
       "      <td>2018-02-27 16:18:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-17 12:33:56</td>\n",
       "      <td>2018-02-17 12:33:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-10-31 20:20:18</td>\n",
       "      <td>2017-10-31 20:20:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-25 20:57:08</td>\n",
       "      <td>2018-02-25 20:57:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  hist_month_lag  hist_purchase_date    observation_date\n",
       "0  C_ID_00007093c1               0 2018-02-27 05:14:57 2018-02-27 05:14:57\n",
       "1  C_ID_0001238066               0 2018-02-27 16:18:59 2018-02-27 16:18:59\n",
       "2  C_ID_0001506ef0               0 2018-02-17 12:33:56 2018-02-17 12:33:56\n",
       "3  C_ID_0001793786               0 2017-10-31 20:20:18 2017-10-31 20:20:18\n",
       "4  C_ID_000183fdda               0 2018-02-25 20:57:08 2018-02-25 20:57:08"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_hist_transaction.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>new_month_lag</th>\n",
       "      <th>new_purchase_date</th>\n",
       "      <th>observation_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>2</td>\n",
       "      <td>2018-04-03 11:13:35</td>\n",
       "      <td>2018-03-03 11:13:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-01 16:48:27</td>\n",
       "      <td>2018-03-01 16:48:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-16 22:21:58</td>\n",
       "      <td>2018-03-16 22:21:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-15 15:44:20</td>\n",
       "      <td>2017-11-15 15:44:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-02 12:26:26</td>\n",
       "      <td>2018-03-02 12:26:26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  new_month_lag   new_purchase_date    observation_date\n",
       "0  C_ID_00007093c1              2 2018-04-03 11:13:35 2018-03-03 11:13:35\n",
       "1  C_ID_0001238066              1 2018-03-01 16:48:27 2018-03-01 16:48:27\n",
       "2  C_ID_0001506ef0              1 2018-03-16 22:21:58 2018-03-16 22:21:58\n",
       "3  C_ID_0001793786              1 2017-11-15 15:44:20 2017-11-15 15:44:20\n",
       "4  C_ID_000183fdda              1 2018-03-02 12:26:26 2018-03-02 12:26:26"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_new_transaction.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_hist_transaction['observation_date'] = last_hist_transaction['observation_date'].dt.to_period('M').dt.to_timestamp() + pd.DateOffset(months=1)\n",
    "first_new_transaction['observation_date'] = first_new_transaction['observation_date'].dt.to_period('M').dt.to_timestamp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>hist_month_lag</th>\n",
       "      <th>hist_purchase_date</th>\n",
       "      <th>observation_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 05:14:57</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-27 16:18:59</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-17 12:33:56</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>0</td>\n",
       "      <td>2017-10-31 20:20:18</td>\n",
       "      <td>2017-11-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>0</td>\n",
       "      <td>2018-02-25 20:57:08</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  hist_month_lag  hist_purchase_date observation_date\n",
       "0  C_ID_00007093c1               0 2018-02-27 05:14:57       2018-03-01\n",
       "1  C_ID_0001238066               0 2018-02-27 16:18:59       2018-03-01\n",
       "2  C_ID_0001506ef0               0 2018-02-17 12:33:56       2018-03-01\n",
       "3  C_ID_0001793786               0 2017-10-31 20:20:18       2017-11-01\n",
       "4  C_ID_000183fdda               0 2018-02-25 20:57:08       2018-03-01"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_hist_transaction.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>new_month_lag</th>\n",
       "      <th>new_purchase_date</th>\n",
       "      <th>observation_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C_ID_00007093c1</td>\n",
       "      <td>2</td>\n",
       "      <td>2018-04-03 11:13:35</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C_ID_0001238066</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-01 16:48:27</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C_ID_0001506ef0</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-16 22:21:58</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C_ID_0001793786</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-11-15 15:44:20</td>\n",
       "      <td>2017-11-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C_ID_000183fdda</td>\n",
       "      <td>1</td>\n",
       "      <td>2018-03-02 12:26:26</td>\n",
       "      <td>2018-03-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           card_id  new_month_lag   new_purchase_date observation_date\n",
       "0  C_ID_00007093c1              2 2018-04-03 11:13:35       2018-03-01\n",
       "1  C_ID_0001238066              1 2018-03-01 16:48:27       2018-03-01\n",
       "2  C_ID_0001506ef0              1 2018-03-16 22:21:58       2018-03-01\n",
       "3  C_ID_0001793786              1 2017-11-15 15:44:20       2017-11-01\n",
       "4  C_ID_000183fdda              1 2018-03-02 12:26:26       2018-03-01"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_new_transaction.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_new_transaction.drop(['new_month_lag', 'new_purchase_date'], axis=1, inplace=True)\n",
    "last_hist_transaction.drop(['hist_month_lag', 'hist_purchase_date'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validate = last_hist_transaction.merge(first_new_transaction, on = 'card_id')\n",
    "all(validate['observation_date_x'] == validate['observation_date_y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'hist_trans_agg_cat' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-21-fba744baa6d0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mhist_trans_agg_cat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhist_trans_agg_month\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhist_trans_agg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'hist_trans_agg_cat' is not defined"
     ]
    }
   ],
   "source": [
    "hist_trans_agg_cat.shape, hist_trans_agg_month.shape, hist_trans_agg.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'new_hist_trans_agg' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-06cf721f7b5e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnew_hist_trans_agg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_hist_trans_agg_month\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_hist_trans_agg_cat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'new_hist_trans_agg' is not defined"
     ]
    }
   ],
   "source": [
    "new_hist_trans_agg.shape, new_hist_trans_agg_month.shape, new_hist_trans_agg_cat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_rows', 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((325540, 114), (325540, 25), (325540, 2), (618851, 6))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans_agg.shape, hist_trans_agg_cat.shape, last_hist_transaction.shape, hist_trans_agg_month.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merge train & test with new & old transactions history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merging all our aggregates & features with train & test data frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def join_dfs(left, right, left_on, right_on=None, suffix='_old'):\n",
    "    if right_on is None: right_on = left_on\n",
    "    return left.merge(right, how='left', left_on=left_on, right_on=right_on, suffixes=(\"\", suffix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 6), (123623, 5))"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = join_dfs(train, new_hist_trans_agg, left_on='card_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = join_dfs(train_df, hist_trans_agg, left_on='card_id', suffix='_old')\n",
    "# train_df = join_dfs(train_df, hist_trans_agg_month, left_on='card_id', suffix='_old')\n",
    "train_df = join_dfs(train_df, hist_trans_agg_cat, left_on='card_id', suffix='_old')\n",
    "train_df = join_dfs(train_df, last_hist_transaction, left_on='card_id', suffix='_old')\n",
    "# train_df = join_dfs(train_df, new_hist_trans_agg_month, left_on='card_id')\n",
    "train_df = join_dfs(train_df, new_hist_trans_agg_cat, left_on='card_id')\n",
    "train_df = join_dfs(train_df, first_new_transaction, left_on='card_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = join_dfs(test, new_hist_trans_agg, left_on='card_id')\n",
    "test_df = join_dfs(test_df, first_new_transaction, left_on='card_id')\n",
    "# test_df = join_dfs(test_df, new_hist_trans_agg_month, left_on='card_id')\n",
    "test_df = join_dfs(test_df, new_hist_trans_agg_cat, left_on='card_id')\n",
    "# test_df = join_dfs(test_df, hist_trans_agg_month, left_on='card_id', suffix='_old')\n",
    "test_df = join_dfs(test_df, hist_trans_agg_cat, left_on='card_id', suffix='_old')\n",
    "test_df = join_dfs(test_df, hist_trans_agg, left_on='card_id', suffix='_old')\n",
    "test_df = join_dfs(test_df, last_hist_transaction, left_on='card_id', suffix='_old')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((123623, 273), (201917, 274))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.shape, train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding train days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "observation_date    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[['observation_date']].isnull().sum().sort_values(ascending=False)*100/len(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_df[['observation_date_old', 'observation_date']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hist_trans_agg_cat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in [train_df, test_df]:\n",
    "    df.drop('observation_date', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding feature interactions between `feature_1, feature_2 & feature_3` of the card & the time elapsed since the observed date aka reference date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_days_feature_interaction(df):\n",
    "    # to datetime\n",
    "    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n",
    "    df['observation_date_old'] = pd.to_datetime(df['observation_date_old'])\n",
    "    # datetime features\n",
    "    df['quarter'] = df['first_active_month'].dt.quarter\n",
    "#     df['elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n",
    "    df['observed_elapsed_time'] = (df['observation_date_old'] - df['first_active_month']).dt.days\n",
    "    df['days_feature1'] = df['observed_elapsed_time'] * df['feature_1']\n",
    "    df['days_feature2'] = df['observed_elapsed_time'] * df['feature_2']\n",
    "    df['days_feature3'] = df['observed_elapsed_time'] * df['feature_3']\n",
    "\n",
    "    df['days_feature1_ratio'] = df['feature_1'] / df['observed_elapsed_time']\n",
    "    df['days_feature2_ratio'] = df['feature_2'] / df['observed_elapsed_time']\n",
    "    df['days_feature3_ratio'] = df['feature_3'] / df['observed_elapsed_time']\n",
    "    \n",
    "    df['feature_sum'] = df['feature_1'] + df['feature_2'] + df['feature_3']\n",
    "    df['feature_mean'] = df['feature_sum']/3\n",
    "    df['feature_max'] = df[['feature_1', 'feature_2', 'feature_3']].max(axis=1)\n",
    "    df['feature_min'] = df[['feature_1', 'feature_2', 'feature_3']].min(axis=1)\n",
    "    df['feature_var'] = df[['feature_1', 'feature_2', 'feature_3']].std(axis=1)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, test_df = [add_days_feature_interaction(d) for d in [train_df, test_df]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add extra columns like age, total transactions, installments, purchase amount, first buy etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 286), (123623, 285))"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "335"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_df[['elapsed_time','observed_elapsed_time']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_df = train_df.loc[:,~train_df.columns.duplicated()]\n",
    "# test_df = test_df.loc[:,~test_df.columns.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['first_active_month', 'card_id', 'feature_1', 'feature_2', 'feature_3', 'target',\n",
       "       'transactions_count', 'subsector_id_nunique', 'merchant_id_nunique', 'merchant_category_id_nunique',\n",
       "       'merchant_address_id_nunique', 'purchase_Month_mean', 'purchase_Month_min', 'purchase_Month_max',\n",
       "       'purchase_Week_nunique', 'purchase_Week_mean', 'purchase_Week_min', 'purchase_Week_max',\n",
       "       'purchase_Dayofweek_mean', 'purchase_Dayofweek_min', 'purchase_Dayofweek_max', 'purchase_Day_nunique',\n",
       "       'purchase_Day_mean', 'purchase_Day_min', 'purchase_Day_max', 'purchase_Hour_nunique',\n",
       "       'purchase_Hour_mean', 'purchase_Hour_min', 'purchase_Hour_max', 'merchant_rating_nunique',\n",
       "       'merchant_rating_mean', 'merchant_rating_min', 'merchant_rating_max', 'purchase_amount_sum',\n",
       "       'purchase_amount_pct_75', 'purchase_amount_pct_25', 'purchase_amount_mean', 'purchase_amount_median',\n",
       "       'purchase_amount_max', 'purchase_amount_min', 'purchase_amount_var', 'purchase_amount_skew',\n",
       "       'purchase_amount_head_sum', 'purchase_amount_head_max', 'purchase_amount_tail_sum',\n",
       "       'purchase_amount_tail_max', 'purchase_amount_gmean', 'purchase_amount_hmean', 'installments_sum',\n",
       "       'installments_pct_75', 'installments_pct_25', 'installments_mean', 'installments_median',\n",
       "       'installments_max', 'installments_min', 'installments_var', 'installments_skew',\n",
       "       'installments_head_sum', 'installments_head_max', 'installments_tail_sum', 'installments_tail_max',\n",
       "       'purchase_date_max', 'purchase_date_min', 'month_lag_max', 'month_lag_min', 'month_lag_mean',\n",
       "       'month_lag_var', 'month_lag_skew', 'month_diff_mean', 'month_diff_var', 'month_diff_skew',\n",
       "       'purchased_on_weekend_mean', 'category_1_mean', 'category_2_mean', 'category_3_mean',\n",
       "       'category_4_mean', 'numerical_range_mean', 'numerical_range_min', 'numerical_range_max',\n",
       "       'numerical_range_skew', 'card_id_size', 'card_id_count', 'price_mean', 'price_max', 'price_min',\n",
       "       'price_var', 'Christmas_Day_2017_mean', 'Children_day_2017_mean', 'Black_Friday_2017_mean',\n",
       "       'Mothers_Day_2018_mean', 'duration_mean', 'duration_min', 'duration_max', 'duration_var',\n",
       "       'duration_skew', 'amount_month_ratio_mean', 'amount_month_ratio_min', 'amount_month_ratio_max',\n",
       "       'amount_month_ratio_var', 'amount_month_ratio_skew', 'purchase_date_successive_diff_mean',\n",
       "       'category_2_mean_mean', 'category_3_mean_mean', 'purchase_date_diff', 'purchase_date_average',\n",
       "       'purchase_date_uptonow', 'purchase_date_uptomin', 'inverse_avg_transactions_per_day',\n",
       "       'repurchase_merchant_rate', 'merchant_category_repurchase', 'avg_spend_per_merchant',\n",
       "       'avg_trans_per_merchant', 'avg_spend_per_transaction', 'transactions_count_old',\n",
       "       'subsector_id_nunique_old', 'merchant_id_nunique_old', 'merchant_category_id_nunique_old',\n",
       "       'merchant_address_id_nunique_old', 'purchase_Month_nunique', 'purchase_Month_mean_old',\n",
       "       'purchase_Month_min_old', 'purchase_Month_max_old', 'purchase_Week_nunique_old',\n",
       "       'purchase_Week_mean_old', 'purchase_Week_min_old', 'purchase_Week_max_old',\n",
       "       'purchase_Dayofweek_mean_old', 'purchase_Day_nunique_old', 'purchase_Day_mean_old',\n",
       "       'purchase_Day_min_old', 'purchase_Hour_nunique_old', 'purchase_Hour_mean_old',\n",
       "       'purchase_Hour_min_old', 'purchase_Hour_max_old', 'merchant_rating_nunique_old',\n",
       "       'merchant_rating_mean_old', 'merchant_rating_min_old', 'merchant_rating_max_old',\n",
       "       'purchase_amount_sum_old', 'purchase_amount_pct_75_old', 'purchase_amount_pct_25_old',\n",
       "       'purchase_amount_mean_old', 'purchase_amount_median_old', 'purchase_amount_max_old',\n",
       "       'purchase_amount_min_old', 'purchase_amount_var_old', 'purchase_amount_skew_old',\n",
       "       'purchase_amount_head_sum_old', 'purchase_amount_head_max_old', 'purchase_amount_tail_sum_old',\n",
       "       'purchase_amount_tail_max_old', 'purchase_amount_gmean_old', 'purchase_amount_hmean_old',\n",
       "       'installments_sum_old', 'installments_pct_75_old', 'installments_pct_25_old', 'installments_mean_old',\n",
       "       'installments_median_old', 'installments_max_old', 'installments_min_old', 'installments_var_old',\n",
       "       'installments_skew_old', 'installments_head_sum_old', 'installments_head_max_old',\n",
       "       'installments_tail_sum_old', 'installments_tail_max_old', 'purchase_date_max_old',\n",
       "       'purchase_date_min_old', 'month_lag_max_old', 'month_lag_min_old', 'month_lag_mean_old',\n",
       "       'month_lag_var_old', 'month_lag_skew_old', 'month_diff_max', 'month_diff_min', 'month_diff_mean_old',\n",
       "       'month_diff_var_old', 'month_diff_skew_old', 'authorized_flag_mean', 'purchased_on_weekend_mean_old',\n",
       "       'category_1_mean_old', 'category_2_mean_old', 'category_3_mean_old', 'category_4_mean_old',\n",
       "       'numerical_range_mean_old', 'numerical_range_min_old', 'numerical_range_max_old',\n",
       "       'numerical_range_skew_old', 'card_id_size_old', 'card_id_count_old', 'price_sum', 'price_mean_old',\n",
       "       'price_max_old', 'price_min_old', 'price_var_old', 'price_skew', 'Christmas_Day_2017_mean_old',\n",
       "       'Mothers_Day_2017_mean', 'fathers_day_2017_mean', 'Children_day_2017_mean_old',\n",
       "       'Valentine_Day_2017_mean', 'Black_Friday_2017_mean_old', 'Mothers_Day_2018_mean_old',\n",
       "       'duration_mean_old', 'duration_min_old', 'duration_max_old', 'duration_var_old', 'duration_skew_old',\n",
       "       'amount_month_ratio_mean_old', 'amount_month_ratio_min_old', 'amount_month_ratio_max_old',\n",
       "       'amount_month_ratio_var_old', 'amount_month_ratio_skew_old', 'purchase_date_successive_diff_mean_old',\n",
       "       'category_2_mean_mean_old', 'category_3_mean_mean_old', 'purchase_date_diff_old',\n",
       "       'purchase_date_average_old', 'purchase_date_uptonow_old', 'purchase_date_uptomin_old',\n",
       "       'inverse_avg_transactions_per_day_old', 'repurchase_merchant_rate_old',\n",
       "       'merchant_category_repurchase_old', 'avg_spend_per_merchant_old', 'avg_trans_per_merchant_old',\n",
       "       'avg_spend_per_transaction_old', 'category_1_0_purchase_amount_mean',\n",
       "       'category_1_1_purchase_amount_mean', 'category_1_0_purchase_amount_sum',\n",
       "       'category_1_1_purchase_amount_sum', 'category_2_1.0_purchase_amount_mean',\n",
       "       'category_2_2.0_purchase_amount_mean', 'category_2_3.0_purchase_amount_mean',\n",
       "       'category_2_4.0_purchase_amount_mean', 'category_2_5.0_purchase_amount_mean',\n",
       "       'category_2_1.0_purchase_amount_sum', 'category_2_2.0_purchase_amount_sum',\n",
       "       'category_2_3.0_purchase_amount_sum', 'category_2_4.0_purchase_amount_sum',\n",
       "       'category_2_5.0_purchase_amount_sum', 'category_3_0_purchase_amount_mean',\n",
       "       'category_3_1_purchase_amount_mean', 'category_3_2_purchase_amount_mean',\n",
       "       'category_3_0_purchase_amount_sum', 'category_3_1_purchase_amount_sum',\n",
       "       'category_3_2_purchase_amount_sum', 'authorized_flag_0_purchase_amount_mean',\n",
       "       'authorized_flag_1_purchase_amount_mean', 'authorized_flag_0_purchase_amount_sum',\n",
       "       'authorized_flag_1_purchase_amount_sum', 'category_1_0_purchase_amount_mean_old',\n",
       "       'category_1_1_purchase_amount_mean_old', 'category_1_0_purchase_amount_sum_old',\n",
       "       'category_1_1_purchase_amount_sum_old', 'category_2_1.0_purchase_amount_mean_old',\n",
       "       'category_2_2.0_purchase_amount_mean_old', 'category_2_3.0_purchase_amount_mean_old',\n",
       "       'category_2_4.0_purchase_amount_mean_old', 'category_2_5.0_purchase_amount_mean_old',\n",
       "       'category_2_1.0_purchase_amount_sum_old', 'category_2_2.0_purchase_amount_sum_old',\n",
       "       'category_2_3.0_purchase_amount_sum_old', 'category_2_4.0_purchase_amount_sum_old',\n",
       "       'category_2_5.0_purchase_amount_sum_old', 'category_3_0_purchase_amount_mean_old',\n",
       "       'category_3_1_purchase_amount_mean_old', 'category_3_2_purchase_amount_mean_old',\n",
       "       'category_3_0_purchase_amount_sum_old', 'category_3_1_purchase_amount_sum_old',\n",
       "       'category_3_2_purchase_amount_sum_old', 'authorized_flag_1_purchase_amount_mean_old',\n",
       "       'authorized_flag_1_purchase_amount_sum_old', 'observation_date_old', 'quarter',\n",
       "       'observed_elapsed_time', 'days_feature1', 'days_feature2', 'days_feature3', 'days_feature1_ratio',\n",
       "       'days_feature2_ratio', 'days_feature3_ratio', 'feature_sum', 'feature_mean', 'feature_max',\n",
       "       'feature_min', 'feature_var'], dtype=object)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have around 285 features adding all our aggregates. Can we squeeze in more? Yes. Let's add some feature interactions with old & new transactions data. I also added new features like `CLV` (my own Customer Lifetime Value formula) which is basically feature interaction between number of transactions, age & sum of the transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for df in [train_df, test_df]:\n",
    "    df['card_id_total'] = df['card_id_size']+df['card_id_size_old']\n",
    "    df['card_id_count_total'] = df['card_id_count']+df['card_id_count_old']\n",
    "    df['card_id_count_ratio'] = df['card_id_count'] / df['card_id_count_old']\n",
    "    df['purchase_amount_total'] = df['purchase_amount_sum_old']+df['purchase_amount_sum']\n",
    "    df['purchase_amount_total_mean'] = df['purchase_amount_mean']+df['purchase_amount_mean_old']\n",
    "    df['purchase_amount_total_max'] = df['purchase_amount_max']+df['purchase_amount_max_old']\n",
    "    df['purchase_amount_total_min'] = df['purchase_amount_min']+df['purchase_amount_min_old']\n",
    "    df['purchase_amount_sum_ratio'] = df['purchase_amount_sum'] / df['purchase_amount_sum_old']\n",
    "    df['hist_first_buy'] = (df['purchase_date_min_old'] - df['first_active_month']).dt.days\n",
    "    df['new_first_buy'] = (df['purchase_date_min'] - df['first_active_month']).dt.days\n",
    "    df['hist_last_buy'] = (df['purchase_date_max_old'] - df['first_active_month']).dt.days\n",
    "    df['new_last_buy'] = (df['purchase_date_max'] - df['first_active_month']).dt.days\n",
    "    df['avg_spend_per_transaction'] = df['purchase_amount_total']/df['card_id_total']\n",
    "    df['purchased_before_issue'] = df['hist_first_buy'] < 0\n",
    "    df['month_diff_mean_total'] = df['month_diff_mean']+df['month_diff_mean_old']\n",
    "    df['month_diff_ratio'] = df['month_diff_mean']/df['month_diff_mean_old']\n",
    "    df['month_lag_mean_total'] = df['month_lag_mean']+df['month_lag_mean_old']\n",
    "    df['month_lag_max_total'] = df['month_lag_max']+df['month_lag_max_old']\n",
    "    df['month_lag_min_total'] = df['month_lag_min']+df['month_lag_min_old']\n",
    "    df['category_1_mean_total'] = df['category_1_mean']+df['category_1_mean_old']\n",
    "    df['category_4_mean_total'] = df['category_4_mean']+df['category_4_mean_old']\n",
    "    df['category_4_mean_ratio'] = df['category_4_mean']/df['category_4_mean_old']\n",
    "    df['category_1_mean_ratio'] = df['category_1_mean']/df['category_1_mean_old']\n",
    "    df['numerical_range_mean_total'] = df['numerical_range_mean']+df['numerical_range_mean_old']\n",
    "    df['numerical_range_mean_ratio'] = df['numerical_range_mean']/df['numerical_range_mean_old']\n",
    "    df['merchant_rating_mean_ratio'] = df['merchant_rating_mean']/df['merchant_rating_mean_old']\n",
    "    df['installments_total'] = df['installments_sum']+df['installments_sum_old']\n",
    "    df['installments_mean_total'] = df['installments_mean']+df['installments_mean_old']\n",
    "    df['installments_max_total'] = df['installments_max']+df['installments_max_old']\n",
    "    df['installments_ratio'] = df['installments_sum']/df['installments_sum_old']\n",
    "    df['price_total'] = df['purchase_amount_total'] / df['installments_total']\n",
    "    df['price_mean'] = df['purchase_amount_mean'] / df['installments_mean']\n",
    "    df['price_max'] = df['purchase_amount_max'] / df['installments_max']\n",
    "    df['duration_mean_total'] = df['duration_mean']+df['duration_mean_old']\n",
    "    df['duration_min_total'] = df['duration_min']+df['duration_min_old']\n",
    "    df['duration_max_total'] = df['duration_max']+df['duration_max_old']\n",
    "    df['amount_month_ratio_mean_total']=df['amount_month_ratio_mean']+df['amount_month_ratio_mean_old']\n",
    "    df['amount_month_ratio_min_total']=df['amount_month_ratio_min']+df['amount_month_ratio_min_old']\n",
    "    df['amount_month_ratio_max_total']=df['amount_month_ratio_max']+df['amount_month_ratio_max_old']\n",
    "    df['CLV'] = df['card_id_count'] * df['purchase_amount_sum'] / df['month_diff_mean']\n",
    "    df['CLV_old'] = df['card_id_count_old'] * df['purchase_amount_sum_old'] / df['month_diff_mean_old']\n",
    "    df['CLV_ratio'] = df['CLV'] / df['CLV_old']\n",
    "    df['category_1_0_purchase_amount_mean_total'] = df['category_1_0_purchase_amount_mean'] + df['category_1_0_purchase_amount_mean_old']\n",
    "    df['category_1_1_purchase_amount_mean_total'] = df['category_1_1_purchase_amount_mean'] + df['category_1_1_purchase_amount_mean_old']\n",
    "    df['category_1_0_purchase_amount_sum_total'] = df['category_1_0_purchase_amount_sum'] + df['category_1_0_purchase_amount_sum_old']\n",
    "    df['category_1_1_purchase_amount_sum_total'] = df['category_1_1_purchase_amount_sum'] + df['category_1_1_purchase_amount_sum_old']\n",
    "    df['category_2_1.0_purchase_amount_mean_total'] = df['category_2_1.0_purchase_amount_mean'] + df['category_2_1.0_purchase_amount_mean_old']\n",
    "    df['category_2_2.0_purchase_amount_mean_total'] = df['category_2_2.0_purchase_amount_mean'] + df['category_2_2.0_purchase_amount_mean_old']\n",
    "    df['category_2_3.0_purchase_amount_mean_total'] = df['category_2_3.0_purchase_amount_mean'] + df['category_2_3.0_purchase_amount_mean_old']\n",
    "    df['category_2_4.0_purchase_amount_mean_total'] = df['category_2_4.0_purchase_amount_mean'] + df['category_2_4.0_purchase_amount_mean_old']\n",
    "    df['category_2_5.0_purchase_amount_mean_total'] = df['category_2_5.0_purchase_amount_mean'] + df['category_2_5.0_purchase_amount_mean_old']\n",
    "    df['category_2_1.0_purchase_amount_sum_total'] = df['category_2_1.0_purchase_amount_sum'] + df['category_2_1.0_purchase_amount_sum_old']\n",
    "    df['category_2_2.0_purchase_amount_sum_total'] = df['category_2_2.0_purchase_amount_sum'] + df['category_2_2.0_purchase_amount_sum_old']\n",
    "    df['category_2_3.0_purchase_amount_sum_total'] = df['category_2_3.0_purchase_amount_sum'] + df['category_2_3.0_purchase_amount_sum_old']\n",
    "    df['category_2_4.0_purchase_amount_sum_total'] = df['category_2_4.0_purchase_amount_sum'] + df['category_2_4.0_purchase_amount_sum_old']\n",
    "    df['category_2_5.0_purchase_amount_sum_total'] = df['category_2_5.0_purchase_amount_sum'] + df['category_2_5.0_purchase_amount_sum_old']\n",
    "    df['category_3_0_purchase_amount_mean_total'] = df['category_3_0_purchase_amount_mean'] + df['category_3_0_purchase_amount_mean_old']\n",
    "    df['category_3_1_purchase_amount_mean_total'] = df['category_3_1_purchase_amount_mean'] + df['category_3_1_purchase_amount_mean_old']\n",
    "    df['category_3_2_purchase_amount_mean_total'] = df['category_3_2_purchase_amount_mean'] + df['category_3_2_purchase_amount_mean_old']\n",
    "    df['category_3_0_purchase_amount_sum_total'] = df['category_3_0_purchase_amount_sum'] + df['category_3_0_purchase_amount_sum_old']\n",
    "    df['category_3_1_purchase_amount_sum_total'] = df['category_3_1_purchase_amount_sum'] + df['category_3_1_purchase_amount_sum_old']\n",
    "    df['category_3_2_purchase_amount_sum_total'] = df['category_3_2_purchase_amount_sum'] + df['category_3_2_purchase_amount_sum_old']\n",
    "    df['category_1_0_purchase_amount_mean_ratio']  = df['category_1_0_purchase_amount_mean'] / df['category_1_0_purchase_amount_mean_old']\n",
    "    df['category_1_1_purchase_amount_mean_ratio']  = df['category_1_1_purchase_amount_mean'] / df['category_1_1_purchase_amount_mean_old']\n",
    "    df['category_1_0_purchase_amount_sum_ratio']  = df['category_1_0_purchase_amount_sum'] / df['category_1_0_purchase_amount_sum_old']\n",
    "    df['category_1_1_purchase_amount_sum_ratio']  = df['category_1_1_purchase_amount_sum'] / df['category_1_1_purchase_amount_sum_old']\n",
    "    df['category_2_1.0_purchase_amount_mean_ratio']  = df['category_2_1.0_purchase_amount_mean'] / df['category_2_1.0_purchase_amount_mean_old']\n",
    "    df['category_2_2.0_purchase_amount_mean_ratio']  = df['category_2_2.0_purchase_amount_mean'] / df['category_2_2.0_purchase_amount_mean_old']\n",
    "    df['category_2_3.0_purchase_amount_mean_ratio']  = df['category_2_3.0_purchase_amount_mean'] / df['category_2_3.0_purchase_amount_mean_old']\n",
    "    df['category_2_4.0_purchase_amount_mean_ratio']  = df['category_2_4.0_purchase_amount_mean'] / df['category_2_4.0_purchase_amount_mean_old']\n",
    "    df['category_2_5.0_purchase_amount_mean_ratio']  = df['category_2_5.0_purchase_amount_mean'] / df['category_2_5.0_purchase_amount_mean_old']\n",
    "    df['category_2_1.0_purchase_amount_sum_ratio']  = df['category_2_1.0_purchase_amount_sum'] / df['category_2_1.0_purchase_amount_sum_old']\n",
    "    df['category_2_2.0_purchase_amount_sum_ratio']  = df['category_2_2.0_purchase_amount_sum'] / df['category_2_2.0_purchase_amount_sum_old']\n",
    "    df['category_2_3.0_purchase_amount_sum_ratio']  = df['category_2_3.0_purchase_amount_sum'] / df['category_2_3.0_purchase_amount_sum_old']\n",
    "    df['category_2_4.0_purchase_amount_sum_ratio']  = df['category_2_4.0_purchase_amount_sum'] / df['category_2_4.0_purchase_amount_sum_old']\n",
    "    df['category_2_5.0_purchase_amount_sum_ratio']  = df['category_2_5.0_purchase_amount_sum'] / df['category_2_5.0_purchase_amount_sum_old']\n",
    "    df['category_3_0_purchase_amount_mean_ratio']  = df['category_3_0_purchase_amount_mean'] / df['category_3_0_purchase_amount_mean_old']\n",
    "    df['category_3_1_purchase_amount_mean_ratio']  = df['category_3_1_purchase_amount_mean'] / df['category_3_1_purchase_amount_mean_old']\n",
    "    df['category_3_2_purchase_amount_mean_ratio']  = df['category_3_2_purchase_amount_mean'] / df['category_3_2_purchase_amount_mean_old']\n",
    "    df['category_3_0_purchase_amount_sum_ratio']  = df['category_3_0_purchase_amount_sum'] / df['category_3_0_purchase_amount_sum_old']\n",
    "    df['category_3_1_purchase_amount_sum_ratio']  = df['category_3_1_purchase_amount_sum'] / df['category_3_1_purchase_amount_sum_old']\n",
    "    df['category_3_2_purchase_amount_sum_ratio']  = df['category_3_2_purchase_amount_sum'] / df['category_3_2_purchase_amount_sum_old']    \n",
    "    df['purchase_amount_sum_total'] = df['purchase_amount_sum'] + df['purchase_amount_sum_old']\n",
    "    df['purchase_amount_sum_ratio'] = df['purchase_amount_sum'] / df['purchase_amount_sum_old']\n",
    "    df['purchase_amount_pct_75_ratio'] = df['purchase_amount_pct_75'] / df['purchase_amount_pct_75_old']\n",
    "    df['purchase_amount_pct_25_ratio'] = df['purchase_amount_pct_25'] / df['purchase_amount_pct_25_old']\n",
    "    df['purchase_amount_inter_quartile'] = df['purchase_amount_pct_75'] - df['purchase_amount_pct_25']\n",
    "    df['purchase_amount_inter_quartile_old'] = df['purchase_amount_pct_75_old'] - df['purchase_amount_pct_25_old']\n",
    "    df['purchase_amount_inter_quartile_ratio'] = df['purchase_amount_inter_quartile'] / df['purchase_amount_inter_quartile_old']\n",
    "    df['purchase_amount_mean_total'] = df['purchase_amount_mean'] + df['purchase_amount_mean_old']\n",
    "    df['purchase_amount_mean_ratio'] = df['purchase_amount_mean'] / df['purchase_amount_mean_old']\n",
    "    df['purchase_amount_median_ratio'] = df['purchase_amount_median'] / df['purchase_amount_median_old']\n",
    "    df['purchase_amount_max_total'] = df['purchase_amount_max'] + df['purchase_amount_max_old']\n",
    "    df['purchase_amount_min_total'] = df['purchase_amount_min'] + df['purchase_amount_min_old']\n",
    "    df['purchase_amount_skew_ratio'] = df['purchase_amount_skew'] / df['purchase_amount_skew_old']\n",
    "    df['purchase_amount_before_after_ratio'] = df['purchase_amount_head_sum'] / df['purchase_amount_tail_sum_old']\n",
    "    df['purchase_amount_first_last_diff'] = df['purchase_amount_tail_sum'] - df['purchase_amount_head_max_old']\n",
    "    df['purchase_amount_fi_last_old_total'] = df['purchase_amount_tail_sum'] -  df['purchase_amount_head_sum']\n",
    "    df['purchase_amount_fi_last_new_total'] = df['purchase_amount_tail_sum_old'] -  df['purchase_amount_head_max_old']\n",
    "    df['purchase_amount_gmean_total'] = df['purchase_amount_gmean'] + df['purchase_amount_gmean_old']\n",
    "    df['purchase_amount_hmean_total'] = df['purchase_amount_hmean'] + df['purchase_amount_hmean_old']\n",
    "    df['purchase_amount_gmean_ratio'] = df['purchase_amount_gmean'] / df['purchase_amount_gmean_old']\n",
    "    df['purchase_amount_hmean_ratio'] = df['purchase_amount_hmean'] / df['purchase_amount_hmean_old']\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Redo some date features with observed time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['first_active_month', 'card_id', 'feature_1', 'feature_2', 'feature_3', 'target',\n",
       "       'transactions_count', 'subsector_id_nunique', 'merchant_id_nunique', 'merchant_category_id_nunique',\n",
       "       'purchase_Month_mean', 'purchase_Month_min', 'purchase_Month_max', 'purchase_Week_nunique',\n",
       "       'purchase_Week_mean', 'purchase_Week_min', 'purchase_Week_max', 'purchase_Dayofweek_mean',\n",
       "       'purchase_Dayofweek_min', 'purchase_Dayofweek_max', 'purchase_Day_nunique', 'purchase_Day_mean',\n",
       "       'purchase_Day_min', 'purchase_Day_max', 'purchase_Hour_nunique', 'purchase_Hour_mean',\n",
       "       'purchase_Hour_min', 'purchase_Hour_max', 'purchase_amount_sum', 'purchase_amount_pct_75',\n",
       "       'purchase_amount_pct_25', 'purchase_amount_mean', 'purchase_amount_median', 'purchase_amount_max',\n",
       "       'purchase_amount_min', 'purchase_amount_var', 'purchase_amount_skew', 'purchase_amount_head_sum',\n",
       "       'purchase_amount_head_max', 'purchase_amount_tail_sum', 'purchase_amount_tail_max',\n",
       "       'installments_sum', 'installments_pct_75', 'installments_pct_25', 'installments_mean',\n",
       "       'installments_median', 'installments_max', 'installments_min', 'installments_var',\n",
       "       'installments_skew', 'installments_head_sum', 'installments_head_max', 'installments_tail_sum',\n",
       "       'installments_tail_max', 'purchase_date_max', 'purchase_date_min', 'month_lag_max', 'month_lag_min',\n",
       "       'month_lag_mean', 'month_lag_var', 'month_lag_skew', 'month_diff_mean', 'month_diff_var',\n",
       "       'month_diff_skew', 'purchased_on_weekend_mean', 'category_1_mean', 'category_2_mean',\n",
       "       'category_3_mean', 'card_id_size', 'card_id_count', 'price_mean', 'price_max', 'price_min',\n",
       "       'price_var', 'Christmas_Day_2017_mean', 'Children_day_2017_mean', 'Black_Friday_2017_mean',\n",
       "       'Mothers_Day_2018_mean', 'duration_mean', 'duration_min', 'duration_max', 'duration_var',\n",
       "       'duration_skew', 'amount_month_ratio_mean', 'amount_month_ratio_min', 'amount_month_ratio_max',\n",
       "       'amount_month_ratio_var', 'amount_month_ratio_skew', 'purchase_date_successive_diff_mean',\n",
       "       'category_2_mean_mean', 'category_3_mean_mean', 'purchase_date_diff', 'purchase_date_average',\n",
       "       'purchase_date_uptonow', 'purchase_date_uptomin', 'inverse_avg_transactions_per_day',\n",
       "       'days_since_last_transaction', 'repurchase_merchant_rate', 'merchant_category_repurchase',\n",
       "       'avg_spend_per_merchant', 'avg_trans_per_merchant', 'avg_spend_per_transaction',\n",
       "       'transactions_count_old', 'subsector_id_nunique_old', 'merchant_id_nunique_old',\n",
       "       'merchant_category_id_nunique_old', 'purchase_Month_nunique', 'purchase_Month_mean_old',\n",
       "       'purchase_Month_min_old', 'purchase_Month_max_old', 'purchase_Week_nunique_old',\n",
       "       'purchase_Week_mean_old', 'purchase_Week_min_old', 'purchase_Week_max_old',\n",
       "       'purchase_Dayofweek_mean_old', 'purchase_Day_nunique_old', 'purchase_Day_mean_old',\n",
       "       'purchase_Day_min_old', 'purchase_Hour_nunique_old', 'purchase_Hour_mean_old',\n",
       "       'purchase_Hour_min_old', 'purchase_Hour_max_old', 'purchase_amount_sum_old',\n",
       "       'purchase_amount_pct_75_old', 'purchase_amount_pct_25_old', 'purchase_amount_mean_old',\n",
       "       'purchase_amount_median_old', 'purchase_amount_max_old', 'purchase_amount_min_old',\n",
       "       'purchase_amount_var_old', 'purchase_amount_skew_old', 'purchase_amount_head_sum_old',\n",
       "       'purchase_amount_head_max_old', 'purchase_amount_tail_sum_old', 'purchase_amount_tail_max_old',\n",
       "       'installments_sum_old', 'installments_pct_75_old', 'installments_pct_25_old', 'installments_mean_old',\n",
       "       'installments_median_old', 'installments_max_old', 'installments_min_old', 'installments_var_old',\n",
       "       'installments_skew_old', 'installments_head_sum_old', 'installments_head_max_old',\n",
       "       'installments_tail_sum_old', 'installments_tail_max_old', 'purchase_date_max_old',\n",
       "       'purchase_date_min_old', 'month_lag_max_old', 'month_lag_min_old', 'month_lag_mean_old',\n",
       "       'month_lag_var_old', 'month_lag_skew_old', 'month_diff_max', 'month_diff_min', 'month_diff_mean_old',\n",
       "       'month_diff_var_old', 'month_diff_skew_old', 'authorized_flag_mean', 'purchased_on_weekend_mean_old',\n",
       "       'category_1_mean_old', 'category_2_mean_old', 'category_3_mean_old', 'card_id_size_old',\n",
       "       'card_id_count_old', 'price_sum', 'price_mean_old', 'price_max_old', 'price_min_old', 'price_var_old',\n",
       "       'price_skew', 'Christmas_Day_2017_mean_old', 'Mothers_Day_2017_mean', 'fathers_day_2017_mean',\n",
       "       'Children_day_2017_mean_old', 'Valentine_Day_2017_mean', 'Black_Friday_2017_mean_old',\n",
       "       'Mothers_Day_2018_mean_old', 'duration_mean_old', 'duration_min_old', 'duration_max_old',\n",
       "       'duration_var_old', 'duration_skew_old', 'amount_month_ratio_mean_old', 'amount_month_ratio_min_old',\n",
       "       'amount_month_ratio_max_old', 'amount_month_ratio_var_old', 'amount_month_ratio_skew_old',\n",
       "       'purchase_date_successive_diff_mean_old', 'category_2_mean_mean_old', 'category_3_mean_mean_old',\n",
       "       'purchase_date_diff_old', 'purchase_date_average_old', 'purchase_date_uptonow_old',\n",
       "       'purchase_date_uptomin_old', 'inverse_avg_transactions_per_day_old',\n",
       "       'days_since_last_transaction_old', 'repurchase_merchant_rate_old', 'merchant_category_repurchase_old',\n",
       "       'avg_spend_per_merchant_old', 'avg_trans_per_merchant_old', 'avg_spend_per_transaction_old',\n",
       "       'category_1_0_purchase_amount_mean', 'category_1_1_purchase_amount_mean',\n",
       "       'category_1_0_purchase_amount_sum', 'category_1_1_purchase_amount_sum',\n",
       "       'category_2_1.0_purchase_amount_mean', 'category_2_2.0_purchase_amount_mean',\n",
       "       'category_2_3.0_purchase_amount_mean', 'category_2_4.0_purchase_amount_mean',\n",
       "       'category_2_5.0_purchase_amount_mean', 'category_2_1.0_purchase_amount_sum',\n",
       "       'category_2_2.0_purchase_amount_sum', 'category_2_3.0_purchase_amount_sum',\n",
       "       'category_2_4.0_purchase_amount_sum', 'category_2_5.0_purchase_amount_sum',\n",
       "       'category_3_0_purchase_amount_mean', 'category_3_1_purchase_amount_mean',\n",
       "       'category_3_2_purchase_amount_mean', 'category_3_0_purchase_amount_sum',\n",
       "       'category_3_1_purchase_amount_sum', 'category_3_2_purchase_amount_sum',\n",
       "       'authorized_flag_0_purchase_amount_mean', 'authorized_flag_1_purchase_amount_mean',\n",
       "       'authorized_flag_0_purchase_amount_sum', 'authorized_flag_1_purchase_amount_sum',\n",
       "       'category_1_0_purchase_amount_mean_old', 'category_1_1_purchase_amount_mean_old',\n",
       "       'category_1_0_purchase_amount_sum_old', 'category_1_1_purchase_amount_sum_old',\n",
       "       'category_2_1.0_purchase_amount_mean_old', 'category_2_2.0_purchase_amount_mean_old',\n",
       "       'category_2_3.0_purchase_amount_mean_old', 'category_2_4.0_purchase_amount_mean_old',\n",
       "       'category_2_5.0_purchase_amount_mean_old', 'category_2_1.0_purchase_amount_sum_old',\n",
       "       'category_2_2.0_purchase_amount_sum_old', 'category_2_3.0_purchase_amount_sum_old',\n",
       "       'category_2_4.0_purchase_amount_sum_old', 'category_2_5.0_purchase_amount_sum_old',\n",
       "       'category_3_0_purchase_amount_mean_old', 'category_3_1_purchase_amount_mean_old',\n",
       "       'category_3_2_purchase_amount_mean_old', 'category_3_0_purchase_amount_sum_old',\n",
       "       'category_3_1_purchase_amount_sum_old', 'category_3_2_purchase_amount_sum_old',\n",
       "       'authorized_flag_1_purchase_amount_mean_old', 'authorized_flag_1_purchase_amount_sum_old',\n",
       "       'observation_date_old', 'quarter', 'observed_elapsed_time', 'days_feature1', 'days_feature2',\n",
       "       'days_feature3', 'days_feature1_ratio', 'days_feature2_ratio', 'days_feature3_ratio', 'feature_sum',\n",
       "       'feature_mean', 'feature_max', 'feature_min', 'feature_var', 'card_id_total', 'card_id_count_total',\n",
       "       'card_id_count_ratio', 'purchase_amount_total', 'purchase_amount_total_mean',\n",
       "       'purchase_amount_total_max', 'purchase_amount_total_min', 'purchase_amount_sum_ratio',\n",
       "       'hist_first_buy', 'new_first_buy', 'hist_last_buy', 'new_last_buy', 'purchased_before_issue',\n",
       "       'month_diff_mean_total', 'month_diff_ratio', 'month_lag_mean_total', 'month_lag_max_total',\n",
       "       'month_lag_min_total', 'category_1_mean_total', 'installments_total', 'installments_mean_total',\n",
       "       'installments_max_total', 'installments_ratio', 'price_total', 'duration_mean_total',\n",
       "       'duration_min_total', 'duration_max_total', 'amount_month_ratio_mean_total',\n",
       "       'amount_month_ratio_min_total', 'amount_month_ratio_max_total', 'CLV', 'CLV_old', 'CLV_ratio',\n",
       "       'category_1_0_purchase_amount_mean_total', 'category_1_1_purchase_amount_mean_total',\n",
       "       'category_1_0_purchase_amount_sum_total', 'category_1_1_purchase_amount_sum_total',\n",
       "       'category_2_1.0_purchase_amount_mean_total', 'category_2_2.0_purchase_amount_mean_total',\n",
       "       'category_2_3.0_purchase_amount_mean_total', 'category_2_4.0_purchase_amount_mean_total',\n",
       "       'category_2_5.0_purchase_amount_mean_total', 'category_2_1.0_purchase_amount_sum_total',\n",
       "       'category_2_2.0_purchase_amount_sum_total', 'category_2_3.0_purchase_amount_sum_total',\n",
       "       'category_2_4.0_purchase_amount_sum_total', 'category_2_5.0_purchase_amount_sum_total',\n",
       "       'category_3_0_purchase_amount_mean_total', 'category_3_1_purchase_amount_mean_total',\n",
       "       'category_3_2_purchase_amount_mean_total', 'category_3_0_purchase_amount_sum_total',\n",
       "       'category_3_1_purchase_amount_sum_total', 'category_3_2_purchase_amount_sum_total',\n",
       "       'category_1_0_purchase_amount_mean_ratio', 'category_1_1_purchase_amount_mean_ratio',\n",
       "       'category_1_0_purchase_amount_sum_ratio', 'category_1_1_purchase_amount_sum_ratio',\n",
       "       'category_2_1.0_purchase_amount_mean_ratio', 'category_2_2.0_purchase_amount_mean_ratio',\n",
       "       'category_2_3.0_purchase_amount_mean_ratio', 'category_2_4.0_purchase_amount_mean_ratio',\n",
       "       'category_2_5.0_purchase_amount_mean_ratio', 'category_2_1.0_purchase_amount_sum_ratio',\n",
       "       'category_2_2.0_purchase_amount_sum_ratio', 'category_2_3.0_purchase_amount_sum_ratio',\n",
       "       'category_2_4.0_purchase_amount_sum_ratio', 'category_2_5.0_purchase_amount_sum_ratio',\n",
       "       'category_3_0_purchase_amount_mean_ratio', 'category_3_1_purchase_amount_mean_ratio',\n",
       "       'category_3_2_purchase_amount_mean_ratio', 'category_3_0_purchase_amount_sum_ratio',\n",
       "       'category_3_1_purchase_amount_sum_ratio', 'category_3_2_purchase_amount_sum_ratio',\n",
       "       'purchase_amount_sum_total', 'purchase_amount_pct_75_ratio', 'purchase_amount_pct_25_ratio',\n",
       "       'purchase_amount_inter_quartile', 'purchase_amount_inter_quartile_old',\n",
       "       'purchase_amount_inter_quartile_ratio', 'purchase_amount_mean_total', 'purchase_amount_mean_ratio',\n",
       "       'purchase_amount_median_ratio', 'purchase_amount_max_total', 'purchase_amount_min_total',\n",
       "       'purchase_amount_skew_ratio', 'purchase_amount_before_after_ratio', 'purchase_amount_first_last_diff',\n",
       "       'purchase_amount_fi_last_old_total', 'purchase_amount_fi_last_new_total', 'outliers'], dtype=object)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially we calculated `purchase_date_uptonow` and `purchase_date_uptomin` as the age of the max and minimum transaction amount till date by the customer. This is a dwindling number as the result will change as time progresses. Let's calculate it against some fixed date. What about our `observation_date` aka reference date. This yielded some boost in the leaderboard personally for me."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in [train_df, test_df]:\n",
    "    df['purchase_date_uptonow'] = (df['observation_date_old'] - df['purchase_date_max']).dt.days\n",
    "    df['purchase_date_uptomin'] = (df['observation_date_old'] - df['purchase_date_min']).dt.days\n",
    "    df['purchase_date_uptonow_old'] = (df['observation_date_old'] - df['purchase_date_max_old']).dt.days\n",
    "    df['purchase_date_uptomin_old'] = (df['observation_date_old'] - df['purchase_date_min_old']).dt.days\n",
    "#     df.drop(['days_since_last_transaction', 'days_since_last_transaction_old'], inplace=True, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 386), (123623, 384))"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mark the outliers"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtUAAAHzCAYAAADvruHdAAAgAElEQVR4Ae3dC/htZV0n8C8jXhgNQTzxEFDayFjqjNcBHKupSLlkQc+k6ZSSkadGLe1pntJuTl4a7a5lNiclwRyNMR2oMGLM7qlgIuYtkDQggaMglo6axTw/e9fTcvc//7P/+N977f3uz/s827XXuy7vej/vOfL9r/Ou9U8UAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgRWQOBdSb56Ba5jykv4piTXJvm7JA+e8kJa2/89ya+twHW4BAIECBAgQIAAgSQfSPJ1MxLfnuSPZ+oOtnqvJLclOfRgO67p9vcnOXOba6++32eb7bu9aRyqe7ffbTvnI0BgTQT+1Zpcp8skQIDAOglMHda/JEndsVd2LjD12O38ih1BgMBKCAjVKzEMLoIAgV0UGN/NPjHJ5Uk+luTGJD/b2vnDtvxomyLx8CT1/4c/kuSDSW5Kcn6Su4+u64lt20eS/OjMXfO6E/vaNsWh2qq759X2nyWpNj6U5BeT3Gl0vrpb/JQkVyX52yTPTfJvkvxpu94LZvYfHXrAa71z688dkrwjSd2x3knZzuC3k3zPzMmuTFJTTaq8qE05qf6/LclXtvrZxaz9f0pyc5J/N9rxC5N8IsmeUd3wte6w/0GSW5N8OMmvDxuS3D/Jpe18Nd4/1LaVy88n+Zv2qe9VV6WmCl2X5AeT3JDkV1v9o5Nc0cavxuTft3oLAgQIECBAgMDaC4wD89CZ2ekf430q1D6h7Xi3JCe371tNQfiOJFcn+dIkte/rkryy7X+/Fla/ogXdn07y96OpKBWqa/2sFs4PS/LQ1l7d+az23pPkGcNFt+knFyY5vIXBTyV5Y2u/wvy7k5w92n/8dbtrrf0ONr3jQNu3O+9jk7xldBEPTFI/YAw/KHxbkqPalJrvbwH1Lm3/g03/+KUkLxyd++lJfnO0Pv766iQ/3Jzr/DUmVb6g/fBSbVd9rZ/Utj0nyZuTVFivoF4huX6IqVKh+jOt/QraNXY1D71+sKrj6weUGof6czUE8XaoBQECBAgQIEBgPQUq2NTDd3X3d/jUHc3xnOpxqK67oj+e5J4z3d0qVFegrTvHQ7lvC8oVin8sSYW5ofzrJJ+eCdXDHdhhn9llBerXjyor2D5itF53d+tu6VB+pt1dHdbHy+2utfY7UGgeznGg7dudt4LqLUlOaCepHywqDB+o1L4VvKscLFRXeP3rJIe0/etfFyrEb1XqXxD2JTluZuPjk7x9pm5YrTv2ZwwrSU5tIbmqKlTXWA4/AFTdS0ehezjsfUnqrrpCgACBLQVM/9iSRSUBAissUHeDjxh9xkF49rLPSfJvk7w3yWVJ6p/0D1S+qE3vGLbXNJAK1EcnqW31No2hVJCvu7TjMt5e9dXub7U7tjUl4ie2CPc1RWEo/69NURmv1x3zrcp217rV/vPWbXfeT7apFnVHuv7bUSF2uJNf5/9v7W58TcuoH3jqbvvsDzMHuo66A16mFXC/rD1EedEBdv6BFr7f2uaN1931KsdvM91lq35V3VD2J6n+DaXmpNcd7+EHt1rW+cfHDPtaEiBA4LMCQrU/CAQI9CxQ85Ur/NU/+9f0gpr3fNd2J3e23zXftsLUUL64TQuo4Ftzosd3RmuKQE11GJe6+zsudbezwnzd2a0pHjW/d7gTO97v9nzf7lpvz/mGYw523vOSfGuSU1oIruk1VWr+dIXdurt8ZPuBp8L1Vv2ddWqnSJ27AntN16lxGofcYZ9a1rznJ7eA+13tbnnNs64famrqzlZlq35V3VBmr6nO9fzRD271Q1z968T4XyuGYy0JECDwWQGh2h8EAgR6FqiQVnNo/7Hdday+1ve6M1nLcQirwPR9Se7d5lTXneV6CK7m21bI+4Yk/7HNIa7pDFsFxrFlzemtO9Q1XaXuvv7X8cbP8/t21zrvqWsudE15GD41d/hg560QXW41NWV8l7r6Wk7lOkyXqR8ktipb2dd+9R7reuixxqymeByoPGb0A05NMalAXNdU/ypwTJu3XnOfx3Oqq1/1EGr9Wai75zWdZ7v3Zv9Kku9uc6prnOsHsa9v5zzQdaknQGDDBYTqDf8DoPsEOhc4rU0RqGBbb6d4XJKaZlFTDepO5J+0sF0PMJ7bgmLNjf6rdqd0eNtFvZ6uvr+m3bWu89WDbPVw4YFKTYf4L+3NHhXSxm+pONAx89Zvd63znqP6VBbD50kHMRjOW4G33tQxDqWXJPmdJH/ZptDUXebZ6TDD8VvZ17ba/89bSP6jYectlv+hPTBZY1BTROqhxmua8yPbDz91N7v+leJr2vHPa2+BqbeVvLO1U3UHKjWnu+6G1xtbKrjXA6z1QKxCgAABAgQIECCwiwI117nuzNZd7U0r9WrB8YOhu9n/+mFhu7C7m205FwECBAgQIECAwAQCNf2j5tXWVIBfbm+aONgUkAkuc6FNVv/r1XQVrHe71BtZ6oHATfxBZbctnY8AAQIECBAgsLICL2uhrx7Aq1fP1Sv3NqnUa+g+nqTerV3zpnez1DujazpHvX9aIUCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECyxZY24dsjjrqqNvuda96rkUhQIAAAQIECBAgsBiBt73tbR9u77nftoHdfthk28Z2c2MF6ssvr1eJKgQIECBAgAABAgQWI3DIIYd8cJ4z++Uv8yjZhwABAgQIECBAgMA2AkL1Njg2ESBAgAABAgQIEJhHQKieR8k+BAgQIECAAAECBLYREKq3wbGJAAECBAgQIECAwDwCQvU8SvYhQIAAAQIECBAgsI2AUL0Njk0ECBAgQIAAAQIE5hEQqudRsg8BAgQIECBAgACBbQSE6m1wbCJAgAABAgQIECAwj4BQPY+SfQgQIECAAAECBAhsIyBUb4NjEwECBAgQIECAAIF5BITqeZTsQ4AAAQIECBAgQGAbAaF6GxybCBAgQIAAAQIECMwjIFTPo2QfAgQIECBAgAABAtsIzBOq75vkitHnY0mekeQeSS5NclVbHtnaOSTJi5NcneTKJA8ZtX9227+Oqe9DeWiSd7Zj6tg6h0KAAAECBAgQIEBgLQTmCdXvS/Kg9qnw+4kkr0/yzCRvTHJCW9Z6ldNbXdXvTfLSVl8h/NlJTkpyYvs+BPHa58mj405rx1gQIECAAAECBAgQWHmBeUL1uBOnJHl/kg8mOTPJeW1jLc9q36v+/CS3JXlzkiOSHJPk1HZH++Ykt7TvFZ5r2+Ft3zqmjh3ONW7bdwIECBAgQIAAAQIrKbDTUP24JK9uPTk6yYfa9xuS1HqVY5Nc277X4rpWt1197TOUYf9h3ZIAAQIECBAgQIDASgvsJFTfKck3JvnfW/So7jDXZ9GlppNcXp/9+/cvui3nJ0CAAAECBAgQIDCXwE5Cdc2V/vMkN7Yz17KmblSp5U3t+/VJjm/fa3Fckqrbrr72Gcqw/7A+Xu5L8rD67NmzZ1zvOwECBAgQIECAAIHJBA7dQcuPH039qMMuam/weEFbXtjOVfVPS/Ka9lDirW2ayCVJfiLJ8HDio5I8K0nNsa43ipyc5C1JnpjkF3ZwXXYlQIAAAQLTCeyr+z07KHvrH10VAgR6E5g3VN81ySOTfNcIoML0BUnOaQ8uPrZtuzjJGe31ePWmkCe1+grPz01yWVt/TgvUtfqUJK9IcliSN7RP282CAAECBAgQIECAwGoLzBuqP57kqJmufCRJvQ1kttTc6qfOVrb1c5PUZ7bUPOkHzFZaJ0CAAAECBAgQILAOAjuZU70O/XGNBAgQIECAAAECBJYuIFQvnVyDBAgQIECAAAECvQkI1b2NqP4QIECAAAECBAgsXUCoXjq5BgkQIECAAAECBHoTEKp7G1H9IUCAAAECBAgQWLqAUL10cg0SIECAAAECBAj0JiBU9zai+kOAAAECBAgQILB0AaF66eQaJECAAAECBAgQ6E1AqO5tRPWHAAECBAgQIEBg6QJC9dLJNUiAAAECBAgQINCbgFDd24jqDwECBAgQIECAwNIFhOqlk2uQAAECBAgQIECgNwGhurcR1R8CBAgQIECAAIGlCwjVSyfXIAECBAgQIECAQG8CQnVvI6o/BAgQIECAAAECSxcQqpdOrkECBAgQIECAAIHeBITq3kZUfwgQIECAAAECBJYuIFQvnVyDBAgQIECAAAECvQkI1b2NqP4QIECAAAECBAgsXUCoXjq5BgkQIECAAAECBHoTEKp7G1H9IUCAAAECBAgQWLqAUL10cg0SIECAAAECBAj0JiBU9zai+kOAAAECBAgQILB0AaF66eQaJECAAAECBAgQ6E1AqO5tRPWHAAECBAgQIEBg6QJC9dLJNUiAAAECBAgQINCbgFDd24jqDwECBAgQIECAwNIFhOqlk2uQAAECBAgQIECgNwGhurcR1R8CBAgQIECAAIGlCwjVSyfXIAECBAgQIECAQG8CQnVvI6o/BAgQIECAAAECSxcQqpdOrkECBAgQIECAAIHeBITq3kZUfwgQIECAAAECBJYuIFQvnVyDBAgQIECAAAECvQkI1b2NqP4QIECAAAECBAgsXUCoXjq5BgkQIECAAAECBHoTEKp7G1H9IUCAAAECBAgQWLqAUL10cg0SIECAAAECBAj0JiBU9zai+kOAAAECBAgQILB0AaF66eQaJECAAAECBAgQ6E1AqO5tRPWHAAECBAgQIEBg6QJC9dLJNUiAAAECBAgQINCbgFDd24jqDwECBAgQIECAwNIFhOqlk2uQAAECBAgQIECgNwGhurcR1R8CBAgQIECAAIGlCwjVSyfXIAECBAgQIECAQG8CQnVvI6o/BAgQIECAAAECSxcQqpdOrkECBAgQIECAAIHeBITq3kZUfwgQIECAAAECBJYuIFQvnVyDBAgQIECAAAECvQkI1b2NqP4QIECAAAECBAgsXWDeUH1EktcmeW+S9yR5eJJ7JLk0yVVteWS7+kOSvDjJ1UmuTPKQUa/ObvvXMfV9KA9N8s52TB1b51AIECBAgAABAgQIrIXAvKH6RUl+J8mXJXlgC9bPTPLGJCe0Za1XOb3VVf3eJC9t9RXCn53kpCQntu9DEK99njw67rR2jAUBAgQIECBAgACBlReYJ1TfPclXJXl5682nk3w0yZlJzmt1tTyrfa/685PcluTNSeou9zFJTm13tG9Ockv7XuG5th3e9q1j6tjhXCsP6AIJECBAgAABAgQIzBOq751kf5JfTfL2JC9LctckRyf5UCO8oa3X6rFJrh3RXtfqtquvfYYy7D+sj5d15/vy+uzfX5ekECBAgAABAgQIEJheYJ5QfWibF11TNB6c5ONJhqkeQw/qDnN9Fl32JXlYffbs2bPotpyfAAECBAgQIECAwFwC84TqunNcn7e0M9YDi/Xw4Y1t6kZV1xSOm9r265McP2r9uCRVt1197TOUYf9h3ZIAAQIECBAgQIDASgvME6prakdN57hv68kpSd6d5KLRGzzqTR4Xtu1V/8T2Bo+Tk9zapolckuRRSerhxPrU96qrKSQfS1L71ls/6tjhXCuN5+IIECBAgAABAgQIlEBN7ZinfE+SVyW5U5JrkjwpSQXyC5Kck+SDSR7bTnRxkjPa6/E+0fatTfWA4nOTXNb2e06rq9WnJHlFksOSvKF92m4WBAgQIECAAAECBFZbYN5QfUWbyzzbm7prPVtqbvVTZyvb+rlJ6jNb6uHDB8xWWidAgAABAgQIECCwDgLzTP9Yh364RgIECBAgQIAAAQKTCQjVk9FrmAABAgQIECBAoBcBobqXkdQPAgQIECBAgACByQSE6snoNUyAAAECBAgQINCLgFDdy0jqBwECBAgQIECAwGQCQvVk9BomQIAAAQIECBDoRUCo7mUk9YMAAQIECBAgQGAyAaF6MnoNEyBAgAABAgQI9CIgVPcykvpBgAABAgQIECAwmYBQPRm9hgkQIECAAAECBHoREKp7GUn9IECAAAECBAgQmExAqJ6MXsMECBAgQIAAAQK9CAjVvYykfhAgQIAAAQIECEwmIFRPRq9hAgQIECBAgACBXgSE6l5GUj8IECBAgAABAgQmExCqJ6PXMAECBAgQIECAQC8CQnUvI6kfBAgQIECAAAECkwkI1ZPRa5gAAQIECBAgQKAXAaG6l5HUDwIECBAgQIAAgckEhOrJ6DVMgAABAgQIECDQi4BQ3ctI6gcBAgQIECBAgMBkAkL1ZPQaJkCAAAECBAgQ6EVAqO5lJPWDAAECBAgQIEBgMgGhejJ6DRMgQIAAAQIECPQiIFT3MpL6QYAAAQIECBAgMJmAUD0ZvYYJECBAgAABAgR6ERCqexlJ/SBAgAABAgQIEJhMQKiejF7DBAgQIECAAAECvQgI1b2MpH4QIECAAAECBAhMJiBUT0avYQIECBAgQIAAgV4EhOpeRlI/CBAgQIAAAQIEJhMQqiej1zABAgQIECBAgEAvAkJ1LyOpHwQIECBAgAABApMJCNWT0WuYAAECBAgQIECgFwGhupeR1A8CBAgQIECAAIHJBITqyeg1TIAAAQIECBAg0IuAUN3LSOoHAQIECBAgQIDAZAJC9WT0GiZAgAABAgQIEOhFQKjuZST1gwABAgQIECBAYDIBoXoyeg0TIECAAAECBAj0IiBU9zKS+kGAAAECBAgQIDCZgFA9Gb2GCRAgQIAAAQIEehEQqnsZSf0gQIAAAQIECBCYTEConoxewwQIECBAgAABAr0ICNW9jKR+ECBAgAABAgQITCYgVE9Gr2ECBAgQIECAAIFeBITqXkZSPwgQIECAAAECBCYTEKono9cwAQIECBAgQIBALwJCdS8jqR8ECBAgQIAAAQKTCQjVk9FrmAABAgQIECBAoBeBeUP1B5K8M8kVSS5vnb9HkkuTXNWWR7b6Q5K8OMnVSa5M8pAR1tlt/zqmvg/loe38dUwdW+dQCBAgQIAAAQIECKyFwLyhujrzNUkelORhrWfPTPLGJCe0Za1XOb3VVf3eJC9t9RXCn53kpCQntu9DEK99njw67rR2jAUBAgQIECBAgACBlRfYSaie7cyZSc5rlbU8q32v+vOT3JbkzUmOSHJMklPbHe2bk9zSvld4rm2Ht33rmDp2ONdsm9YJECBAgAABAgQIrJzAvKG6wu7vJnlbu/tcHTk6yYdaj25o67V6bJJrRz29rtVtV1/7DGXYf1i3JECAAAECBAgQILDSAofOeXVfkeT6JF/Y7jC/d+a4Ct31WXSp6ST1yf79+xfdlvMTIECAAAECBAgQmEtg3jvVFair3JTk9W1O9I1t6kbV1xSO2lal9j2+fa/Fca1uu/raZyjD/sP6eLmvzel+2J49e8b1vhMgQIAAAQIECBCYTGCeUH3XJF/QrrC+PyrJXyS5aPQGj3qTx4Vtn6p/YnuDx8lJbm3TRC5px9bDifWp81RdTSH5WJLat976UccO55oMRsMECBAgQIAAAQIE5hWYZ/pHzZ2uu9NVav//leR3klyW5IIk5yT5YJLHtn0uTnJGe6XeJ5I8qdXXA4rPbcdV1XOSVF2VpyR5RZLDkryhfdomCwIECBAgQIAAAQKrLTBPqL4myQO36MZHkpyyRX3NrX7qFvVVdW77zG6ud18/YLbSOgECBAgQIECAAIF1EJhn+sc69MM1EiBAgAABAgQIEJhMQKiejF7DBAgQIECAAAECvQgI1b2MpH4QIECAAAECBAhMJiBUT0avYQIECBAgQIAAgV4EhOpeRlI/CBAgQIAAAQIEJhMQqiej1zABAgQIECBAgEAvAkJ1LyOpHwQIECBAgAABApMJCNWT0WuYAAECBAgQIECgFwGhupeR1A8CBAgQIECAAIHJBITqyeg1TIAAAQIECBAg0IuAUN3LSOoHAQIECBAgQIDAZAJC9WT0GiZAgAABAgQIEOhFQKjuZST1gwABAgQIECBAYDIBoXoyeg0TIECAAAECBAj0IiBU9zKS+kGAAAECBAgQIDCZgFA9Gb2GCRAgQIAAAQIEehEQqnsZSf0gQIAAAQIECBCYTEConoxewwQIECBAgAABAr0ICNW9jKR+ECBAgAABAgQITCYgVE9Gr2ECBAgQIECAAIFeBITqXkZSPwgQIECAAAECBCYTEKono9cwAQIECBAgQIBALwJCdS8jqR8ECBAgQIAAAQKTCQjVk9FrmAABAgQIECBAoBcBobqXkdQPAgQIECBAgACByQSE6snoNUyAAAECBAgQINCLgFDdy0jqBwECBAgQIECAwGQCQvVk9BomQIAAAQIECBDoRUCo7mUk9YMAAQIECBAgQGAyAaF6MnoNEyBAgAABAgQI9CIgVPcykvpBgAABAgQIECAwmYBQPRm9hgkQIECAAAECBHoREKp7GUn9IECAAAECBAgQmExAqJ6MXsMECBAgQIAAAQK9CAjVvYykfhAgQIAAAQIECEwmIFRPRq9hAgQIECBAgACBXgSE6l5GUj8IECBAgAABAgQmExCqJ6PXMAECBAgQIECAQC8CQnUvI6kfBAgQIECAAAECkwkI1ZPRa5gAAQIECBAgQKAXAaG6l5HUDwIECBAgQIAAgckEhOrJ6DVMgAABAgQIECDQi4BQ3ctI6gcBAgQIECBAgMBkAkL1ZPQaJkCAAAECBAgQ6EVAqO5lJPWDAAECBAgQIEBgMgGhejJ6DRMgQIAAAQIECPQiIFT3MpL6QYAAAQIECBAgMJmAUD0ZvYYJECBAgAABAgR6ERCqexlJ/SBAgAABAgQIEJhMQKiejF7DBAgQIECAAAECvQgcuoOO3CHJ5UmuT/LoJPdO8pokRyV5W5InJPl0kjsnOT/JQ5N8JMm3JPlAa+dZSc5J8g9JvjfJJa3+tCQvSlJtvCzJC3ZwXXYlQIAAAQK7I7Bv3+6cx1kIENg4gZ3cqX56kveMhF6Y5OeS3CfJLS0s1+YKzbVe9bW99qtyvySPS3L/JBWif6mF6ArSL0lyetvn8W3ZDrMgQIAAAQIECBAgsNoC84bq45J8fbuLXD06JMnXJnlt6955Sc5q389MUutVavspbf+qrzvbn0ryV0muTnJi+9T3a9qd7tqn9lUIECBAgAABAgQIrIXAvKH655P8QJJ/bL2qKR8fTfKZtn5dkmPb91pe277X9lvbFJFxfW0ejjlQfTuFBQECBAgQIECAAIHVFpgnVNf86ZvavOmpe7O3zeu+fP/+/VNfi/YJECBAgAABAgQIfFZgngcVH5HkG5OckeQuSQ5vDxUekaSOr7vRNT2kHmCsUsvj253o2n739sDiUN92+5xjav+hjM811A3LeoLks0+R7Nmz57ah0pIAAQIECBAgQIDAlALz3KmuN3ZU0L1Xe9Dw95J8a5I3JfnmdvFnJ7mwfb8oSa1Xqe21fwXgqq8HFevtIPXmkBOSvDXJZe171d2p7VP7KgQIECBAgAABAgTWQmCeO9UH6sgPtgcPn5fk7Ule3nas5Svbg4g3t5Bcm96V5IIk7253t5/aXq1X257WXq9XbwI5t+17oHbVEyBAgAABAgQIEFgpgZ2G6t9PUp8q9baOenvHbPlkksfMVrb15yepz2y5OEl9FAIECBAgQIAAAQJrJzDP9I+165QLJkCAAAECBAgQILBMAaF6mdraIkCAAAECBAgQ6FJAqO5yWHWKAAECBAgQIEBgmQJC9TK1tUWAAAECBAgQINClgFDd5bDqFAECBAgQIECAwDIFhOplamuLAAECBAgQIECgSwGhusth1SkCBAgQIECAAIFlCgjVy9TWFgECBAgQIECAQJcCQnWXw6pTBAgQIECAAAECyxQQqpeprS0CBAgQIECAAIEuBYTqLodVpwgQIECAAAECBJYpIFQvU1tbBAgQIECAAAECXQoI1V0Oq04RIECAAAECBAgsU0CoXqa2tggQIECAAAECBLoUEKq7HFadIkCAAAECBAgQWKaAUL1MbW0RIECAAAECBAh0KSBUdzmsOkWAAAECBAgQILBMAaF6mdraIkCAAAECBAgQ6FJAqO5yWHWKAAECBAgQIEBgmQKHLrMxbREgQIAAgY0X2LdvZwR79+5sf3sTIDCJgDvVk7BrlAABAgQIECBAoCcBobqn0dQXAgQIECBAgACBSQSE6knYNUqAAAECBAgQINCTgFDd02jqCwECBAgQIECAwCQCQvUk7BolQIAAAQIECBDoSUCo7mk09YUAAQIECBAgQGASAaF6EnaNEiBAgAABAgQI9CQgVPc0mvpCgAABAgQIECAwiYBQPQm7RgkQIECAAAECBHoSEKp7Gk19IUCAAAECBAgQmERAqJ6EXaMECBAgQIAAAQI9CQjVPY2mvhAgQIAAAQIECEwiIFRPwq5RAgQIECBAgACBngSE6p5GU18IECBAgAABAgQmERCqJ2HXKAECBAgQIECAQE8CQnVPo6kvBAgQIECAAAECkwgI1ZOwa5QAAQIECBAgQKAnAaG6p9HUFwIECBAgQIAAgUkEhOpJ2DVKgAABAgQIECDQk4BQ3dNo6gsBAgQIECBAgMAkAkL1JOwaJUCAAAECBAgQ6ElAqO5pNPWFAAECBAgQIEBgEgGhehJ2jRIgQIAAAQIECPQkIFT3NJr6QoAAAQIECBAgMImAUD0Ju0YJECBAgAABAgR6EhCqexpNfSFAgAABAgQIEJhEQKiehF2jBAgQIECAAAECPQkI1T2Npr4QIECAAAECBAhMIiBUT8KuUQIECBAgQIAAgZ4EhOqeRlNfCBAgQIAAAQIEJhEQqidh1ygBAgQIECBAgEBPAvOE6rskeWuSdyR5V5IfbwD3TvKWJFcn+fUkd2r1d27rVV/b7zUCe1bb/31JTh3Vn5ak6uqYZ47qfSVAgF12z1cAABsiSURBVAABAgQIECCw8gLzhOpPJfnaJA9M8qAkFYBPTvLCJD+X5D5JbklyTuttLWu96mt77Vflfkkel+T+7Ry/lOQO7fOSJKe3fR7flu0wCwIECBAgQIAAAQKrLTBPqL4tyd+1btwxSX2qroL2a1v9eUnOat/PTFLrVWr7KUkOSVL1r0lSIf2v2l3pE5PUp+5QX5Pk022f2lchQIAAAQIECBAgsBYC84Tq6kjdUb4iyU1JLk3y/iQfTfKZ1svrkhzbvtfy2va9tt+a5Ki2faivzcMx4/3H9e0UFgQIECBAgAABAgRWW2DeUP0PberHce3O8pdN1K29SS6vz/79+ye6BM0SIECAAAECBAgQ+FyBeUP1cFTdnX5TkocnOSLJoW1Dhe3r2/daHt++1/a7J/lI2z7U1+bhmPH+4/p2is9Z7EvysPrs2bPnczZYIUCAAAECBAgQIDCVwDyhutJrBegqhyV5ZJL3tHD9za3+7CQXtu8XJan1KrX999oc7KqvBxXr7SD15pAT2ltFLmvfq67eIFL71L4KAQIECBAgQIAAgbUQGO40b3exx7QHD2tedYXwC5L8VpJ3t4cKn5fk7Ule3k5Sy1e2hw9vbiG5NtXr+OrYOq7mWj81SU0rqfK0JJe0udvntn3bJgsCBAgQIECAAAECqy0wT6i+MsmDt+hGva2j3twxWz6Z5DGzlW39+UnqM1suTlIfhQABAgQIECBAgMDaCcwz/WPtOuWCCRAgQIAAAQIECCxTQKhepra2CBAgQIAAAQIEuhQQqrscVp0iQIAAAQIECBBYpoBQvUxtbREgQIAAAQIECHQpIFR3Oaw6RYAAAQIECBAgsEwBoXqZ2toiQIAAAQIECBDoUkCo7nJYdYoAAQIECBAgQGCZAkL1MrW1RYAAAQIECBAg0KWAUN3lsOoUAQIECBAgQIDAMgWE6mVqa4sAAQIECBAgQKBLAaG6y2HVKQIECBAgQIAAgWUKCNXL1NYWAQIECBAgQIBAlwJCdZfDqlMECBAgQIAAAQLLFBCql6mtLQIECBAgQIAAgS4FhOouh1WnCBAgQIAAAQIElikgVC9TW1sECBAgQIAAAQJdCgjVXQ6rThEgQIAAAQIECCxTQKhepra2CBAgQIAAAQIEuhQQqrscVp0iQIAAAQIECBBYpoBQvUxtbREgQIAAAQIECHQpIFR3Oaw6RYAAAQIECBAgsEwBoXqZ2toiQIAAAQIECBDoUkCo7nJYdYoAAQIECBAgQGCZAkL1MrW1RYAAAQIECBAg0KWAUN3lsOoUAQIECBAgQIDAMgWE6mVqa4sAAQIECBAgQKBLAaG6y2HVKQIECBAgQIAAgWUKCNXL1NYWAQIECBAgQIBAlwJCdZfDqlMECBAgQIAAAQLLFBCql6mtLQIECBAgQIAAgS4FhOouh1WnCBAgQIAAAQIElikgVC9TW1sECBAgQIAAAQJdCgjVXQ6rThEgQIAAAQIECCxTQKhepra2CBAgQIAAAQIEuhQQqrscVp0iQIAAAQIECBBYpoBQvUxtbREgQIAAAQIECHQpIFR3Oaw6RYAAAQIECBAgsEwBoXqZ2toiQIAAAQIECBDoUkCo7nJYdYoAAQIECBAgQGCZAkL1MrW1RYAAAQIECBAg0KWAUN3lsOoUAQIECBAgQIDAMgWE6mVqa4sAAQIECBAgQKBLAaG6y2HVKQIECBAgQIAAgWUKCNXL1NYWAQIECBAgQIBAlwJCdZfDqlMECBAgQIAAAQLLFBCql6mtLQIECBAgQIAAgS4FhOouh1WnCBAgQIAAAQIElikgVC9TW1sECBAgQIAAAQJdCgjVXQ6rThEgQIAAAQIECCxTQKhepra2CBAgQIAAAQIEuhSYJ1Qfn+RNSd6d5F1Jnt4k7pHk0iRXteWRrf6QJC9OcnWSK5M8ZCR3dtu/jqnvQ3lokne2Y+rYOodCgAABAgQIECBAYC0E5gnVn0ny/Unul+TkJE9t35+Z5I1JTmjLWq9yequr+r1JXtrqK4Q/O8lJSU5s34cgXvs8eXTcae0YCwIECBAgQIAAAQIrLzBPqP5Qkj9vPfnbJO9JcmySM5Oc1+preVb7XvXnJ7ktyZuTHJHkmCSntjvaNye5pX2v8FzbDm/71jF17HCulQd0gQQIECBAgAABAgTmCdVjpXsleXCStyQ5OkkF7io3tPX6XoH72n+q/uz/XtfqtquvfYYy7D+sWxIgQIAAAQIECBBYaYFDd3B1d0vyG0mekeRjM8fVHeb6LLrUdJL6ZP/+/Ytuy/kJECBAgAABAgQIzCUw753qO7ZA/aokr2tnvrFN3ajVmsJxU6u/Pkk93DiU45JU3Xb1tc9Qhv2H9fFyX5KH1WfPnj3jet8JECBAgAABAgQITCYwT6iuN3G8vM2l/tnRlV40eoNHvcnjwrat6p/Y3uBRDzbe2qaJXJLkUUnq4cT61Peqqykkdee79q226tjhXKPmfCVAgAABAgQIECCwmgLzTP94RJIntFfeXdG68UNJXpDkgiTnJPlgkse2bRcnOaO9Hu8TSZ7U6usBxecmuaytPydJ1VV5SpJXJDksyRvap22yIECAAAECBAgQILDaAvOE6j/e5r3Rp2zRvZpbXa/d26qcm6Q+s+XyJA+YrbROgAABAgQIECBAYB0E5pn+sQ79cI0ECBAgQIAAAQIEJhMQqiej1zABAgQIECBAgEAvAkJ1LyOpHwQIECBAgAABApMJCNWT0WuYAAECBAgQIECgFwGhupeR1A8CBAgQIECAAIHJBITqyeg1TIAAAQIECBAg0IuAUN3LSOoHAQIECBAgQIDAZAJC9WT0GiZAgAABAgQIEOhFQKjuZST1gwABAgQIECBAYDIBoXoyeg0TIECAAAECBAj0IiBU9zKS+kGAAAECBAgQIDCZgFA9Gb2GCRAgQIAAAQIEehEQqnsZSf0gQIAAAQIECBCYTEConoxewwQIECBAgAABAr0ICNW9jKR+ECBAgAABAgQITCYgVE9Gr2ECBAgQIECAAIFeBITqXkZSPwgQIECAAAECBCYTEKono9cwAQIECBAgQIBALwKH9tIR/SBAgAABAv9CYN++f1GlggABAosQcKd6EarOSYAAAQIECBAgsFECQvVGDbfOEiBAgAABAgQILEJAqF6EqnMSIECAAAECBAhslIBQvVHDrbMECBAgQIAAAQKLEBCqF6HqnAQIECBAgAABAhslIFRv1HDrLAECBAgQIECAwCIEhOpFqDonAQIECBAgQIDARgkI1Rs13DpLgAABAgQIECCwCAGhehGqzkmAAAECBAgQILBRAkL1Rg23zhIgQIAAAQIECCxCQKhehKpzEiBAgAABAgQIbJSAUL1Rw62zBAgQIECAAAECixAQqheh6pwECBAgQIAAAQIbJXDoRvVWZwkQIECAwLoJ7Nu3syveu3dn+9ubAIFdEXCnelcYnYQAAQIECBAgQGCTBYTqTR59fSdAgAABAgQIENgVAaF6VxidhAABAgQIECBAYJMFhOpNHn19J0CAAAECBAgQ2BUBoXpXGJ2EAAECBAgQIEBgkwWE6k0efX0nQIAAAQIECBDYFQGhelcYnYQAAQIECBAgQGCTBYTqTR59fSdAgAABAgQIENgVAaF6VxidhAABAgQIECBAYJMFhOpNHn19J0CAAAECBAgQ2BUBoXpXGJ2EAAECBAgQIEBgkwWE6k0efX0nQIAAAQIECBDYFQGhelcYnYQAAQIECBAgQGCTBYTqTR59fSdAgAABAgQIENgVAaF6VxidhAABAgQIECBAYJMFhOpNHn19J0CAAAECBAgQ2BUBoXpXGJ2EAAECBAgQIEBgkwWE6k0efX0nQIAAAQIECBDYFYF5QvW5SW5K8hejFu+R5NIkV7XlkW3bIUlenOTqJFcmecjomLPb/nVMfR/KQ5O8sx1Tx9Y5FAIECBAgQIAAAQJrIzBPqH5FktNmevTMJG9MckJb1nqV01td1e9N8tJWXyH82UlOSnJi+z4E8drnyaPjZttqp7AgQIAAAQIECBAgsJoC84TqP0xy88zln5nkvFZXy7Pa96o/P8ltSd6c5IgkxyQ5td3RrvPc0r5XeK5th7d965g6djjXTJNWCRAgQIAAAQIECKymwKG387KOTvKhduwNSWq9yrFJrm3fa3Fdq9uuvvYZyrD/sD67rLvf9cn+/ftnt1knQIAAAQIECBAgMInA7Q3V44utO8z1WUbZl6Q+2bNnz7LaXEa/tEGAAAECBAgQILDGAvNM/9iqeze2qRu1raZw1IOMVa5Pcnz7XovjWt129bXPUIb9h3VLAgQIECBAgAABAisvcHtD9UWjN3jUmzwubD2t+ie2N3icnOTWNk3kkiSPSlIPJ9anvlddTSH5WJLat976UccO51p5PBdIgAABAgQIECBAoATmmf7x6iRfneSebY50vcXjBUkuSHJOkg8meWzjvDjJGe31eJ9I8qRWXw8oPjfJZW39OaOHH5+SpN4wcliSN7RP282CAAECBAgQIECAwOoLzBOqH3+AbpyyRX3Nc37qFvVVVe+7rs9suTzJA2YrrRMgQIAAAQIECBBYF4HbO/1jXfrnOgkQIECAAAECBAgsXECoXjixBggQIECAAAECBHoXEKp7H2H9I0CAAAECBAgQWLiAUL1wYg0QIECAAAECBAj0LiBU9z7C+keAAAECBAgQILBwAaF64cQaIECAAAECBAgQ6F1AqO59hPWPAAECBAgQIEBg4QJC9cKJNUCAAAECBAgQINC7gFDd+wjrHwECBAgQIECAwMIFhOqFE2uAAAECBAgQIECgdwGhuvcR1j8CBAgQIECAAIGFCwjVCyfWAAECBAgQIECAQO8CQnXvI6x/BAgQIECAAAECCxcQqhdOrAECBAgQIECAAIHeBQ7tvYP6R4AAAQIdCezb11FndIUAgZ4E3KnuaTT1hQABAgQIECBAYBIBoXoSdo0SIECAAAECBAj0JCBU9zSa+kKAAAECBAgQIDCJgDnVk7BrlAABAgQILEhgp/PO9+5d0IU4LYHNEnCnerPGW28JECBAgAABAgQWICBULwDVKQkQIECAAAECBDZLQKjerPHWWwIECBAgQIAAgQUICNULQHVKAgQIECBAgACBzRIQqjdrvPWWAAECBAgQIEBgAQJC9QJQnZIAAQIECBAgQGCzBITqzRpvvSVAgAABAgQIEFiAgFC9AFSnJECAAAECBAgQ2CwBoXqzxltvCRAgQIAAAQIEFiAgVC8A1SkJECBAgAABAgQ2S0Co3qzx1lsCBAgQIECAAIEFCAjVC0B1SgIECBAgQIAAgc0SEKo3a7z1lgABAgQIECBAYAECQvUCUJ2SAAECBAgQIEBgswSE6s0ab70lQIAAAQIECBBYgIBQvQBUpyRAgAABAgQIENgsAaF6s8ZbbwkQIECAAAECBBYgcOgCzumUBAgQIEBgPoF9++bbz14ECBBYcQF3qld8gFweAQIECBAgQIDA6gu4U736Y+QKCRAgQIDA4gR2+q8Fe/cu7lqcmcAaC7hTvcaD59IJECBAgAABAgRWQ0CoXo1xcBUECBAgQIAAAQJrLGD6x04Hzz+T7VTM/gQIECBAgACB7gXcqe5+iHWQAAECBAgQIEBg0QJC9aKFnZ8AAQIECBAgQKB7AaG6+yHWQQIECBAgQIAAgUULmFO9aGHnJ0CAwCYJ7PS5k02y0VcCBLoWEKq7Hl6dI0CAAAECuyxwe35w8m7rXR4Ep1tFAdM/VnFUXBMBAgQIECBAgMBaCQjVazVcLpYAAQIECBAgQGAVBYTqVRwV10SAAAECBAgQILBWAqs0p/q0JC9KcockL0vygrWSdLEECBDoUeD2zJ/t0UGfPj+Bnf45Mgf78/N29CQCq3KnuoL0S5KcnuR+SR7flpOgaJQAAQIECBAgQIDATgRW5U71iUmuTnJNu/jXJDkzybt30hn7EiBAgMA2Aju9W7jNqWwisFCBnf5ZdWd7ocPh5PMJrEqoPjbJtaNLvi7JSaN1XwkQIEBgVmCnwWP2eOsEehHY6d8FIbyXkV+pfhyyIlfzzUlqTvV3tut5QgvVT5u5vr1J6lPlvkne175bJPdM8mEQXQoY2y6H9bOdMrbGtl+Bfnvm7+3mje2XJNmzLt1+eJJLRhf7rCT1UeYXuHz+Xe25ZgLGds0GbAeXa2x3gLVmuxrbNRuwHVyusd0B1prt+nmN7ao8qHhZkhOS3DvJnZI8LslFazYQLpcAAQIECBAgQGBDBVZlTvVnktRUj7pbXW8COTfJuzZ0THSbAAECBAgQIECAAIGJBIa55hM1r9kFChjbBeJOfGpjO/EALLB5Y7tA3IlPbWwnHoAFNm9sF4jr1AQIECBAgAABAgQIECBAgAABAgQIECBAgMDBBJ6b5MokVyT53SRf1A6o1yW+uP1Sndr+kIOdyPaVE/ipJO9t4/v6JEeMrrDejlO/MKleK3nqqN7X9RB4THtu5B+TPGzmko3tDMgartYrYuvvZv0dfeYaXr9L/meBesbrpiR/8c9VuUeSS5Nc1ZZHjrb5uh4Cxyd5U/slg/UM39PbZRvb9Ri/hV3l4aMzf2+SX27rZyR5Q5IK1ycnectoP1/XQ+BRSYaHiV+YpD5V7pfkHUnu3N6Y8/72gG/bbLEGAl/e3rX/+zOh2tiuweAd5BLrYfv6O/ml7W1W9Xe1xlVZT4GvajelxqH6J0c/LNUPTcP/N69nDzfzqo8Z3Wz8giR/2f6eGtvN/POwZa/rDtdL25b/meTxo73qrkn9IVLWU+CbkryqXXqNc32GUm/NqXe9K+snMBuqje36jeHsFfu9C7Mi679+r5k71eP/ntZ/V2tdWW+BC5M8so3lkJV2PLar8p7q9R6K6a/++e3XvH9rkh9rl7PVr36vOmU9Bb6j/ctDXb2xXc8xnOeqje08Squ9jzFc7fHZjas7OsmH2oluSFLryvoK1A9ND27/ov95ja1QvR5/CP5v+ym5/vlp/DmzXf4PJ6n5QXUnc/ZXu69HDzf3Kg82tiVT41vvch/uVG+u1nr1fJ6xXa8euVoCBGYFbktSH2U9Be6W5DeSPCPJx2a6sOOxHeZrzpzH6ooJfN2c11Oh6+Ikz05yfQvaw6HHtbph3XI1BA42tt+e5NFJThn9H7exXY2xO9hVHGxstzre2G6lsl51xnC9xuv2XO2NbTpl3a2uKQL1IKOyfgJ3bIG6stPr2uUb2/Ubx1294vr17kP5niSvbStfP/Og4luHnSzXRqDeIPDuJHtmrvj+Mw8qXuNBxRmh9VmdnVNtbNdn7A50pXWzqv5O3nv0oGKNq7K+ArNzquvNTMNbXWpZD7cp6yVQL3E4P8nPz1y2sZ0B2bTV+meLmhJSr837zTbftgzqD8xL2lPo75x5w8CmGa1rf+t1XNe21yXWKxOHN7tUf2pKSL1hoB6QOX1dO7jB110Pnl6X5FNJ6s5IPWw6FGM7SKzvst6+VG8TqL+jNZ7K+gq8us2f/vv2d/acJEcleWN7pV5N86rXsCnrJfAV7V9/h1cS139j6++tsV2vcXS1BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQBcC9QuC/riLnugEAQIE1ljArylf48Fz6QQIrKTAB5Lcnt+muFudqV+te5/dOpnzECBAgMB8AkL1fE72IkCAAIHPX6B+26BCgACBLgWE6i6HVacIEFhRgScnqd+UeXOSi5J8UbvO+u2nPzNzzbX9+1pd/Srk+u18f9t+dX39Rsatyh+2ynck+bsk39J+4+o3jHa+Y5IPJ3nwqG74es8kv5Xko+0a/yjJ8N+J45O8Lsn+JB9J8ovtoNr+I0k+mOSm9qt/79621a93rjvn9Vvo/jrJ77X6k5P8aWunrvWrhwuwJECAAAECBAgQIFACB5r+8bUtzD4kyZ2T/EKSIQSfmORvRgG2wu0nkhzdSB/TAngF2ArKH09yTNs2O6d6dvrHDyT59dHQnJnknaP18df/keSXk1Twrs9XJjkkyR2SVPj9uSR3TXKXJPVrfqt8R/tB4UuT3K0F71e2bUOoPr8dd1iSY1sor18JXP15ZFvf046xIECAAAECBAgQIHDAUP3yJD858qkA+vdJKnhWeU8LmPX9aUku/qfqLf/3iiQVjqscLFTX3fC6w3142/+1SSpob1Wek+TCLeZkP7zdod5q+sYbkzxldLL7tn7VvkOorsA9lB9MMoTuoe6SJGcPK5YECBBYR4Hhn/XW8dpdMwECBNZJoMJtTZEYSk3PqGkUdee2ynlJvq19r+U4eD4xSQXpmpZRnwckqbvZ85S6A/4nSf5zkiOSnJ7kVQc48KfaXeffTXJNkpp2UqWmftS1f6atjxez/ar9KlAPd9lr32tHB3xJkrrzPvSllnXXe7jzPtrVVwIECKyPwFZ3Hdbn6l0pAQIE1kegwm0FyqHUNIqjklzfKn6tzX9+YJIvT/J/Wn0d8ytJTknyZ0n+oQXsmpYxb6nA/p0t7NY5hjZnj6872t/fPhXcaw70ZS0Uf3E7fjZYz/ar9qt9bkxyXGugpqQMpQJ2/cBQ88sVAgQIECBAgAABAlsK1Jzquhtc846HT93AqNfs1UN+D2pzql+0xfulL01yZZJzR2e+X5JPJqlpFTW3+UkttFZIrjI7/eOGJI9q24ZFzWW+pYX2uut9oPLoNvWjAnvdnf5Qkq8Zzan+6dGc6ke0k9R1XJXk3m1OdU0vqR8QqgzTP8Y3cOq8dY2ntvOWUT2oOATwdqgFAQIECBAgQIDAJgtUqK47s+PP8xrId7e3eNTbP+otG7NBsqZ91HEVZMfl+e1tHPXWjp9N8gftznPtMxuqq40KwzWt4rGjk7ysPeBYc7kPVOptI3X99SDkdUl+dLRj3YGuu+c1ZaWu48VtW00j/LF2N7t+aKhAfWTbtlWork0ntT6UQx3z20nq/AoBAgQIECBAgACBz1vgq9qr53YytWPeRiv4DneQ5z3GfgQIECBAgAABAgTWSqBeYfcb7a7vbl/4PdqDhhXaFQIECBAgQIAAAQJdCtSDiTXlon4hyvDqu93qaD0QWOeu908rBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQGAv8fzN0wjU74/YDAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add an additional feature which flags the outliers. Let's plot the distribution of `target` using a histogram.\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    199710\n",
       "1      2207\n",
       "Name: outliers, dtype: int64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['outliers'] = 0\n",
    "train_df.loc[train_df['target'] < -30, 'outliers'] = 1\n",
    "train_df['outliers'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have around 2207 outliers in our dataset which is around 1.105% of the dataset. This is a very skewed distribution and the outliers penalise our metric RMSE heavily as they have huge variance with non-outliers. It's important to retain them in our dataset instead of deleting them. Later we will learn how to build models with and without outliers and optimise both models to improve our prediction accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target encode the outliers "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 386), (123623, 384))"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1120"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "for f in ['feature_1','feature_2','feature_3']:\n",
    "    order_label = train_df.groupby([f])['outliers'].mean()\n",
    "    train_df[f] = train_df[f].map(order_label)\n",
    "    test_df[f] = test_df[f].map(order_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_df.to_feather('train_df_alpha')\n",
    "test_df.to_feather('test_df_alpha')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = feather.read_dataframe('train_df_alpha')\n",
    "test_df = feather.read_dataframe('test_df_alpha')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Scrap pad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1.963031e+06\n",
       "mean     1.476515e+00\n",
       "std      4.994483e-01\n",
       "min      1.000000e+00\n",
       "25%      1.000000e+00\n",
       "50%      1.000000e+00\n",
       "75%      2.000000e+00\n",
       "max      2.000000e+00\n",
       "Name: month_lag, dtype: float64"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans['month_lag'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month_diff</th>\n",
       "      <th>month_lag</th>\n",
       "      <th>purchase_date</th>\n",
       "      <th>card_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-03-01 03:24:51</td>\n",
       "      <td>C_ID_7c628841cb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-03-01 11:01:06</td>\n",
       "      <td>C_ID_25d399500c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-03-01 11:27:39</td>\n",
       "      <td>C_ID_e3542c52f1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-03-01 11:55:11</td>\n",
       "      <td>C_ID_fabd47ca44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>2017-03-01 12:37:26</td>\n",
       "      <td>C_ID_6f9a771d17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   month_diff  month_lag       purchase_date          card_id\n",
       "0          24          1 2017-03-01 03:24:51  C_ID_7c628841cb\n",
       "1          24          1 2017-03-01 11:01:06  C_ID_25d399500c\n",
       "2          24          1 2017-03-01 11:27:39  C_ID_e3542c52f1\n",
       "3          24          1 2017-03-01 11:55:11  C_ID_fabd47ca44\n",
       "4          24          1 2017-03-01 12:37:26  C_ID_6f9a771d17"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans[['month_diff', 'month_lag', 'purchase_date', 'card_id']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.911236e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.269206e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.007556e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.100000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.200000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.200000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.200000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2.400000e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         month_diff\n",
       "count  2.911236e+07\n",
       "mean   1.269206e+01\n",
       "std    2.007556e+00\n",
       "min    1.100000e+01\n",
       "25%    1.200000e+01\n",
       "50%    1.200000e+01\n",
       "75%    1.200000e+01\n",
       "max    2.400000e+01"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans[['month_diff']].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>card_id</th>\n",
       "      <th>observation_date_old</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>59734</th>\n",
       "      <td>C_ID_7c628841cb</td>\n",
       "      <td>2017-03-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               card_id observation_date_old\n",
       "59734  C_ID_7c628841cb           2017-03-01"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['card_id'] == 'C_ID_7c628841cb'][['card_id', 'observation_date_old']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['observation_date_old'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['observation_date_old'].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['outliers'] == 1]['observation_date_old'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['outliers'] == 1]['first_active_month'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['first_active_month'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df['first_active_month'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_rows', 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.32852396"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_sorted[new_hist_sorted['card_id'] == 'C_ID_00007093c1']['purchase_amount'].head(2).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def trendline(data, order=1):\n",
    "    coeffs = np.polyfit(data.index.values, list(data), order)\n",
    "    slope = coeffs[-2]\n",
    "    return float(slope)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "314"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans['merchant_category_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "226129"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans['merchant_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "209"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans[new_hist_trans['category_1'] == 1]['merchant_category_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "311"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans[new_hist_trans['category_1'] == 0]['merchant_category_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "314"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_hist_trans['merchant_category_id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_rows', 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "outliers_df = train_df[train_df['outliers'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "non_outliers_df = train_df[train_df['outliers'] == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "outliers_df_summary = DataFrameSummary(outliers_df).summary().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "non_outliers_df_summary = DataFrameSummary(non_outliers_df).summary().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count            199710\n",
       "mean            33.7506\n",
       "std             30.7368\n",
       "min                   1\n",
       "25%                  13\n",
       "50%                  25\n",
       "75%                  44\n",
       "max                 398\n",
       "counts           199710\n",
       "uniques             313\n",
       "missing               0\n",
       "missing_perc         0%\n",
       "types           numeric\n",
       "Name: merchant_id_nunique_old, dtype: object"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_outliers_df_summary.T['merchant_id_nunique_old']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Transform purchase_value to actual amount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans['purchase_amount_new'] = np.round(new_hist_trans['purchase_amount'] / 0.00150265118 + 497.06,2)\n",
    "hist_trans['purchase_amount_new'] = np.round(hist_trans['purchase_amount'] / 0.00150265118 + 497.06,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans_purchase_aggs = new_hist_trans.groupby('card_id').agg({\n",
    "    'purchase_amount_new': [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                            ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                            ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                            ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                            ('tail_sum', tail_sum), ('tail_max', tail_max), ('gmean', scipy.stats.gmean ),\n",
    "                            ('hmean', scipy.stats.hmean)]\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hist_trans_purchase_aggs.columns = [ '_'.join(c).strip() for c in new_hist_trans_purchase_aggs.columns.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans.loc[hist_trans['purchase_amount_new'] == 0, 'purchase_amount_new'] = 0.0001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_purchase_aggs = hist_trans.groupby('card_id').agg({\n",
    "        'purchase_amount_new': [('sum', 'sum'), ('pct_75', lambda x: np.percentile(x, q = 75)), \n",
    "                                ('pct_25', lambda x: np.percentile(x, q = 25)), ('mean', 'mean'), \n",
    "                                ('median', 'median'), ('max', 'max'), ('min', 'min'), ('var', 'var'), \n",
    "                                ('skew', 'skew'), ('head_sum', head_sum), ('head_max', head_max), \n",
    "                                ('tail_sum', tail_sum), ('tail_max', tail_max), ('gmean', scipy.stats.gmean ),\n",
    "                                ('hmean', scipy.stats.hmean)]\n",
    "    })\n",
    "hist_trans_purchase_aggs.columns = [ '_'.join(c).strip() for c in hist_trans_purchase_aggs.columns.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_trans_purchase_aggs.reset_index().to_feather('hist_trans_purchase_aggs_alpha')\n",
    "new_hist_trans_purchase_aggs.reset_index().to_feather('new_hist_trans_purchase_aggs_alpha')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 357), (123623, 355), (325540, 15), (290001, 15))"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape, hist_trans_purchase_aggs.shape, new_hist_trans_purchase_aggs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = join_dfs(train_df, hist_trans_purchase_aggs, left_on='card_id', suffix='_old')\n",
    "train_df = join_dfs(train_df, new_hist_trans_purchase_aggs, left_on='card_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = join_dfs(test_df, hist_trans_purchase_aggs, left_on='card_id', suffix='_old')\n",
    "test_df = join_dfs(test_df, new_hist_trans_purchase_aggs, left_on='card_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['purchase_amount_new_sum', 'purchase_amount_new_pct_75', 'purchase_amount_new_pct_25',\n",
       "       'purchase_amount_new_mean', 'purchase_amount_new_median', 'purchase_amount_new_max',\n",
       "       'purchase_amount_new_min', 'purchase_amount_new_var', 'purchase_amount_new_skew',\n",
       "       'purchase_amount_new_head_sum', 'purchase_amount_new_head_max', 'purchase_amount_new_tail_sum',\n",
       "       'purchase_amount_new_tail_max', 'purchase_amount_new_gmean', 'purchase_amount_new_hmean'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_trans_purchase_aggs.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "for df in [train_df, test_df]:\n",
    "    df['purchase_amount_new_sum_total'] = df['purchase_amount_new_sum'] + df['purchase_amount_new_sum_old']\n",
    "    df['purchase_amount_new_sum_ratio'] = df['purchase_amount_new_sum'] / df['purchase_amount_new_sum_old']\n",
    "    df['purchase_amount_new_pct_75_ratio'] = df['purchase_amount_new_pct_75'] / df['purchase_amount_new_pct_75_old']\n",
    "    df['purchase_amount_new_pct_25_ratio'] = df['purchase_amount_new_pct_25'] / df['purchase_amount_new_pct_25_old']\n",
    "    df['purchase_amount_new_inter_quartile'] = df['purchase_amount_new_pct_75'] - df['purchase_amount_new_pct_25']\n",
    "    df['purchase_amount_new_inter_quartile_old'] = df['purchase_amount_new_pct_75_old'] - df['purchase_amount_new_pct_25_old']\n",
    "    df['purchase_amount_new_inter_quartile_ratio'] = df['purchase_amount_new_inter_quartile'] / df['purchase_amount_new_inter_quartile_old']\n",
    "    df['purchase_amount_new_mean_total'] = df['purchase_amount_new_mean'] + df['purchase_amount_new_mean_old']\n",
    "    df['purchase_amount_new_mean_ratio'] = df['purchase_amount_new_mean'] / df['purchase_amount_new_mean_old']\n",
    "    df['purchase_amount_new_median_ratio'] = df['purchase_amount_new_median'] / df['purchase_amount_new_median_old']\n",
    "    df['purchase_amount_new_max_total'] = df['purchase_amount_new_max'] + df['purchase_amount_new_max_old']\n",
    "    df['purchase_amount_new_min_total'] = df['purchase_amount_new_min'] + df['purchase_amount_new_min_old']\n",
    "    df['purchase_amount_new_skew_ratio'] = df['purchase_amount_new_skew'] / df['purchase_amount_new_skew_old']\n",
    "    df['purchase_amount_new_before_after_ratio'] = df['purchase_amount_new_head_sum'] / df['purchase_amount_new_tail_sum_old']\n",
    "    df['purchase_amount_new_first_last_diff'] = df['purchase_amount_new_tail_sum'] - df['purchase_amount_new_head_max_old']\n",
    "    df['purchase_amount_new_fi_last_old_total'] = df['purchase_amount_new_tail_sum'] -  df['purchase_amount_new_head_sum']\n",
    "    df['purchase_amount_new_fi_last_new_total'] = df['purchase_amount_new_tail_sum_old'] -  df['purchase_amount_new_head_max_old']\n",
    "    df['purchase_amount_new_gmean_total'] = df['purchase_amount_new_gmean'] + df['purchase_amount_new_gmean_old']\n",
    "    df['purchase_amount_new_hmean_total'] = df['purchase_amount_new_hmean'] + df['purchase_amount_new_hmean_old']\n",
    "    df['purchase_amount_new_gmean_ratio'] = df['purchase_amount_new_gmean'] / df['purchase_amount_new_gmean_old']\n",
    "    df['purchase_amount_new_hmean_ratio'] = df['purchase_amount_new_hmean'] / df['purchase_amount_new_hmean_old']\n",
    "    df['purchase_amount_sum_total'] = df['purchase_amount_sum'] + df['purchase_amount_sum_old']\n",
    "    df['purchase_amount_sum_ratio'] = df['purchase_amount_sum'] / df['purchase_amount_sum_old']\n",
    "    df['purchase_amount_pct_75_ratio'] = df['purchase_amount_pct_75'] / df['purchase_amount_pct_75_old']\n",
    "    df['purchase_amount_pct_25_ratio'] = df['purchase_amount_pct_25'] / df['purchase_amount_pct_25_old']\n",
    "    df['purchase_amount_inter_quartile'] = df['purchase_amount_pct_75'] - df['purchase_amount_pct_25']\n",
    "    df['purchase_amount_inter_quartile_old'] = df['purchase_amount_pct_75_old'] - df['purchase_amount_pct_25_old']\n",
    "    df['purchase_amount_inter_quartile_ratio'] = df['purchase_amount_inter_quartile'] / df['purchase_amount_inter_quartile_old']\n",
    "    df['purchase_amount_mean_total'] = df['purchase_amount_mean'] + df['purchase_amount_mean_old']\n",
    "    df['purchase_amount_mean_ratio'] = df['purchase_amount_mean'] / df['purchase_amount_mean_old']\n",
    "    df['purchase_amount_median_ratio'] = df['purchase_amount_median'] / df['purchase_amount_median_old']\n",
    "    df['purchase_amount_max_total'] = df['purchase_amount_max'] + df['purchase_amount_max_old']\n",
    "    df['purchase_amount_min_total'] = df['purchase_amount_min'] + df['purchase_amount_min_old']\n",
    "    df['purchase_amount_skew_ratio'] = df['purchase_amount_skew'] / df['purchase_amount_skew_old']\n",
    "    df['purchase_amount_before_after_ratio'] = df['purchase_amount_head_sum'] / df['purchase_amount_tail_sum_old']\n",
    "    df['purchase_amount_first_last_diff'] = df['purchase_amount_tail_sum'] - df['purchase_amount_head_max_old']\n",
    "    df['purchase_amount_fi_last_old_total'] = df['purchase_amount_tail_sum'] -  df['purchase_amount_head_sum']\n",
    "    df['purchase_amount_fi_last_new_total'] = df['purchase_amount_tail_sum_old'] -  df['purchase_amount_head_max_old']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((201917, 424), (123623, 422))"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.to_feather('train_df_alpha')\n",
    "test_df.to_feather('test_df_alpha')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "624px",
    "left": "114px",
    "top": "111.133px",
    "width": "288px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}