examples/00_quick_start/lightgbm_tinycriteo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LightGBM: A Highly Efficient Gradient Boosting Decision Tree\n",
    "This notebook will give you an example of how to train a LightGBM model to estimate click-through rates on an e-commerce advertisement. We will train a LightGBM based model on the Criteo dataset.\n",
    "\n",
    "[LightGBM](https://github.com/Microsoft/LightGBM) is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient with the following advantages:\n",
    "* Fast training speed and high efficiency.\n",
    "* Low memory usage.\n",
    "* Great accuracy.\n",
    "* Support of parallel and GPU learning.\n",
    "* Capable of handling large-scale data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Settings and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) \n",
      "[GCC 7.3.0]\n",
      "LightGBM version: 2.2.1\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "import lightgbm as lgb\n",
    "import papermill as pm\n",
    "import scrapbook as sb\n",
    "import pandas as pd\n",
    "import category_encoders as ce\n",
    "from tempfile import TemporaryDirectory\n",
    "from sklearn.metrics import roc_auc_score, log_loss\n",
    "\n",
    "import recommenders.models.lightgbm.lightgbm_utils as lgb_utils\n",
    "import recommenders.datasets.criteo as criteo\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"LightGBM version: {}\".format(lgb.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter Setting\n",
    "Let's set the main related parameters for LightGBM now. Basically, the task is a binary classification (predicting click or no click), so the objective function is set to binary logloss, and 'AUC' metric, is used as a metric which is less effected by imbalance in the classes of the dataset.\n",
    "\n",
    "Generally, we can adjust the number of leaves (MAX_LEAF), the minimum number of data in each leaf (MIN_DATA), maximum number of trees (NUM_OF_TREES), the learning rate of trees (TREE_LEARNING_RATE) and EARLY_STOPPING_ROUNDS (to avoid overfitting) in the model to get better performance.\n",
    "\n",
    "Besides, we can also adjust some other listed parameters to optimize the results. [In this link](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst), a list of all the parameters is shown. Also, some advice on how to tune these parameters can be found [in this url](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters-Tuning.rst). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MAX_LEAF = 64\n",
    "MIN_DATA = 20\n",
    "NUM_OF_TREES = 100\n",
    "TREE_LEARNING_RATE = 0.15\n",
    "EARLY_STOPPING_ROUNDS = 20\n",
    "METRIC = \"auc\"\n",
    "SIZE = \"sample\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'task': 'train',\n",
    "    'boosting_type': 'gbdt',\n",
    "    'num_class': 1,\n",
    "    'objective': \"binary\",\n",
    "    'metric': METRIC,\n",
    "    'num_leaves': MAX_LEAF,\n",
    "    'min_data': MIN_DATA,\n",
    "    'boost_from_average': True,\n",
    "    #set it according to your cpu cores.\n",
    "    'num_threads': 20,\n",
    "    'feature_fraction': 0.8,\n",
    "    'learning_rate': TREE_LEARNING_RATE,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "Here we use CSV format as the example data input. Our example data is a sample (about 100 thousand samples) from [Criteo dataset](https://www.kaggle.com/c/criteo-display-ad-challenge). The Criteo dataset is a well-known industry benchmarking dataset for developing CTR prediction models, and it's frequently adopted as evaluation dataset by research papers. The original dataset is too large for a lightweight demo, so we sample a small portion from it as a demo dataset.\n",
    "\n",
    "Specifically, there are 39 columns of features in Criteo, where 13 columns are numerical features (I1-I13) and the other 26 columns are categorical features (C1-C26)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "8.79MB [00:00, 12.1MB/s]                            \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>I1</th>\n",
       "      <th>I2</th>\n",
       "      <th>I3</th>\n",
       "      <th>I4</th>\n",
       "      <th>I5</th>\n",
       "      <th>I6</th>\n",
       "      <th>I7</th>\n",
       "      <th>I8</th>\n",
       "      <th>I9</th>\n",
       "      <th>...</th>\n",
       "      <th>C17</th>\n",
       "      <th>C18</th>\n",
       "      <th>C19</th>\n",
       "      <th>C20</th>\n",
       "      <th>C21</th>\n",
       "      <th>C22</th>\n",
       "      <th>C23</th>\n",
       "      <th>C24</th>\n",
       "      <th>C25</th>\n",
       "      <th>C26</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1382.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>181.0</td>\n",
       "      <td>...</td>\n",
       "      <td>e5ba7672</td>\n",
       "      <td>f54016b9</td>\n",
       "      <td>21ddcdc9</td>\n",
       "      <td>b1252a9d</td>\n",
       "      <td>07b5194c</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3a171ecb</td>\n",
       "      <td>c5c50484</td>\n",
       "      <td>e8b83407</td>\n",
       "      <td>9727dd16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>07c540c4</td>\n",
       "      <td>b04e4670</td>\n",
       "      <td>21ddcdc9</td>\n",
       "      <td>5840adea</td>\n",
       "      <td>60f6221e</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3a171ecb</td>\n",
       "      <td>43f13e8b</td>\n",
       "      <td>e8b83407</td>\n",
       "      <td>731c3655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>767.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>245.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8efede7f</td>\n",
       "      <td>3412118d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>e587c466</td>\n",
       "      <td>ad3062eb</td>\n",
       "      <td>3a171ecb</td>\n",
       "      <td>3b183c5c</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>893</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4392.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1e88c74f</td>\n",
       "      <td>74ef3502</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6b3a5ca6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3a171ecb</td>\n",
       "      <td>9117a34a</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>-1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1e88c74f</td>\n",
       "      <td>26b3c7a7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21c9516a</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32c7478e</td>\n",
       "      <td>b34f3128</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Label   I1   I2    I3    I4      I5    I6    I7   I8     I9  ...       C17  \\\n",
       "0      0  1.0    1   5.0   0.0  1382.0   4.0  15.0  2.0  181.0  ...  e5ba7672   \n",
       "1      0  2.0    0  44.0   1.0   102.0   8.0   2.0  2.0    4.0  ...  07c540c4   \n",
       "2      0  2.0    0   1.0  14.0   767.0  89.0   4.0  2.0  245.0  ...  8efede7f   \n",
       "3      0  NaN  893   NaN   NaN  4392.0   NaN   0.0  0.0    0.0  ...  1e88c74f   \n",
       "4      0  3.0   -1   NaN   0.0     2.0   0.0   3.0  0.0    0.0  ...  1e88c74f   \n",
       "\n",
       "        C18       C19       C20       C21       C22       C23       C24  \\\n",
       "0  f54016b9  21ddcdc9  b1252a9d  07b5194c       NaN  3a171ecb  c5c50484   \n",
       "1  b04e4670  21ddcdc9  5840adea  60f6221e       NaN  3a171ecb  43f13e8b   \n",
       "2  3412118d       NaN       NaN  e587c466  ad3062eb  3a171ecb  3b183c5c   \n",
       "3  74ef3502       NaN       NaN  6b3a5ca6       NaN  3a171ecb  9117a34a   \n",
       "4  26b3c7a7       NaN       NaN  21c9516a       NaN  32c7478e  b34f3128   \n",
       "\n",
       "        C25       C26  \n",
       "0  e8b83407  9727dd16  \n",
       "1  e8b83407  731c3655  \n",
       "2       NaN       NaN  \n",
       "3       NaN       NaN  \n",
       "4       NaN       NaN  \n",
       "\n",
       "[5 rows x 40 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nume_cols = [\"I\" + str(i) for i in range(1, 14)]\n",
    "cate_cols = [\"C\" + str(i) for i in range(1, 27)]\n",
    "label_col = \"Label\"\n",
    "\n",
    "header = [label_col] + nume_cols + cate_cols\n",
    "with TemporaryDirectory() as tmp:\n",
    "    all_data = criteo.load_pandas_df(size=SIZE, local_cache_path=tmp, header=header)\n",
    "display(all_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we cut three sets (train_data (first 80%), valid_data (middle 10%) and test_data (last 10%)), cut from the original all data. <br>\n",
    "Notably, considering the Criteo is a kind of time-series streaming data, which is also very common in recommendation scenario, we split the data by its order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split data to 3 sets    \n",
    "length = len(all_data)\n",
    "train_data = all_data.loc[:0.8*length-1]\n",
    "valid_data = all_data.loc[0.8*length:0.9*length-1]\n",
    "test_data = all_data.loc[0.9*length:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage\n",
    "### Ordinal Encoding\n",
    "Considering LightGBM could handle the low-frequency features and missing value by itself, for basic usage, we only encode the string-like categorical features by an ordinal encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Data Shape: X: (80000, 39); Y: (80000,).\n",
      "Valid Data Shape: X: (10000, 39); Y: (10000,).\n",
      "Test Data Shape: X: (10000, 39); Y: (10000,).\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>I1</th>\n",
       "      <th>I2</th>\n",
       "      <th>I3</th>\n",
       "      <th>I4</th>\n",
       "      <th>I5</th>\n",
       "      <th>I6</th>\n",
       "      <th>I7</th>\n",
       "      <th>I8</th>\n",
       "      <th>I9</th>\n",
       "      <th>I10</th>\n",
       "      <th>...</th>\n",
       "      <th>C17</th>\n",
       "      <th>C18</th>\n",
       "      <th>C19</th>\n",
       "      <th>C20</th>\n",
       "      <th>C21</th>\n",
       "      <th>C22</th>\n",
       "      <th>C23</th>\n",
       "      <th>C24</th>\n",
       "      <th>C25</th>\n",
       "      <th>C26</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1382.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>181.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>767.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>245.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>893</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4392.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>-1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 39 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    I1   I2    I3    I4      I5    I6    I7   I8     I9  I10  ...  C17  C18  \\\n",
       "0  1.0    1   5.0   0.0  1382.0   4.0  15.0  2.0  181.0  1.0  ...    1    1   \n",
       "1  2.0    0  44.0   1.0   102.0   8.0   2.0  2.0    4.0  1.0  ...    2    2   \n",
       "2  2.0    0   1.0  14.0   767.0  89.0   4.0  2.0  245.0  1.0  ...    3    3   \n",
       "3  NaN  893   NaN   NaN  4392.0   NaN   0.0  0.0    0.0  NaN  ...    4    4   \n",
       "4  3.0   -1   NaN   0.0     2.0   0.0   3.0  0.0    0.0  1.0  ...    4    5   \n",
       "\n",
       "   C19  C20  C21  C22  C23  C24  C25  C26  \n",
       "0    1    1    1    1    1    1    1    1  \n",
       "1    1    2    2    1    1    2    1    2  \n",
       "2    2    3    3    2    1    3    2    3  \n",
       "3    2    3    4    1    1    4    2    3  \n",
       "4    2    3    5    1    2    5    2    3  \n",
       "\n",
       "[5 rows x 39 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ord_encoder = ce.ordinal.OrdinalEncoder(cols=cate_cols)\n",
    "\n",
    "def encode_csv(df, encoder, label_col, typ='fit'):\n",
    "    if typ == 'fit':\n",
    "        df = encoder.fit_transform(df)\n",
    "    else:\n",
    "        df = encoder.transform(df)\n",
    "    y = df[label_col].values\n",
    "    del df[label_col]\n",
    "    return df, y\n",
    "\n",
    "train_x, train_y = encode_csv(train_data, ord_encoder, label_col)\n",
    "valid_x, valid_y = encode_csv(valid_data, ord_encoder, label_col, 'transform')\n",
    "test_x, test_y = encode_csv(test_data, ord_encoder, label_col, 'transform')\n",
    "\n",
    "print('Train Data Shape: X: {trn_x_shape}; Y: {trn_y_shape}.\\nValid Data Shape: X: {vld_x_shape}; Y: {vld_y_shape}.\\nTest Data Shape: X: {tst_x_shape}; Y: {tst_y_shape}.\\n'\n",
    "      .format(trn_x_shape=train_x.shape,\n",
    "              trn_y_shape=train_y.shape,\n",
    "              vld_x_shape=valid_x.shape,\n",
    "              vld_y_shape=valid_y.shape,\n",
    "              tst_x_shape=test_x.shape,\n",
    "              tst_y_shape=test_y.shape,))\n",
    "train_x.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create model\n",
    "When both hyper-parameters and data are ready, we can create a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.728695\n",
      "Training until validation scores don't improve for 20 rounds.\n",
      "[2]\tvalid_0's auc: 0.742373\n",
      "[3]\tvalid_0's auc: 0.747298\n",
      "[4]\tvalid_0's auc: 0.747969\n",
      "[5]\tvalid_0's auc: 0.751102\n",
      "[6]\tvalid_0's auc: 0.753734\n",
      "[7]\tvalid_0's auc: 0.755335\n",
      "[8]\tvalid_0's auc: 0.75658\n",
      "[9]\tvalid_0's auc: 0.757071\n",
      "[10]\tvalid_0's auc: 0.758572\n",
      "[11]\tvalid_0's auc: 0.759742\n",
      "[12]\tvalid_0's auc: 0.760415\n",
      "[13]\tvalid_0's auc: 0.760602\n",
      "[14]\tvalid_0's auc: 0.761192\n",
      "[15]\tvalid_0's auc: 0.7616\n",
      "[16]\tvalid_0's auc: 0.761697\n",
      "[17]\tvalid_0's auc: 0.762255\n",
      "[18]\tvalid_0's auc: 0.76253\n",
      "[19]\tvalid_0's auc: 0.763092\n",
      "[20]\tvalid_0's auc: 0.762172\n",
      "[21]\tvalid_0's auc: 0.762066\n",
      "[22]\tvalid_0's auc: 0.761866\n",
      "[23]\tvalid_0's auc: 0.761433\n",
      "[24]\tvalid_0's auc: 0.761588\n",
      "[25]\tvalid_0's auc: 0.761017\n",
      "[26]\tvalid_0's auc: 0.761086\n",
      "[27]\tvalid_0's auc: 0.761177\n",
      "[28]\tvalid_0's auc: 0.760893\n",
      "[29]\tvalid_0's auc: 0.760635\n",
      "[30]\tvalid_0's auc: 0.760104\n",
      "[31]\tvalid_0's auc: 0.759298\n",
      "[32]\tvalid_0's auc: 0.759176\n",
      "[33]\tvalid_0's auc: 0.758384\n",
      "[34]\tvalid_0's auc: 0.758168\n",
      "[35]\tvalid_0's auc: 0.757902\n",
      "[36]\tvalid_0's auc: 0.758005\n",
      "[37]\tvalid_0's auc: 0.757782\n",
      "[38]\tvalid_0's auc: 0.757542\n",
      "[39]\tvalid_0's auc: 0.756966\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's auc: 0.763092\n"
     ]
    }
   ],
   "source": [
    "lgb_train = lgb.Dataset(train_x, train_y.reshape(-1), params=params, categorical_feature=cate_cols)\n",
    "lgb_valid = lgb.Dataset(valid_x, valid_y.reshape(-1), reference=lgb_train, categorical_feature=cate_cols)\n",
    "lgb_test = lgb.Dataset(test_x, test_y.reshape(-1), reference=lgb_train, categorical_feature=cate_cols)\n",
    "lgb_model = lgb.train(params,\n",
    "                      lgb_train,\n",
    "                      num_boost_round=NUM_OF_TREES,\n",
    "                      early_stopping_rounds=EARLY_STOPPING_ROUNDS,\n",
    "                      valid_sets=lgb_valid,\n",
    "                      categorical_feature=cate_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see what is the model's performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.7674356153037237, 'logloss': 0.466876775528735}\n"
     ]
    },
    {
     "data": {
      "application/papermill.record+json": {
       "res_basic": {
        "auc": 0.7674356153037237,
        "logloss": 0.466876775528735
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_preds = lgb_model.predict(test_x)\n",
    "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n",
    "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds), eps=1e-12)\n",
    "res_basic = {\"auc\": auc, \"logloss\": logloss}\n",
    "print(res_basic)\n",
    "sb.glue(\"res_basic\", res_basic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<script type=\"text/javascript\" src=\"http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default\"></script>\n",
    "## Optimized Usage\n",
    "### Label-encoding and Binary-encoding\n",
    "Next, since LightGBM has a better capability in handling dense numerical features effectively, we try to convert all the categorical features in original data into numerical ones, by label-encoding [3] and binary-encoding [4]. Also due to the sequence property of Criteo, the label-encoding we adopted is executed one-by-one, which means we encode the samples in order, by the information of the previous samples before each sample (sequential label-encoding and sequential count-encoding). Besides, we also filter the low-frequency categorical features and fill the missing values by the mean of corresponding columns for the numerical features. (consulting `lgb_utils.NumEncoder`)\n",
    "\n",
    "Specifically, in `lgb_utils.NumEncoder`, the main steps are as follows.\n",
    "* Firstly, we convert the low-frequency categorical features to `\"LESS\"` and the missing categorical features to `\"UNK\"`. \n",
    "* Secondly, we convert the missing numerical features into the mean of corresponding columns. \n",
    "* Thirdly, the string-like categorical features are ordinal encoded like the example shown in basic usage. \n",
    "* And then, we target encode the categorical features in the samples order one-by-one. For each sample, we add the label and count information of its former samples into the data and produce new features. Formally, for $i=1,2,...,n$, we add $\\frac{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c) \\cdot y}{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c)}$ as a new label feature for current sample $x_i$, where $c$ is a category to encode in current sample, so $(i-1)$ is the number of former samples, and $I(\\cdot)$ is the indicator function that check the former samples contain $c$ (whether $x_j=c$) or not. At the meantime, we also add the count frequency of $c$, which is $\\frac{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c)}{i-1}$, as a new count feature. \n",
    "* Finally, based on the results of ordinal encoding, we add the binary encoding results as new columns into the data.\n",
    "\n",
    "Note that the statistics used in the above process only updates when fitting the training set, while maintaining static when transforming the testing set because the label of test data should be considered as unknown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-04-29 11:26:21,158 [INFO] Filtering and fillna features\n",
      "100%|██████████| 26/26 [00:02<00:00, 12.36it/s]\n",
      "100%|██████████| 13/13 [00:00<00:00, 711.59it/s]\n",
      "2019-04-29 11:26:23,286 [INFO] Ordinal encoding cate features\n",
      "2019-04-29 11:26:24,680 [INFO] Target encoding cate features\n",
      "100%|██████████| 26/26 [00:03<00:00,  6.72it/s]\n",
      "2019-04-29 11:26:28,554 [INFO] Start manual binary encoding\n",
      "100%|██████████| 65/65 [00:04<00:00, 15.87it/s]\n",
      "100%|██████████| 26/26 [00:02<00:00,  8.17it/s]\n",
      "2019-04-29 11:26:35,518 [INFO] Filtering and fillna features\n",
      "100%|██████████| 26/26 [00:00<00:00, 171.81it/s]\n",
      "100%|██████████| 13/13 [00:00<00:00, 2174.25it/s]\n",
      "2019-04-29 11:26:35,690 [INFO] Ordinal encoding cate features\n",
      "2019-04-29 11:26:35,854 [INFO] Target encoding cate features\n",
      "100%|██████████| 26/26 [00:00<00:00, 53.42it/s]\n",
      "2019-04-29 11:26:36,344 [INFO] Start manual binary encoding\n",
      "100%|██████████| 65/65 [00:03<00:00, 20.02it/s]\n",
      "100%|██████████| 26/26 [00:01<00:00, 17.67it/s]\n",
      "2019-04-29 11:26:41,081 [INFO] Filtering and fillna features\n",
      "100%|██████████| 26/26 [00:00<00:00, 158.08it/s]\n",
      "100%|██████████| 13/13 [00:00<00:00, 2203.78it/s]\n",
      "2019-04-29 11:26:41,267 [INFO] Ordinal encoding cate features\n",
      "2019-04-29 11:26:41,429 [INFO] Target encoding cate features\n",
      "100%|██████████| 26/26 [00:00<00:00, 53.08it/s]\n",
      "2019-04-29 11:26:41,922 [INFO] Start manual binary encoding\n",
      "100%|██████████| 65/65 [00:03<00:00, 20.10it/s]\n",
      "100%|██████████| 26/26 [00:01<00:00, 18.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Data Shape: X: (80000, 268); Y: (80000, 1).\n",
      "Valid Data Shape: X: (10000, 268); Y: (10000, 1).\n",
      "Test Data Shape: X: (10000, 268); Y: (10000, 1).\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "label_col = 'Label'\n",
    "num_encoder = lgb_utils.NumEncoder(cate_cols, nume_cols, label_col)\n",
    "train_x, train_y = num_encoder.fit_transform(train_data)\n",
    "valid_x, valid_y = num_encoder.transform(valid_data)\n",
    "test_x, test_y = num_encoder.transform(test_data)\n",
    "del num_encoder\n",
    "print('Train Data Shape: X: {trn_x_shape}; Y: {trn_y_shape}.\\nValid Data Shape: X: {vld_x_shape}; Y: {vld_y_shape}.\\nTest Data Shape: X: {tst_x_shape}; Y: {tst_y_shape}.\\n'\n",
    "      .format(trn_x_shape=train_x.shape,\n",
    "              trn_y_shape=train_y.shape,\n",
    "              vld_x_shape=valid_x.shape,\n",
    "              vld_y_shape=valid_y.shape,\n",
    "              tst_x_shape=test_x.shape,\n",
    "              tst_y_shape=test_y.shape,))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.731759\n",
      "Training until validation scores don't improve for 20 rounds.\n",
      "[2]\tvalid_0's auc: 0.747705\n",
      "[3]\tvalid_0's auc: 0.751667\n",
      "[4]\tvalid_0's auc: 0.75589\n",
      "[5]\tvalid_0's auc: 0.758054\n",
      "[6]\tvalid_0's auc: 0.758094\n",
      "[7]\tvalid_0's auc: 0.759904\n",
      "[8]\tvalid_0's auc: 0.761098\n",
      "[9]\tvalid_0's auc: 0.761744\n",
      "[10]\tvalid_0's auc: 0.762308\n",
      "[11]\tvalid_0's auc: 0.762473\n",
      "[12]\tvalid_0's auc: 0.763606\n",
      "[13]\tvalid_0's auc: 0.764222\n",
      "[14]\tvalid_0's auc: 0.765004\n",
      "[15]\tvalid_0's auc: 0.765933\n",
      "[16]\tvalid_0's auc: 0.766507\n",
      "[17]\tvalid_0's auc: 0.767192\n",
      "[18]\tvalid_0's auc: 0.767284\n",
      "[19]\tvalid_0's auc: 0.767859\n",
      "[20]\tvalid_0's auc: 0.768619\n",
      "[21]\tvalid_0's auc: 0.769045\n",
      "[22]\tvalid_0's auc: 0.768987\n",
      "[23]\tvalid_0's auc: 0.769601\n",
      "[24]\tvalid_0's auc: 0.77011\n",
      "[25]\tvalid_0's auc: 0.770183\n",
      "[26]\tvalid_0's auc: 0.770539\n",
      "[27]\tvalid_0's auc: 0.77096\n",
      "[28]\tvalid_0's auc: 0.771164\n",
      "[29]\tvalid_0's auc: 0.771296\n",
      "[30]\tvalid_0's auc: 0.771402\n",
      "[31]\tvalid_0's auc: 0.771596\n",
      "[32]\tvalid_0's auc: 0.771476\n",
      "[33]\tvalid_0's auc: 0.771697\n",
      "[34]\tvalid_0's auc: 0.77169\n",
      "[35]\tvalid_0's auc: 0.771836\n",
      "[36]\tvalid_0's auc: 0.771832\n",
      "[37]\tvalid_0's auc: 0.771948\n",
      "[38]\tvalid_0's auc: 0.772098\n",
      "[39]\tvalid_0's auc: 0.772136\n",
      "[40]\tvalid_0's auc: 0.771748\n",
      "[41]\tvalid_0's auc: 0.771748\n",
      "[42]\tvalid_0's auc: 0.771724\n",
      "[43]\tvalid_0's auc: 0.771676\n",
      "[44]\tvalid_0's auc: 0.77169\n",
      "[45]\tvalid_0's auc: 0.771916\n",
      "[46]\tvalid_0's auc: 0.771864\n",
      "[47]\tvalid_0's auc: 0.771851\n",
      "[48]\tvalid_0's auc: 0.771891\n",
      "[49]\tvalid_0's auc: 0.771629\n",
      "[50]\tvalid_0's auc: 0.771984\n",
      "[51]\tvalid_0's auc: 0.772085\n",
      "[52]\tvalid_0's auc: 0.771988\n",
      "[53]\tvalid_0's auc: 0.771778\n",
      "[54]\tvalid_0's auc: 0.771485\n",
      "[55]\tvalid_0's auc: 0.7715\n",
      "[56]\tvalid_0's auc: 0.770778\n",
      "[57]\tvalid_0's auc: 0.770816\n",
      "[58]\tvalid_0's auc: 0.770408\n",
      "[59]\tvalid_0's auc: 0.770489\n",
      "Early stopping, best iteration is:\n",
      "[39]\tvalid_0's auc: 0.772136\n"
     ]
    }
   ],
   "source": [
    "lgb_train = lgb.Dataset(train_x, train_y.reshape(-1), params=params)\n",
    "lgb_valid = lgb.Dataset(valid_x, valid_y.reshape(-1), reference=lgb_train)\n",
    "lgb_model = lgb.train(params,\n",
    "                      lgb_train,\n",
    "                      num_boost_round=NUM_OF_TREES,\n",
    "                      early_stopping_rounds=EARLY_STOPPING_ROUNDS,\n",
    "                      valid_sets=lgb_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.7757371640011422, 'logloss': 0.4606505068849181}\n"
     ]
    },
    {
     "data": {
      "application/papermill.record+json": {
       "res_optim": {
        "auc": 0.7757371640011422,
        "logloss": 0.4606505068849181
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_preds = lgb_model.predict(test_x)\n",
    "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n",
    "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds), eps=1e-12)\n",
    "res_optim = {\"auc\": auc, \"logloss\": logloss}\n",
    "print(res_optim)\n",
    "sb.glue(\"res_optim\", res_optim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model saving and loading\n",
    "Now we finish the basic training and testing for LightGBM, next let's try to save and reload the model, and then evaluate it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.7757371640011422, 'logloss': 0.4606505068849181}\n"
     ]
    }
   ],
   "source": [
    "with TemporaryDirectory() as tmp:\n",
    "    save_file = os.path.join(tmp, r'finished.model')\n",
    "    lgb_model.save_model(save_file)\n",
    "    loaded_model = lgb.Booster(model_file=save_file)\n",
    "\n",
    "# eval the performance again\n",
    "test_preds = loaded_model.predict(test_x)\n",
    "\n",
    "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n",
    "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds), eps=1e-12)\n",
    "print({\"auc\": auc, \"logloss\": logloss})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Reading\n",
    "\n",
    "\\[1\\] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems. 3146–3154.<br>\n",
    "\\[2\\] The parameters of LightGBM: https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst <br>\n",
    "\\[3\\] Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363 (2018).<br>\n",
    "\\[4\\] Scikit-learn. 2018. categorical_encoding. https://github.com/scikit-learn-contrib/categorical-encoding<br>\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_pyspark)",
   "language": "python",
   "name": "reco_pyspark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}