conference_notebooks/KDD_2019/plasticc/Part_2-2_Train_XGBoost_&_MLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2.2 - Train Final XGBoost Classifier\n",
    "We will use XGBoost to train a model for predicting the astronomical classes. XGBoost uses a machine learning technique called gradient boosting. Similar in concept to random forests, gradient boosting uses an ensemble of decision trees for prediction.\n",
    "\n",
    "While random forest ensembles can be built in parallel, gradient boosting ensembles are built by iteratively adding decision trees, which minimize some objective function, to the forest. Each iteration is focused on improving predictions for observations that performed poorly in previous iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf as gd\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "import xgboost as xgb\n",
    "from termcolor import colored\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from utils import xgb_cross_entropy_loss, cross_entropy\n",
    "\n",
    "from cuml.preprocessing import LabelEncoder\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_final_gd = gd.from_pandas(pd.read_pickle(\"train_gdf.pkl\"))\n",
    "test_final_gd = gd.from_pandas(pd.read_pickle(\"test_gdf.pkl\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_final_gd.head().to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's extract the set of unique classes from our training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = train_final_gd['target']\n",
    "classes = sorted(np.unique(y.to_array()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our classifier will expect labels to be in the range `[0, n_classes-1]`, we can use cuML's `LabelEncoder()` function to encode them in the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = LabelEncoder().fit_transform(y).to_array()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess our columns to fill `nan` values with zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = [i for i in test_final_gd.columns if i not in ['object_id','target']]\n",
    "for col in cols:\n",
    "    train_final_gd[col] = train_final_gd[col].fillna(0).astype('float32')\n",
    "\n",
    "for col in cols:\n",
    "    test_final_gd[col] = test_final_gd[col].fillna(0).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `train_test_split()` function from Scikit-learn to perform a stratified split of our training dataset into 90% training and 10% validation datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = train_final_gd[cols].as_matrix()\n",
    "Xt = test_final_gd[cols].as_matrix()\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.1,stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost models are configured using a dictionary for parameters. You can learn more about the various different configuration parameters in the XGBoost [docs](https://xgboost.readthedocs.io/en/latest/parameter.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_params = {\n",
    "            'objective': 'multi:softprob',    # softmax, return probabilities for each class\n",
    "            'nthread': 16,                    \n",
    "            'num_class':len(classes),         \n",
    "            'max_depth': 7,                   \n",
    "            'silent': 1,  \n",
    "            'subsample':0.7,\n",
    "            'colsample_bytree': 0.7,\n",
    "            \"tree_method\": \"gpu_hist\"         # compute histograms for splits on GPU\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost uses a `DMatrix` object to represent data inputs. We can construct them with our training, test, and validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtrain = xgb.DMatrix(data=X_train, label=y_train)\n",
    "dvalid = xgb.DMatrix(data=X_test, label=y_test)\n",
    "dtest = xgb.DMatrix(data=Xt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train our XGBoost model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "watchlist = [(dvalid, 'eval'), (dtrain, 'train')]\n",
    "\n",
    "clf = xgb.train(gpu_params, \n",
    "                dtrain=dtrain,\n",
    "                num_boost_round=60,\n",
    "                evals=watchlist,\n",
    "                feval=xgb_cross_entropy_loss(classes),\n",
    "                early_stopping_rounds=10,\n",
    "                verbose_eval=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yp = clf.predict(dvalid)\n",
    "\n",
    "gpu_loss = cross_entropy(y_test, yp, classes)\n",
    "\n",
    "ysub = clf.predict(dtest)\n",
    "\n",
    "line = 'validation loss %.4f'%gpu_loss\n",
    "print(colored(line,'green'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indepdendent Exercise\n",
    "\n",
    "Now that `yp` contains output probabilities for each class in the validation dataset, the accuracy can be computed by taking the number of correct predictions and diving by the total number of predictions. \n",
    "\n",
    "Recall that the result from `predict()` is an array of size `(n_samples, n_classes)` containing the class probabilities for each sample. \n",
    "\n",
    "In the cell below, use these class probabilties to compute `y_pred` so that it contains the actual predicted class labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = # Compute the actual class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, we would use our validation data to tune our model and our test data to evaluate final performance of our model. \n",
    "\n",
    "We don't have labels for our test set so we will just compute the accuracy of our validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(y_pred, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A confusion matrix will give us a good indication of the performance for each class in our validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import plot_confusion_matrix\n",
    "plot_confusion_matrix(y_test, y_pred, np.arange(1, len(classes)+1), normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Exercise\n",
    "\n",
    "Now that you have trained an XGBoost model using both the timeseries embeddings and the statistical features,\n",
    "\n",
    "1. Re-run the previous notebook, but don't merge the timseries features into your training and test datasets before storing them. \n",
    "2. Use your new dataset to train a new XGBoost classifier. \n",
    "3. Compare the accuracy and confusion matrix against the model that included the timeseries embedding features. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rapids",
   "language": "python",
   "name": "rapids"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}