experiments/04_PlanetKaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Experiment 04: Amazon Planet\n",
    "\n",
    "This experiment uses the data from the Kaggle competition [Planet: Understanding the Amazon from Space](https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/leaderboard). Here we use a pretrained ResNet50 model to generate the features from the dataset.\n",
    "\n",
    "For details of virtual machine we used and the versions of LightGBM and XGBoost, please refer to [experiment 1](01_airline.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.5.2 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:53:06) \n",
      "[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]\n",
      "XGBoost version: 0.6\n",
      "LightGBM version: 0.2\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "from collections import defaultdict\n",
    "import numpy as np\n",
    "import pkg_resources\n",
    "from libs.loaders import load_planet_kaggle\n",
    "from libs.planet_kaggle import threshold_prediction\n",
    "from libs.timer import Timer\n",
    "from libs.utils import get_number_processors\n",
    "from lightgbm import LGBMClassifier\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from tqdm import tqdm\n",
    "import json\n",
    "import warnings; warnings.simplefilter('ignore')\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"XGBoost version: {}\".format(pkg_resources.get_distribution('xgboost').version))\n",
    "print(\"LightGBM version: {}\".format(pkg_resources.get_distribution('lightgbm').version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: MOUNT_POINT=/datadrive\n"
     ]
    }
   ],
   "source": [
    "%env MOUNT_POINT=/datadrive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The images are loaded and featurised using a pretrained ResNet50 model available from Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train, X_test, y_test = load_planet_kaggle()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(35000, 2048)\n",
      "(35000, 17)\n",
      "(5479, 2048)\n",
      "(5479, 17)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## XGBoost \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of processors:  24\n"
     ]
    }
   ],
   "source": [
    "number_processors = get_number_processors()\n",
    "print(\"Number of processors: \", number_processors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will use a one-v-rest. So each classifier will be responsible for determining whether the assigned tag applies to the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def train_and_validate_xgboost(params, train_features, train_labels, validation_features, num_boost_round):\n",
    "    n_classes = train_labels.shape[1]\n",
    "    y_val_pred = np.zeros((validation_features.shape[0], n_classes))\n",
    "    time_results = defaultdict(list)\n",
    "    for class_i in tqdm(range(n_classes)):\n",
    "        dtrain = xgb.DMatrix(data=train_features, label=train_labels[:, class_i])\n",
    "        dtest = xgb.DMatrix(data=validation_features)\n",
    "        with Timer() as t:\n",
    "            model = xgb.train(params, dtrain, num_boost_round=num_boost_round)\n",
    "        time_results['train_time'].append(t.interval)\n",
    "        \n",
    "        with Timer() as t:\n",
    "            y_val_pred[:, class_i] = model.predict(dtest)\n",
    "        time_results['test_time'].append(t.interval)\n",
    "        \n",
    "    return y_val_pred, time_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def train_and_validate_lightgbm(params, train_features, train_labels, validation_features, num_boost_round):\n",
    "    n_classes = train_labels.shape[1]\n",
    "    y_val_pred = np.zeros((validation_features.shape[0], n_classes))\n",
    "    time_results = defaultdict(list)\n",
    "    for class_i in tqdm(range(n_classes)):\n",
    "        lgb_train = lgb.Dataset(train_features, train_labels[:, class_i], free_raw_data=False)\n",
    "        with Timer() as t:\n",
    "            model = lgb.train(params, lgb_train, num_boost_round = num_boost_round)\n",
    "        time_results['train_time'].append(t.interval)\n",
    "        \n",
    "        with Timer() as t:\n",
    "            y_val_pred[:, class_i] = model.predict(validation_features)\n",
    "        time_results['test_time'].append(t.interval)\n",
    "        \n",
    "    return y_val_pred, time_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "metrics_dict = {\n",
    "    'Accuracy': accuracy_score,\n",
    "    'Precision': lambda y_true, y_pred: precision_score(y_true, y_pred, average='samples'),\n",
    "    'Recall': lambda y_true, y_pred: recall_score(y_true, y_pred, average='samples'),\n",
    "    'F1': lambda y_true, y_pred: f1_score(y_true, y_pred, average='samples'),\n",
    "}\n",
    "\n",
    "def classification_metrics(metrics, y_true, y_pred):\n",
    "    return {metric_name:metric(y_true, y_pred) for metric_name, metric in metrics.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict = dict()\n",
    "num_rounds = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we are going to define the different models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "xgb_params = {'max_depth':6, \n",
    "              'objective':'binary:logistic', \n",
    "              'min_child_weight':1, \n",
    "              'learning_rate':0.1, \n",
    "              'scale_pos_weight':2, \n",
    "              'gamma':0.1, \n",
    "              'reg_lamda':1, \n",
    "              'subsample':1,\n",
    "              'nthread':number_processors\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 17/17 [05:36<00:00, 19.88s/it]\n"
     ]
    }
   ],
   "source": [
    "y_pred, timing_results = train_and_validate_xgboost(xgb_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['xgb']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "xgb_hist_params = {'max_depth':0, \n",
    "                  'max_leaves':2**6, \n",
    "                  'objective':'binary:logistic', \n",
    "                  'min_child_weight':1, \n",
    "                  'learning_rate':0.1, \n",
    "                  'scale_pos_weight':2, \n",
    "                  'gamma':0.1, \n",
    "                  'reg_lamda':1, \n",
    "                  'subsample':1,\n",
    "                  'nthread':number_processors,\n",
    "                  'tree_method':'hist', \n",
    "                  'grow_policy':'lossguide',\n",
    "                  'max_bins': 63\n",
    "                 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 17/17 [35:26<00:00, 116.33s/it]\n"
     ]
    }
   ],
   "source": [
    "y_pred, timing_results = train_and_validate_xgboost(xgb_hist_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['xgb_hist']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lgb_params = {'num_leaves': 2**6,\n",
    "             'learning_rate': 0.1,\n",
    "             'scale_pos_weight': 2,\n",
    "             'min_split_gain': 0.1,\n",
    "             'min_child_weight': 1,\n",
    "             'reg_lambda': 1,\n",
    "             'subsample': 1,\n",
    "             'objective':'binary',\n",
    "             'task': 'train',\n",
    "             'nthread':number_processors,\n",
    "             'max_bin': 63\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 17/17 [03:13<00:00,  7.91s/it]\n"
     ]
    }
   ],
   "source": [
    "y_pred, timing_results = train_and_validate_lightgbm(lgb_params, X_train, y_train, X_test, num_boost_round=num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "results_dict['lgbm']={\n",
    "    'train_time': np.sum(timing_results['train_time']),\n",
    "    'test_time': np.sum(timing_results['test_time']),\n",
    "    'performance': classification_metrics(metrics_dict, \n",
    "                                          y_test, \n",
    "                                          threshold_prediction(y_pred, threshold=0.1)) \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we show the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"lgbm\": {\n",
      "        \"performance\": {\n",
      "            \"Accuracy\": 0.37233071728417594,\n",
      "            \"F1\": 0.822258366139549,\n",
      "            \"Precision\": 0.7439077632634851,\n",
      "            \"Recall\": 0.9734099462015139\n",
      "        },\n",
      "        \"test_time\": 0.1641630920021271,\n",
      "        \"train_time\": 194.57900593099475\n",
      "    },\n",
      "    \"xgb\": {\n",
      "        \"performance\": {\n",
      "            \"Accuracy\": 0.34057309728052565,\n",
      "            \"F1\": 0.8048263053953228,\n",
      "            \"Precision\": 0.7184218531362171,\n",
      "            \"Recall\": 0.9766441564762427\n",
      "        },\n",
      "        \"test_time\": 0.1852665030019125,\n",
      "        \"train_time\": 313.8951129560046\n",
      "    },\n",
      "    \"xgb_hist\": {\n",
      "        \"performance\": {\n",
      "            \"Accuracy\": 0.37871874429640445,\n",
      "            \"F1\": 0.8220252909027159,\n",
      "            \"Precision\": 0.7447899193746976,\n",
      "            \"Recall\": 0.9720717197264013\n",
      "        },\n",
      "        \"test_time\": 0.19687007299944526,\n",
      "        \"train_time\": 2115.2851170680005\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Results\n",
    "print(json.dumps(results_dict, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This dataset shows an interesting behavior. It is the only notebook where XGBoost hist behaves worse than XGBoost. The reason could be because the number of features is high, 2048, and that could be causing a memory overhead. LightGBM and the standard version of XGBoost can manage this high number of features, so there is no overhead. You can try to use a higher complexity to improve the performance. For example, setting `max_depth=8` in XGBoost, `max_leaves=2**8` in XGBoost hist and `num_leaves=2**6` in LightGBM. This will cause an overhead in XGBoost hist."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python3.5 (Strata)",
   "language": "python",
   "name": "strata"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}