experiments/01_airline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Experiment 01: Airline dataset\n",
    "\n",
    "In this notebook we are going to perform a prediction experiment with XGBoost and LightGBM using the Airline dataset. We are also going to analyze if the dataset has [concept drift](https://en.wikipedia.org/wiki/Concept_drift). \n",
    "\n",
    "In this experiment we use [the airline dataset](http://kt.ijs.si/elena_ikonomovska/data.html) to predict arrival delay. The dataset consists of a large amount of records, containing flight arrival and departure details for all the commercial flights within the USA, from October 1987 to April 2008. Its size is around 116 million records and 5.76 GB of memory.\n",
    "\n",
    "The machine we used was an [Azure NV24 VM](https://azure.microsoft.com/en-gb/blog/azure-n-series-general-availability-on-december-1/), which has [NVIDIA M60](http://www.nvidia.com/object/tesla-m60.html) GPUs. Its operating system is Ubuntu 16.04.\n",
    "\n",
    "For both XGBoost and LightGBM we compiled from source, to get the last improvements. In XGboost we used the commit `6776292951565c8cd72e69afd9d94de1474f00c0` of May 26th. For LightGBM we used the commit `73968a96829e212b333c88cd44725c8c39c03ad1` of June 2nd. To get these versions and replicate our experiments:\n",
    "```python\n",
    "git clone --recursive *url_of_library*\n",
    "git checkout *oldcommit*\n",
    "git submodule update --recursive\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.5.3 |Anaconda 4.4.0 (64-bit)| (default, Mar  6 2017, 11:58:13) \n",
      "[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]\n",
      "XGBoost version: 0.6\n",
      "LightGBM version: 0.2\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import sys\n",
    "import warnings\n",
    "\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pkg_resources\n",
    "from IPython.display import SVG, display\n",
    "from bokeh.io import export_svgs\n",
    "from bokeh.io import output_notebook, show\n",
    "from bokeh.plotting import figure\n",
    "from libs.conversion import convert_cols_categorical_to_numeric, convert_related_cols_categorical_to_numeric\n",
    "from libs.loaders import load_airline\n",
    "from libs.notebook_memory_management import start_watching_memory\n",
    "from libs.timer import Timer\n",
    "from libs.utils import get_number_processors\n",
    "from lightgbm.sklearn import LGBMRegressor, LGBMClassifier\n",
    "from sklearn.metrics import (accuracy_score, roc_auc_score, f1_score, precision_score,\n",
    "                             recall_score)\n",
    "from libs.metrics import classification_metrics_binary, classification_metrics_binary_prob, binarize_prediction\n",
    "from toolz import curry\n",
    "from xgboost import XGBRegressor\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"XGBoost version: {}\".format(pkg_resources.get_distribution('xgboost').version))\n",
    "print(\"LightGBM version: {}\".format(pkg_resources.get_distribution('lightgbm').version))\n",
    "\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The dataset we are going to use in this notebook is huge, therefore we want to monitor the memory consumption. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=\"bk-root\">\n",
       "        <a href=\"http://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
       "        <span id=\"512bf407-c82b-45d4-81e7-09811187f69d\">Loading BokehJS ...</span>\n",
       "    </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "(function(global) {\n",
       "  function now() {\n",
       "    return new Date();\n",
       "  }\n",
       "\n",
       "  var force = true;\n",
       "\n",
       "  if (typeof (window._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
       "    window._bokeh_onload_callbacks = [];\n",
       "    window._bokeh_is_loading = undefined;\n",
       "  }\n",
       "\n",
       "\n",
       "  \n",
       "  if (typeof (window._bokeh_timeout) === \"undefined\" || force === true) {\n",
       "    window._bokeh_timeout = Date.now() + 5000;\n",
       "    window._bokeh_failed_load = false;\n",
       "  }\n",
       "\n",
       "  var NB_LOAD_WARNING = {'data': {'text/html':\n",
       "     \"<div style='background-color: #fdd'>\\n\"+\n",
       "     \"<p>\\n\"+\n",
       "     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
       "     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
       "     \"</p>\\n\"+\n",
       "     \"<ul>\\n\"+\n",
       "     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
       "     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
       "     \"</ul>\\n\"+\n",
       "     \"<code>\\n\"+\n",
       "     \"from bokeh.resources import INLINE\\n\"+\n",
       "     \"output_notebook(resources=INLINE)\\n\"+\n",
       "     \"</code>\\n\"+\n",
       "     \"</div>\"}};\n",
       "\n",
       "  function display_loaded() {\n",
       "    if (window.Bokeh !== undefined) {\n",
       "      var el = document.getElementById(\"512bf407-c82b-45d4-81e7-09811187f69d\");\n",
       "      el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
       "    } else if (Date.now() < window._bokeh_timeout) {\n",
       "      setTimeout(display_loaded, 100)\n",
       "    }\n",
       "  }\n",
       "\n",
       "  function run_callbacks() {\n",
       "    try {\n",
       "      window._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
       "    }\n",
       "    finally {\n",
       "      delete window._bokeh_onload_callbacks\n",
       "    }\n",
       "    console.info(\"Bokeh: all callbacks have finished\");\n",
       "  }\n",
       "\n",
       "  function load_libs(js_urls, callback) {\n",
       "    window._bokeh_onload_callbacks.push(callback);\n",
       "    if (window._bokeh_is_loading > 0) {\n",
       "      console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "      return null;\n",
       "    }\n",
       "    if (js_urls == null || js_urls.length === 0) {\n",
       "      run_callbacks();\n",
       "      return null;\n",
       "    }\n",
       "    console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "    window._bokeh_is_loading = js_urls.length;\n",
       "    for (var i = 0; i < js_urls.length; i++) {\n",
       "      var url = js_urls[i];\n",
       "      var s = document.createElement('script');\n",
       "      s.src = url;\n",
       "      s.async = false;\n",
       "      s.onreadystatechange = s.onload = function() {\n",
       "        window._bokeh_is_loading--;\n",
       "        if (window._bokeh_is_loading === 0) {\n",
       "          console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
       "          run_callbacks()\n",
       "        }\n",
       "      };\n",
       "      s.onerror = function() {\n",
       "        console.warn(\"failed to load library \" + url);\n",
       "      };\n",
       "      console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "    }\n",
       "  };var element = document.getElementById(\"512bf407-c82b-45d4-81e7-09811187f69d\");\n",
       "  if (element == null) {\n",
       "    console.log(\"Bokeh: ERROR: autoload.js configured with elementid '512bf407-c82b-45d4-81e7-09811187f69d' but no matching script tag was found. \")\n",
       "    return false;\n",
       "  }\n",
       "\n",
       "  var js_urls = [\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.6.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.6.min.js\"];\n",
       "\n",
       "  var inline_js = [\n",
       "    function(Bokeh) {\n",
       "      Bokeh.set_log_level(\"info\");\n",
       "    },\n",
       "    \n",
       "    function(Bokeh) {\n",
       "      \n",
       "    },\n",
       "    \n",
       "    function(Bokeh) {\n",
       "      \n",
       "      document.getElementById(\"512bf407-c82b-45d4-81e7-09811187f69d\").textContent = \"BokehJS is loading...\";\n",
       "    },\n",
       "    function(Bokeh) {\n",
       "      console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-0.12.6.min.css\");\n",
       "      Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.6.min.css\");\n",
       "      console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.6.min.css\");\n",
       "      Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.6.min.css\");\n",
       "    }\n",
       "  ];\n",
       "\n",
       "  function run_inline_js() {\n",
       "    \n",
       "    if ((window.Bokeh !== undefined) || (force === true)) {\n",
       "      for (var i = 0; i < inline_js.length; i++) {\n",
       "        inline_js[i](window.Bokeh);\n",
       "      }if (force === true) {\n",
       "        display_loaded();\n",
       "      }} else if (Date.now() < window._bokeh_timeout) {\n",
       "      setTimeout(run_inline_js, 100);\n",
       "    } else if (!window._bokeh_failed_load) {\n",
       "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "      window._bokeh_failed_load = true;\n",
       "    } else if (force !== true) {\n",
       "      var cell = $(document.getElementById(\"512bf407-c82b-45d4-81e7-09811187f69d\")).parents('.cell').data().cell;\n",
       "      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
       "    }\n",
       "\n",
       "  }\n",
       "\n",
       "  if (window._bokeh_is_loading === 0) {\n",
       "    console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
       "    run_inline_js();\n",
       "  } else {\n",
       "    load_libs(js_urls, function() {\n",
       "      console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "      run_inline_js();\n",
       "    });\n",
       "  }\n",
       "}(this));"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [3] used 11.8828 MiB RAM in 0.32s, total RAM usage 243.68 MiB\n"
     ]
    }
   ],
   "source": [
    "start_watching_memory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 1) XGBoost vs LightGBM benchmark\n",
    "In the next section we compare both libraries speed, accuracy and other metrics for the dataset of airline arrival delay. \n",
    "\n",
    "### Data loading and management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:libs.loaders:MOUNT_POINT not found in environment. Defaulting to /fileshare\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(115069017, 14)\n",
      "CPU times: user 1min 38s, sys: 15.4 s, total: 1min 53s\n",
      "Wall time: 3min 37s\n",
      "In [4] used 21994.5117 MiB RAM in 217.27s, total RAM usage 22238.19 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_plane = load_airline()\n",
    "print(df_plane.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayofWeek</th>\n",
       "      <th>CRSDepTime</th>\n",
       "      <th>CRSArrTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>FlightNum</th>\n",
       "      <th>ActualElapsedTime</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Diverted</th>\n",
       "      <th>ArrDelay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>556</td>\n",
       "      <td>AA</td>\n",
       "      <td>190</td>\n",
       "      <td>247</td>\n",
       "      <td>SFO</td>\n",
       "      <td>ORD</td>\n",
       "      <td>1846</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>114</td>\n",
       "      <td>EA</td>\n",
       "      <td>57</td>\n",
       "      <td>74</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SFO</td>\n",
       "      <td>337</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>35</td>\n",
       "      <td>HP</td>\n",
       "      <td>351</td>\n",
       "      <td>167</td>\n",
       "      <td>ICT</td>\n",
       "      <td>LAS</td>\n",
       "      <td>987</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>DL</td>\n",
       "      <td>251</td>\n",
       "      <td>35</td>\n",
       "      <td>MCO</td>\n",
       "      <td>PBI</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>-2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>517</td>\n",
       "      <td>UA</td>\n",
       "      <td>500</td>\n",
       "      <td>208</td>\n",
       "      <td>LAS</td>\n",
       "      <td>ORD</td>\n",
       "      <td>1515</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Month  DayofMonth  DayofWeek  CRSDepTime  CRSArrTime UniqueCarrier  \\\n",
       "0  1987     10           1          4           1         556            AA   \n",
       "1  1987     10           1          4           5         114            EA   \n",
       "2  1987     10           1          4           5          35            HP   \n",
       "3  1987     10           1          4           5          40            DL   \n",
       "4  1987     10           1          4           8         517            UA   \n",
       "\n",
       "   FlightNum  ActualElapsedTime Origin Dest  Distance  Diverted  ArrDelay  \n",
       "0        190                247    SFO  ORD      1846         0        27  \n",
       "1         57                 74    LAX  SFO       337         0         5  \n",
       "2        351                167    ICT  LAS       987         0        17  \n",
       "3        251                 35    MCO  PBI       142         0        -2  \n",
       "4        500                208    LAS  ORD      1515         0        17  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [5] used 0.3828 MiB RAM in 0.12s, total RAM usage 22238.57 MiB\n"
     ]
    }
   ],
   "source": [
    "df_plane.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The first step is to convert the categorical features to numeric features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 44s, sys: 6.62 s, total: 1min 51s\n",
      "Wall time: 1min 50s\n",
      "In [6] used 5270.3672 MiB RAM in 110.78s, total RAM usage 27508.94 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_plane_numeric = convert_related_cols_categorical_to_numeric(df_plane, col_list=['Origin','Dest'])\n",
    "del df_plane"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayofWeek</th>\n",
       "      <th>CRSDepTime</th>\n",
       "      <th>CRSArrTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>FlightNum</th>\n",
       "      <th>ActualElapsedTime</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Diverted</th>\n",
       "      <th>ArrDelay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>556</td>\n",
       "      <td>AA</td>\n",
       "      <td>190</td>\n",
       "      <td>247</td>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>1846</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>114</td>\n",
       "      <td>EA</td>\n",
       "      <td>57</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>337</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>35</td>\n",
       "      <td>HP</td>\n",
       "      <td>351</td>\n",
       "      <td>167</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>987</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>DL</td>\n",
       "      <td>251</td>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>-2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>517</td>\n",
       "      <td>UA</td>\n",
       "      <td>500</td>\n",
       "      <td>208</td>\n",
       "      <td>4</td>\n",
       "      <td>33</td>\n",
       "      <td>1515</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Month  DayofMonth  DayofWeek  CRSDepTime  CRSArrTime UniqueCarrier  \\\n",
       "0  1987     10           1          4           1         556            AA   \n",
       "1  1987     10           1          4           5         114            EA   \n",
       "2  1987     10           1          4           5          35            HP   \n",
       "3  1987     10           1          4           5          40            DL   \n",
       "4  1987     10           1          4           8         517            UA   \n",
       "\n",
       "   FlightNum  ActualElapsedTime  Origin  Dest  Distance  Diverted  ArrDelay  \n",
       "0        190                247       0    33      1846         0        27  \n",
       "1         57                 74       1     0       337         0         5  \n",
       "2        351                167       2     4       987         0        17  \n",
       "3        251                 35       3    41       142         0        -2  \n",
       "4        500                208       4    33      1515         0        17  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [7] used 0.0703 MiB RAM in 0.12s, total RAM usage 27509.01 MiB\n"
     ]
    }
   ],
   "source": [
    "df_plane_numeric.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min, sys: 6.96 s, total: 1min 7s\n",
      "Wall time: 1min 6s\n",
      "In [8] used 13168.7422 MiB RAM in 67.08s, total RAM usage 40677.75 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_plane_numeric = convert_cols_categorical_to_numeric(df_plane_numeric, col_list='UniqueCarrier')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayofWeek</th>\n",
       "      <th>CRSDepTime</th>\n",
       "      <th>CRSArrTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>FlightNum</th>\n",
       "      <th>ActualElapsedTime</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Diverted</th>\n",
       "      <th>ArrDelay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>556</td>\n",
       "      <td>0</td>\n",
       "      <td>190</td>\n",
       "      <td>247</td>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>1846</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>114</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>337</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>35</td>\n",
       "      <td>2</td>\n",
       "      <td>351</td>\n",
       "      <td>167</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>987</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>3</td>\n",
       "      <td>251</td>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>-2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>517</td>\n",
       "      <td>4</td>\n",
       "      <td>500</td>\n",
       "      <td>208</td>\n",
       "      <td>4</td>\n",
       "      <td>33</td>\n",
       "      <td>1515</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Month  DayofMonth  DayofWeek  CRSDepTime  CRSArrTime  UniqueCarrier  \\\n",
       "0  1987     10           1          4           1         556              0   \n",
       "1  1987     10           1          4           5         114              1   \n",
       "2  1987     10           1          4           5          35              2   \n",
       "3  1987     10           1          4           5          40              3   \n",
       "4  1987     10           1          4           8         517              4   \n",
       "\n",
       "   FlightNum  ActualElapsedTime  Origin  Dest  Distance  Diverted  ArrDelay  \n",
       "0        190                247       0    33      1846         0        27  \n",
       "1         57                 74       1     0       337         0         5  \n",
       "2        351                167       2     4       987         0        17  \n",
       "3        251                 35       3    41       142         0        -2  \n",
       "4        500                208       4    33      1515         0        17  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [9] used 0.0039 MiB RAM in 0.12s, total RAM usage 40677.75 MiB\n"
     ]
    }
   ],
   "source": [
    "df_plane_numeric.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To simplify the pipeline, we are going to set a classification problem where the goal is to classify wheather a flight has arrived delayed or not. For that we need to binarize the variable `ArrDelay`.\n",
    "\n",
    "If you want to extend this experiment, you can set a regression problem and try to identify the number of minutes of delay a fight has. Both XGBoost and LightGBM have regression classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [10] used 3072.6328 MiB RAM in 12.91s, total RAM usage 43750.39 MiB\n"
     ]
    }
   ],
   "source": [
    "df_plane_numeric = df_plane_numeric.apply(lambda x: x.astype('int16'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 440 ms, sys: 388 ms, total: 828 ms\n",
      "Wall time: 826 ms\n",
      "In [11] used 877.9102 MiB RAM in 0.93s, total RAM usage 44628.30 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_plane_numeric['ArrDelayBinary'] = 1*(df_plane_numeric['ArrDelay'] > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayofWeek</th>\n",
       "      <th>CRSDepTime</th>\n",
       "      <th>CRSArrTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>FlightNum</th>\n",
       "      <th>ActualElapsedTime</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Diverted</th>\n",
       "      <th>ArrDelay</th>\n",
       "      <th>ArrDelayBinary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>556</td>\n",
       "      <td>0</td>\n",
       "      <td>190</td>\n",
       "      <td>247</td>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>1846</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>114</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>337</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>35</td>\n",
       "      <td>2</td>\n",
       "      <td>351</td>\n",
       "      <td>167</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>987</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>3</td>\n",
       "      <td>251</td>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>41</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1987</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>517</td>\n",
       "      <td>4</td>\n",
       "      <td>500</td>\n",
       "      <td>208</td>\n",
       "      <td>4</td>\n",
       "      <td>33</td>\n",
       "      <td>1515</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Month  DayofMonth  DayofWeek  CRSDepTime  CRSArrTime  UniqueCarrier  \\\n",
       "0  1987     10           1          4           1         556              0   \n",
       "1  1987     10           1          4           5         114              1   \n",
       "2  1987     10           1          4           5          35              2   \n",
       "3  1987     10           1          4           5          40              3   \n",
       "4  1987     10           1          4           8         517              4   \n",
       "\n",
       "   FlightNum  ActualElapsedTime  Origin  Dest  Distance  Diverted  ArrDelay  \\\n",
       "0        190                247       0    33      1846         0        27   \n",
       "1         57                 74       1     0       337         0         5   \n",
       "2        351                167       2     4       987         0        17   \n",
       "3        251                 35       3    41       142         0        -2   \n",
       "4        500                208       4    33      1515         0        17   \n",
       "\n",
       "   ArrDelayBinary  \n",
       "0               1  \n",
       "1               1  \n",
       "2               1  \n",
       "3               0  \n",
       "4               1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [12] used 0.0000 MiB RAM in 0.12s, total RAM usage 44628.30 MiB\n"
     ]
    }
   ],
   "source": [
    "df_plane_numeric.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once the features are prepared, let's split the dataset into train and test set. We won't use validation for this example (however, you can try to add it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [13] used 0.0000 MiB RAM in 0.10s, total RAM usage 44628.30 MiB\n"
     ]
    }
   ],
   "source": [
    "def split_train_val_test_df(df, val_size=0.2, test_size=0.2):\n",
    "    train, validate, test = np.split(df.sample(frac=1), \n",
    "                                     [int((1-val_size-test_size)*len(df)), int((1-test_size)*len(df))])\n",
    "    return train, validate, test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(92055213, 15)\n",
      "(0, 15)\n",
      "(23013804, 15)\n",
      "CPU times: user 36.3 s, sys: 2.95 s, total: 39.2 s\n",
      "Wall time: 39 s\n",
      "In [14] used 4653.2422 MiB RAM in 39.13s, total RAM usage 49281.54 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "train, validate, test = split_train_val_test_df(df_plane_numeric, val_size=0, test_size=0.2)\n",
    "print(train.shape)\n",
    "print(validate.shape)\n",
    "print(test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [15] used 0.0000 MiB RAM in 0.11s, total RAM usage 49281.54 MiB\n"
     ]
    }
   ],
   "source": [
    "def generate_feables(df):\n",
    "    X = df[df.columns.difference(['ArrDelay', 'ArrDelayBinary'])]\n",
    "    y = df['ArrDelayBinary']\n",
    "    return X,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 868 ms, sys: 604 ms, total: 1.47 s\n",
      "Wall time: 1.47 s\n",
      "In [16] used 2852.9844 MiB RAM in 1.58s, total RAM usage 52134.52 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X_train, y_train = generate_feables(train)\n",
    "X_val, y_val = generate_feables(validate)\n",
    "X_test, y_test = generate_feables(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [17] used 0.2188 MiB RAM in 0.10s, total RAM usage 52134.74 MiB\n"
     ]
    }
   ],
   "source": [
    "del train, validate, test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Training \n",
    "Now we are going to create two pipelines, one of XGBoost and one for LightGBM. The technology behind both libraries is different, so it is difficult to compare them in the exact same model setting. XGBoost grows the trees depth-wise and controls model complexity with `max_depth`. Instead, LightGBM uses a leaf-wise algorithm and controls the model complexity by `num_leaves`. As a tradeoff, we use XGBoost with `max_depth=8`, which will have max number leaves of 255, and compare it with LightGBM with `num_leaves=255`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [18] used 0.0000 MiB RAM in 0.10s, total RAM usage 52134.74 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict = dict()\n",
    "num_rounds = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n",
      "In [19] used 0.0000 MiB RAM in 0.10s, total RAM usage 52134.74 MiB\n"
     ]
    }
   ],
   "source": [
    "number_processors = get_number_processors()\n",
    "print(number_processors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's start with the XGBoost classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [20] used 0.0000 MiB RAM in 0.10s, total RAM usage 52134.74 MiB\n"
     ]
    }
   ],
   "source": [
    "xgb_clf_pipeline = XGBRegressor(max_depth=8,\n",
    "                                n_estimators=num_rounds,\n",
    "                                min_child_weight=30,\n",
    "                                learning_rate=0.1,\n",
    "                                scale_pos_weight=2,\n",
    "                                gamma=0.1,\n",
    "                                reg_lambda=1,\n",
    "                                subsample=1,\n",
    "                                n_jobs=number_processors,\n",
    "                                random_state=77)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "```python\n",
    "with Timer() as t:\n",
    "    xgb_clf_pipeline.fit(X_train, y_train)\n",
    "\n",
    "results_dict['xgb']={ 'train_time': t.interval }\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Training XGBoost model with leave-wise growth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [21] used 0.0000 MiB RAM in 0.10s, total RAM usage 52134.74 MiB\n"
     ]
    }
   ],
   "source": [
    "xgb_hist_clf_pipeline = XGBRegressor(max_depth=0,\n",
    "                                    max_leaves=255,\n",
    "                                    n_estimators=num_rounds,\n",
    "                                    min_child_weight=30,\n",
    "                                    learning_rate=0.1,\n",
    "                                    scale_pos_weight=2,\n",
    "                                    gamma=0.1,\n",
    "                                    reg_lambda=1,\n",
    "                                    subsample=1,\n",
    "                                    grow_policy='lossguide',\n",
    "                                    tree_method='hist',\n",
    "                                    n_jobs=number_processors,\n",
    "                                    random_state=77)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [22] used 22842.2461 MiB RAM in 1236.61s, total RAM usage 74976.99 MiB\n"
     ]
    }
   ],
   "source": [
    "with Timer() as t:\n",
    "    xgb_hist_clf_pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [23] used 0.0078 MiB RAM in 0.10s, total RAM usage 74977.00 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['xgb_hist']={\n",
    "    'train_time': t.interval\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Training LightGBM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [24] used 0.0000 MiB RAM in 0.10s, total RAM usage 74977.00 MiB\n"
     ]
    }
   ],
   "source": [
    "lgbm_clf_pipeline = LGBMRegressor(num_leaves=255,\n",
    "                                  n_estimators=num_rounds,\n",
    "                                  min_child_weight=30,\n",
    "                                  learning_rate=0.1,\n",
    "                                  scale_pos_weight=2,\n",
    "                                  min_split_gain=0.1,\n",
    "                                  reg_lambda=1,\n",
    "                                  subsample=1,\n",
    "                                  nthread=number_processors,\n",
    "                                  seed=77)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [25] used 2371.1797 MiB RAM in 1051.03s, total RAM usage 77348.18 MiB\n"
     ]
    }
   ],
   "source": [
    "with Timer() as t:\n",
    "    lgbm_clf_pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [26] used 0.0039 MiB RAM in 0.10s, total RAM usage 77348.18 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['lgbm']={\n",
    "    'train_time': t.interval\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As it can be seen in the results, given the specific versions and parameters of both XGBoost and LightGBM and in this specific dataset, LightGBM is faster. \n",
    "\n",
    "In general terms, leaf-wise algorithms are more efficient, they converge much faster than depth-wise. However, it may cause over-fitting when the data is small or there are too many leaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Evaluation\n",
    "Now let's evaluate the model in the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "```python\n",
    "with Timer() as t:\n",
    "    y_prob_xgb = np.clip(xgb_clf_pipeline.predict(X_test), 0.0001, 0.9999)\n",
    "    \n",
    "results_dict['xgb']['test_time'] = t.interval\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [27] used 87.7930 MiB RAM in 36.21s, total RAM usage 77435.97 MiB\n"
     ]
    }
   ],
   "source": [
    "with Timer() as t:\n",
    "    y_prob_xgb_hist = np.clip(xgb_hist_clf_pipeline.predict(X_test), 0.0001, 0.9999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [28] used 0.0078 MiB RAM in 0.10s, total RAM usage 77435.98 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['xgb_hist']['test_time'] = t.interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [29] used 746.2227 MiB RAM in 45.10s, total RAM usage 78182.20 MiB\n"
     ]
    }
   ],
   "source": [
    "with Timer() as t:\n",
    "    y_prob_lgbm = np.clip(lgbm_clf_pipeline.predict(X_test), 0.0001, 0.9999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [30] used 0.0039 MiB RAM in 0.10s, total RAM usage 78182.21 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['lgbm']['test_time'] = t.interval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Metrics\n",
    "We are going to obtain some metrics to evaluate the performance of each of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [31] used 350.9219 MiB RAM in 0.42s, total RAM usage 78533.13 MiB\n"
     ]
    }
   ],
   "source": [
    "#y_pred_xgb = binarize_prediction(y_prob_xgb)\n",
    "y_pred_xgb_hist = binarize_prediction(y_prob_xgb_hist)\n",
    "y_pred_lgbm = binarize_prediction(y_prob_lgbm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "```python\n",
    "report_xgb = classification_metrics_binary(y_test, y_pred_xgb)\n",
    "report2_xgb = classification_metrics_binary_prob(y_test, y_prob_xgb)\n",
    "report_xgb.update(report2_xgb)\n",
    "\n",
    "results_dict['xgb']['performance'] = report_xgb\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [32] used 512.2422 MiB RAM in 29.18s, total RAM usage 79045.37 MiB\n"
     ]
    }
   ],
   "source": [
    "report_xgb_hist = classification_metrics_binary(y_test, y_pred_xgb_hist)\n",
    "report2_xgb_hist = classification_metrics_binary_prob(y_test, y_prob_xgb_hist)\n",
    "report_xgb_hist.update(report2_xgb_hist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [33] used 0.0039 MiB RAM in 0.10s, total RAM usage 79045.38 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['xgb_hist']['performance'] = report_xgb_hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [34] used 0.0000 MiB RAM in 25.08s, total RAM usage 79045.38 MiB\n"
     ]
    }
   ],
   "source": [
    "report_lgbm = classification_metrics_binary(y_test, y_pred_lgbm)\n",
    "report2_lgbm = classification_metrics_binary_prob(y_test, y_prob_lgbm)\n",
    "report_lgbm.update(report2_lgbm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [35] used 0.0000 MiB RAM in 0.10s, total RAM usage 79045.38 MiB\n"
     ]
    }
   ],
   "source": [
    "results_dict['lgbm']['performance'] = report_lgbm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"lgbm\": {\n",
      "        \"performance\": {\n",
      "            \"AUC\": 0.8402372555827792,\n",
      "            \"Accuracy\": 0.7643602509172321,\n",
      "            \"F1\": 0.7327938097266504,\n",
      "            \"Precision\": 0.7934939262260393,\n",
      "            \"Recall\": 0.6807205483883755\n",
      "        },\n",
      "        \"test_time\": 45.19994992000284,\n",
      "        \"train_time\": 1056.2073636010027\n",
      "    },\n",
      "    \"xgb_hist\": {\n",
      "        \"performance\": {\n",
      "            \"AUC\": 0.8384000721790869,\n",
      "            \"Accuracy\": 0.7163687932685965,\n",
      "            \"F1\": 0.7414381273804327,\n",
      "            \"Precision\": 0.6534895643276287,\n",
      "            \"Recall\": 0.856740851230127\n",
      "        },\n",
      "        \"test_time\": 36.26451232100226,\n",
      "        \"train_time\": 1242.098871958995\n",
      "    }\n",
      "}\n",
      "In [36] used 0.0000 MiB RAM in 0.10s, total RAM usage 79045.38 MiB\n"
     ]
    }
   ],
   "source": [
    "# Results\n",
    "print(json.dumps(results_dict, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The experiment shows a fairly similar performance in both libraries, being LightGBM slightly better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [37] used -28481.8008 MiB RAM in 0.18s, total RAM usage 50563.57 MiB\n"
     ]
    }
   ],
   "source": [
    "del xgb_clf_pipeline, xgb_hist_clf_pipeline, lgbm_clf_pipeline, X_train, X_test, X_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 2) Concept drift\n",
    "In this section we are trying to find concept drift in the dataset to check if retraining is valuable.\n",
    "\n",
    "### Data management\n",
    "We are going to pack the data yearly to try to find concept drift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [38] used 0.0000 MiB RAM in 0.10s, total RAM usage 50563.57 MiB\n"
     ]
    }
   ],
   "source": [
    "initial_year = 1987\n",
    "num_ini = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [39] used 0.0000 MiB RAM in 0.10s, total RAM usage 50563.57 MiB\n"
     ]
    }
   ],
   "source": [
    "def generate_subset_by_year(df, year_ini, year_end):\n",
    "    return df[df['Year'].isin(range(year_ini, year_end))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(16810190, 15)\n",
      "CPU times: user 1.54 s, sys: 336 ms, total: 1.87 s\n",
      "Wall time: 1.85 s\n",
      "In [40] used 577.1367 MiB RAM in 1.95s, total RAM usage 51140.71 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "subset_base = generate_subset_by_year(df_plane_numeric, initial_year, initial_year + num_ini)\n",
    "print(subset_base.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(98258827, 15)\n",
      "CPU times: user 1min 10s, sys: 4.7 s, total: 1min 14s\n",
      "Wall time: 1min 14s\n",
      "In [41] used 10267.1797 MiB RAM in 74.74s, total RAM usage 61407.89 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rest_df = df_plane_numeric.loc[df_plane_numeric.index.difference(subset_base.index)]\n",
    "print(rest_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Traininig\n",
    "Let's see what happens when we train on a subset of data and then evaluate in the data of the following years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [42] used -285.5078 MiB RAM in 0.32s, total RAM usage 61122.38 MiB\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = generate_feables(subset_base)\n",
    "del(subset_base)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [43] used 0.0000 MiB RAM in 0.10s, total RAM usage 61122.38 MiB\n"
     ]
    }
   ],
   "source": [
    "clf = LGBMClassifier(num_leaves=255,\n",
    "                    n_estimators=100,\n",
    "                    min_child_weight=30,\n",
    "                    learning_rate=0.1,\n",
    "                    subsample=0.80,\n",
    "                    colsample_bytree=0.80,\n",
    "                    seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28min 52s, sys: 55.9 s, total: 29min 48s\n",
      "Wall time: 1min 19s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', colsample_bytree=0.8, learning_rate=0.1,\n",
       "        max_bin=255, max_depth=-1, min_child_samples=10,\n",
       "        min_child_weight=30, min_split_gain=0, n_estimators=100,\n",
       "        nthread=-1, num_leaves=255, objective='binary', reg_alpha=0,\n",
       "        reg_lambda=0, seed=42, silent=True, subsample=0.8,\n",
       "        subsample_for_bin=50000, subsample_freq=1)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [44] used -2008.7539 MiB RAM in 79.43s, total RAM usage 59113.63 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [45] used 0.0000 MiB RAM in 0.10s, total RAM usage 59113.63 MiB\n"
     ]
    }
   ],
   "source": [
    "@curry\n",
    "def predict_accuracy(clf, test_df):\n",
    "    X_test, y_test = generate_feables(test_df)\n",
    "    y_pred = clf.predict(X_test)\n",
    "    return accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year\n",
      "1992    0.757375\n",
      "1993    0.756093\n",
      "1994    0.744296\n",
      "1995    0.731503\n",
      "1996    0.722490\n",
      "1997    0.720751\n",
      "1998    0.706178\n",
      "1999    0.700762\n",
      "2000    0.689513\n",
      "2001    0.674223\n",
      "2002    0.680176\n",
      "2003    0.686131\n",
      "2004    0.680697\n",
      "2005    0.673467\n",
      "2006    0.663597\n",
      "2007    0.651520\n",
      "2008    0.636613\n",
      "dtype: float64\n",
      "CPU times: user 35min 40s, sys: 23.3 s, total: 36min 4s\n",
      "Wall time: 2min 9s\n",
      "In [46] used 5923.4336 MiB RAM in 129.90s, total RAM usage 65037.06 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "accuracy_series = rest_df.groupby('Year').apply(predict_accuracy(clf))\n",
    "print(accuracy_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "From the results we can observe that the accuracy of the model gets worse as the years pass on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Retraining\n",
    "Now let's see what happens when we retrain and evaluate in the data of the following years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [47] used 0.0000 MiB RAM in 0.10s, total RAM usage 65037.06 MiB\n"
     ]
    }
   ],
   "source": [
    "new_init = 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(69425349, 15)\n",
      "CPU times: user 4.36 s, sys: 1.56 s, total: 5.93 s\n",
      "Wall time: 5.9 s\n",
      "In [48] used 2913.2109 MiB RAM in 6.01s, total RAM usage 67950.27 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "subset_retrain = generate_subset_by_year(df_plane_numeric, initial_year, initial_year + new_init) \n",
    "print(subset_retrain.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [49] used 1176.3672 MiB RAM in 0.98s, total RAM usage 69126.64 MiB\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = generate_feables(subset_retrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [50] used 0.0000 MiB RAM in 0.10s, total RAM usage 69126.64 MiB\n"
     ]
    }
   ],
   "source": [
    "clf_retrain = LGBMClassifier(num_leaves=255,\n",
    "                            n_estimators=100,\n",
    "                            min_child_weight=30,\n",
    "                            learning_rate=0.1,\n",
    "                            subsample=0.80,\n",
    "                            colsample_bytree=0.80,\n",
    "                            seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1h 40min 56s, sys: 2min 28s, total: 1h 43min 25s\n",
      "Wall time: 4min 40s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', colsample_bytree=0.8, learning_rate=0.1,\n",
       "        max_bin=255, max_depth=-1, min_child_samples=10,\n",
       "        min_child_weight=30, min_split_gain=0, n_estimators=100,\n",
       "        nthread=-1, num_leaves=255, objective='binary', reg_alpha=0,\n",
       "        reg_lambda=0, seed=42, silent=True, subsample=0.8,\n",
       "        subsample_for_bin=50000, subsample_freq=1)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [51] used 4634.6836 MiB RAM in 280.48s, total RAM usage 73761.32 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf_retrain.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(45643668, 15)\n",
      "CPU times: user 29.3 s, sys: 3.16 s, total: 32.4 s\n",
      "Wall time: 30.6 s\n",
      "In [52] used -1183.8203 MiB RAM in 30.74s, total RAM usage 72577.50 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rest_df = df_plane_numeric.loc[df_plane_numeric.index.difference(subset_retrain.index)]\n",
    "print(rest_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year\n",
      "2002    0.749985\n",
      "2003    0.748284\n",
      "2004    0.724928\n",
      "2005    0.719791\n",
      "2006    0.707446\n",
      "2007    0.695424\n",
      "2008    0.697949\n",
      "dtype: float64\n",
      "CPU times: user 14min 47s, sys: 10.1 s, total: 14min 57s\n",
      "Wall time: 56 s\n",
      "In [53] used -1113.7734 MiB RAM in 56.14s, total RAM usage 71463.73 MiB\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "accuracy_retrain = rest_df.groupby('Year').apply(predict_accuracy(clf_retrain))\n",
    "print(accuracy_retrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [54] used 0.0000 MiB RAM in 0.11s, total RAM usage 71463.73 MiB\n"
     ]
    }
   ],
   "source": [
    "def plot_metrics(metric1, metric2, legend1=None, legend2=None, x_label=None, y_label=None):\n",
    "    lists = sorted(metric1.items()) \n",
    "    x, y = zip(*lists) \n",
    "    fig, ax = plt.subplots()\n",
    "    ax.plot(x, y, label=legend1, color='#5975a4')\n",
    "    lists2 = sorted(metric2.items()) \n",
    "    x2, y2 = zip(*lists2) \n",
    "    ax.plot(x2, y2, label=legend2, color='#5f9e6f')\n",
    "    legend = ax.legend(loc=0)\n",
    "    ax.set_xlabel(x_label)\n",
    "    ax.set_ylabel(y_label)\n",
    "    plt.show()\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [55] used 0.0000 MiB RAM in 0.10s, total RAM usage 71463.73 MiB\n"
     ]
    }
   ],
   "source": [
    "from bokeh.models.sources import ColumnDataSource\n",
    "from bokeh.models import HoverTool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [56] used 0.1250 MiB RAM in 0.09s, total RAM usage 71463.86 MiB\n"
     ]
    }
   ],
   "source": [
    "data_cds = ColumnDataSource(pd.DataFrame({\n",
    "    'train_acc': accuracy_series,\n",
    "    'retrain_acc': accuracy_retrain\n",
    "}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "    <div class=\"bk-root\">\n",
       "        <div class=\"bk-plotdiv\" id=\"418a5452-0f41-427a-8537-ab1314ad1dce\"></div>\n",
       "    </div>\n",
       "<script type=\"text/javascript\">\n",
       "  \n",
       "  (function(global) {\n",
       "    function now() {\n",
       "      return new Date();\n",
       "    }\n",
       "  \n",
       "    var force = false;\n",
       "  \n",
       "    if (typeof (window._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
       "      window._bokeh_onload_callbacks = [];\n",
       "      window._bokeh_is_loading = undefined;\n",
       "    }\n",
       "  \n",
       "  \n",
       "    \n",
       "    if (typeof (window._bokeh_timeout) === \"undefined\" || force === true) {\n",
       "      window._bokeh_timeout = Date.now() + 0;\n",
       "      window._bokeh_failed_load = false;\n",
       "    }\n",
       "  \n",
       "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
       "       \"<div style='background-color: #fdd'>\\n\"+\n",
       "       \"<p>\\n\"+\n",
       "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
       "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
       "       \"</p>\\n\"+\n",
       "       \"<ul>\\n\"+\n",
       "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
       "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
       "       \"</ul>\\n\"+\n",
       "       \"<code>\\n\"+\n",
       "       \"from bokeh.resources import INLINE\\n\"+\n",
       "       \"output_notebook(resources=INLINE)\\n\"+\n",
       "       \"</code>\\n\"+\n",
       "       \"</div>\"}};\n",
       "  \n",
       "    function display_loaded() {\n",
       "      if (window.Bokeh !== undefined) {\n",
       "        var el = document.getElementById(\"418a5452-0f41-427a-8537-ab1314ad1dce\");\n",
       "        el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
       "      } else if (Date.now() < window._bokeh_timeout) {\n",
       "        setTimeout(display_loaded, 100)\n",
       "      }\n",
       "    }\n",
       "  \n",
       "    function run_callbacks() {\n",
       "      try {\n",
       "        window._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
       "      }\n",
       "      finally {\n",
       "        delete window._bokeh_onload_callbacks\n",
       "      }\n",
       "      console.info(\"Bokeh: all callbacks have finished\");\n",
       "    }\n",
       "  \n",
       "    function load_libs(js_urls, callback) {\n",
       "      window._bokeh_onload_callbacks.push(callback);\n",
       "      if (window._bokeh_is_loading > 0) {\n",
       "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "        return null;\n",
       "      }\n",
       "      if (js_urls == null || js_urls.length === 0) {\n",
       "        run_callbacks();\n",
       "        return null;\n",
       "      }\n",
       "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "      window._bokeh_is_loading = js_urls.length;\n",
       "      for (var i = 0; i < js_urls.length; i++) {\n",
       "        var url = js_urls[i];\n",
       "        var s = document.createElement('script');\n",
       "        s.src = url;\n",
       "        s.async = false;\n",
       "        s.onreadystatechange = s.onload = function() {\n",
       "          window._bokeh_is_loading--;\n",
       "          if (window._bokeh_is_loading === 0) {\n",
       "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
       "            run_callbacks()\n",
       "          }\n",
       "        };\n",
       "        s.onerror = function() {\n",
       "          console.warn(\"failed to load library \" + url);\n",
       "        };\n",
       "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      }\n",
       "    };var element = document.getElementById(\"418a5452-0f41-427a-8537-ab1314ad1dce\");\n",
       "    if (element == null) {\n",
       "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '418a5452-0f41-427a-8537-ab1314ad1dce' but no matching script tag was found. \")\n",
       "      return false;\n",
       "    }\n",
       "  \n",
       "    var js_urls = [];\n",
       "  \n",
       "    var inline_js = [\n",
       "      function(Bokeh) {\n",
       "        (function() {\n",
       "          var fn = function() {\n",
       "            var docs_json = {\"e7fa339d-3562-48e3-8583-ebdecfbcd60a\":{\"roots\":{\"references\":[{\"attributes\":{},\"id\":\"6b660d2f-f07f-434a-ab59-9fa76d814205\",\"type\":\"LinearScale\"},{\"attributes\":{\"bottom_units\":\"screen\",\"fill_alpha\":{\"value\":0.5},\"fill_color\":{\"value\":\"lightgrey\"},\"left_units\":\"screen\",\"level\":\"overlay\",\"line_alpha\":{\"value\":1.0},\"line_color\":{\"value\":\"black\"},\"line_dash\":[4,4],\"line_width\":{\"value\":2},\"plot\":null,\"render_mode\":\"css\",\"right_units\":\"screen\",\"top_units\":\"screen\"},\"id\":\"49000e90-873c-4cbb-8655-365ffa46adf2\",\"type\":\"BoxAnnotation\"},{\"attributes\":{\"line_alpha\":{\"value\":0.1},\"line_cap\":\"round\",\"line_color\":{\"value\":\"#1f77b4\"},\"line_width\":{\"value\":6},\"x\":{\"field\":\"Year\"},\"y\":{\"field\":\"train_acc\"}},\"id\":\"f462be3a-b3d2-4ff8-822a-55a2d6f2d6cf\",\"type\":\"Line\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"93bf2247-9d54-4267-a7ca-a9e95d1823fd\",\"type\":\"PanTool\"},{\"id\":\"3209c089-f316-4982-abb3-7b7d70cb1010\",\"type\":\"WheelZoomTool\"},{\"id\":\"9cb6adb4-3763-4936-83d8-d3f3ef0da6c3\",\"type\":\"BoxZoomTool\"},{\"id\":\"c3896de3-0531-49f2-ae8c-a54c340052fd\",\"type\":\"ResetTool\"},{\"id\":\"037d20e5-51a2-4d65-ac48-0d0d2ea9e8de\",\"type\":\"HoverTool\"}]},\"id\":\"527bb0f1-0907-48b7-97a3-dc70f93f8e60\",\"type\":\"Toolbar\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"Year\",\"retrain_acc\",\"train_acc\"],\"data\":{\"Year\":[1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008],\"retrain_acc\":{\"__ndarray__\":\"AAAAAAAA+H8AAAAAAAD4fwAAAAAAAPh/AAAAAAAA+H8AAAAAAAD4fwAAAAAAAPh/AAAAAAAA+H8AAAAAAAD4fwAAAAAAAPh/AAAAAAAA+H9p7Z6H4P/nP+LNMkfw8ec/yUzwYZwy5z+P6ZsthwjnP3PZJ0hlo+Y/HpCZJ+pA5j+TOzRemFXmPw==\",\"dtype\":\"float64\",\"shape\":[17]},\"train_acc\":{\"__ndarray__\":\"XSTXhGs86D8amxN06THoP5N2tEJG0ec/Vx2k8ndo5z/OEjdfox7nP2T0+yhlEOc//MgQPQOZ5j9Hkax0pGzmP6pJ/U59EOY/HhetHjyT5T+9rVecAMTlPxLE717J9OU/EKZYBkXI5T+IXGYtCo3lP56mOGAwPOU/RdeycEHZ5D//c+OhIV/kPw==\",\"dtype\":\"float64\",\"shape\":[17]}}},\"id\":\"6ed81d3d-2835-4ca6-b2fc-1a0249c6ae79\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"data_source\":{\"id\":\"6ed81d3d-2835-4ca6-b2fc-1a0249c6ae79\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"5e73e87e-8055-4e57-aeb4-102d0656101e\",\"type\":\"Line\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"f462be3a-b3d2-4ff8-822a-55a2d6f2d6cf\",\"type\":\"Line\"},\"selection_glyph\":null},\"id\":\"94881363-659d-4e67-a976-e012c03bf336\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"370c8f66-3d41-40d2-905d-3709f5c11f15\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1e0eeb42-c4a4-4280-92bd-9ee68960ecea\",\"type\":\"LinearScale\"},{\"attributes\":{\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"50ccf509-88f1-4bd2-bf7f-c7159b29ecbe\",\"type\":\"BasicTicker\"}},\"id\":\"28088f8f-ba79-408a-b690-a699ca585d97\",\"type\":\"Grid\"},{\"attributes\":{\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"}},\"id\":\"93bf2247-9d54-4267-a7ca-a9e95d1823fd\",\"type\":\"PanTool\"},{\"attributes\":{\"axis_label\":\"Accuracy\",\"formatter\":{\"id\":\"4750e200-556a-4c98-993f-4492f0bdc5e4\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"db69d08a-b02a-433d-a63f-7c9dfe7fa96b\",\"type\":\"BasicTicker\"}},\"id\":\"4f993ab1-8945-46b9-9b10-a123b31b8962\",\"type\":\"LinearAxis\"},{\"attributes\":{\"overlay\":{\"id\":\"49000e90-873c-4cbb-8655-365ffa46adf2\",\"type\":\"BoxAnnotation\"},\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"}},\"id\":\"9cb6adb4-3763-4936-83d8-d3f3ef0da6c3\",\"type\":\"BoxZoomTool\"},{\"attributes\":{\"plot\":null,\"text\":\"\"},\"id\":\"9d16c9fe-108d-45aa-a002-1c9551a768ba\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"50ccf509-88f1-4bd2-bf7f-c7159b29ecbe\",\"type\":\"BasicTicker\"},{\"attributes\":{\"formatter\":{\"id\":\"370c8f66-3d41-40d2-905d-3709f5c11f15\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"50ccf509-88f1-4bd2-bf7f-c7159b29ecbe\",\"type\":\"BasicTicker\"}},\"id\":\"a2d3e295-3313-4571-a237-f266a9ccc5a2\",\"type\":\"LinearAxis\"},{\"attributes\":{\"callback\":null,\"mode\":\"vline\",\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"renderers\":[{\"id\":\"94881363-659d-4e67-a976-e012c03bf336\",\"type\":\"GlyphRenderer\"}],\"tooltips\":[[\"Train\",\"@{train_acc}{0.4f}\"],[\"Retrain\",\"@{retrain_acc}{0.4f}\"]]},\"id\":\"037d20e5-51a2-4d65-ac48-0d0d2ea9e8de\",\"type\":\"HoverTool\"},{\"attributes\":{\"line_alpha\":{\"value\":0.1},\"line_cap\":\"round\",\"line_color\":{\"value\":\"#1f77b4\"},\"line_width\":{\"value\":6},\"x\":{\"field\":\"Year\"},\"y\":{\"field\":\"retrain_acc\"}},\"id\":\"9ebf72d4-aae1-4bf6-a947-3c78706a6200\",\"type\":\"Line\"},{\"attributes\":{\"below\":[{\"id\":\"a2d3e295-3313-4571-a237-f266a9ccc5a2\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"4f993ab1-8945-46b9-9b10-a123b31b8962\",\"type\":\"LinearAxis\"}],\"plot_height\":350,\"plot_width\":700,\"renderers\":[{\"id\":\"a2d3e295-3313-4571-a237-f266a9ccc5a2\",\"type\":\"LinearAxis\"},{\"id\":\"28088f8f-ba79-408a-b690-a699ca585d97\",\"type\":\"Grid\"},{\"id\":\"4f993ab1-8945-46b9-9b10-a123b31b8962\",\"type\":\"LinearAxis\"},{\"id\":\"9abb3969-b48a-4fcc-ad9e-938fae803ed3\",\"type\":\"Grid\"},{\"id\":\"49000e90-873c-4cbb-8655-365ffa46adf2\",\"type\":\"BoxAnnotation\"},{\"id\":\"af9e2c58-c3c3-412a-9e73-0f6b813994e9\",\"type\":\"Legend\"},{\"id\":\"94881363-659d-4e67-a976-e012c03bf336\",\"type\":\"GlyphRenderer\"},{\"id\":\"3a0c191c-a99e-478b-8d15-53457a1e6ace\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"9d16c9fe-108d-45aa-a002-1c9551a768ba\",\"type\":\"Title\"},\"tool_events\":{\"id\":\"32977e61-43d6-489f-a45a-8cc95a40c4b7\",\"type\":\"ToolEvents\"},\"toolbar\":{\"id\":\"527bb0f1-0907-48b7-97a3-dc70f93f8e60\",\"type\":\"Toolbar\"},\"x_range\":{\"id\":\"7e22e927-23bc-41c0-9c35-ae86dd53f590\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"1e0eeb42-c4a4-4280-92bd-9ee68960ecea\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"d380f234-3990-4165-b9b7-fd100ca7c957\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"6b660d2f-f07f-434a-ab59-9fa76d814205\",\"type\":\"LinearScale\"}},\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"4750e200-556a-4c98-993f-4492f0bdc5e4\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"callback\":null},\"id\":\"d380f234-3990-4165-b9b7-fd100ca7c957\",\"type\":\"DataRange1d\"},{\"attributes\":{\"label\":{\"value\":\" Retrain AUC\"},\"renderers\":[{\"id\":\"3a0c191c-a99e-478b-8d15-53457a1e6ace\",\"type\":\"GlyphRenderer\"}]},\"id\":\"41a03b31-54fe-48f2-96d4-6eb207250962\",\"type\":\"LegendItem\"},{\"attributes\":{\"items\":[{\"id\":\"8b929341-ab33-4e31-b456-45209b0a92c8\",\"type\":\"LegendItem\"},{\"id\":\"41a03b31-54fe-48f2-96d4-6eb207250962\",\"type\":\"LegendItem\"}],\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"}},\"id\":\"af9e2c58-c3c3-412a-9e73-0f6b813994e9\",\"type\":\"Legend\"},{\"attributes\":{\"line_cap\":\"round\",\"line_color\":{\"value\":\"#a1bae3\"},\"line_width\":{\"value\":6},\"x\":{\"field\":\"Year\"},\"y\":{\"field\":\"retrain_acc\"}},\"id\":\"cb43fae0-2f9a-4d8b-90b7-c3d140121f74\",\"type\":\"Line\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"db69d08a-b02a-433d-a63f-7c9dfe7fa96b\",\"type\":\"BasicTicker\"}},\"id\":\"9abb3969-b48a-4fcc-ad9e-938fae803ed3\",\"type\":\"Grid\"},{\"attributes\":{\"label\":{\"value\":\" Train AUC\"},\"renderers\":[{\"id\":\"94881363-659d-4e67-a976-e012c03bf336\",\"type\":\"GlyphRenderer\"}]},\"id\":\"8b929341-ab33-4e31-b456-45209b0a92c8\",\"type\":\"LegendItem\"},{\"attributes\":{\"callback\":null},\"id\":\"7e22e927-23bc-41c0-9c35-ae86dd53f590\",\"type\":\"DataRange1d\"},{\"attributes\":{\"line_cap\":\"round\",\"line_color\":{\"value\":\"#5975a4\"},\"line_width\":{\"value\":6},\"x\":{\"field\":\"Year\"},\"y\":{\"field\":\"train_acc\"}},\"id\":\"5e73e87e-8055-4e57-aeb4-102d0656101e\",\"type\":\"Line\"},{\"attributes\":{},\"id\":\"32977e61-43d6-489f-a45a-8cc95a40c4b7\",\"type\":\"ToolEvents\"},{\"attributes\":{},\"id\":\"db69d08a-b02a-433d-a63f-7c9dfe7fa96b\",\"type\":\"BasicTicker\"},{\"attributes\":{\"data_source\":{\"id\":\"6ed81d3d-2835-4ca6-b2fc-1a0249c6ae79\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"cb43fae0-2f9a-4d8b-90b7-c3d140121f74\",\"type\":\"Line\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"9ebf72d4-aae1-4bf6-a947-3c78706a6200\",\"type\":\"Line\"},\"selection_glyph\":null},\"id\":\"3a0c191c-a99e-478b-8d15-53457a1e6ace\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"}},\"id\":\"3209c089-f316-4982-abb3-7b7d70cb1010\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"plot\":{\"id\":\"b1b2d702-ede3-4588-9e21-322d323956e5\",\"subtype\":\"Figure\",\"type\":\"Plot\"}},\"id\":\"c3896de3-0531-49f2-ae8c-a54c340052fd\",\"type\":\"ResetTool\"}],\"root_ids\":[\"b1b2d702-ede3-4588-9e21-322d323956e5\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.6\"}};\n",
       "            var render_items = [{\"docid\":\"e7fa339d-3562-48e3-8583-ebdecfbcd60a\",\"elementid\":\"418a5452-0f41-427a-8537-ab1314ad1dce\",\"modelid\":\"b1b2d702-ede3-4588-9e21-322d323956e5\"}];\n",
       "            \n",
       "            Bokeh.embed.embed_items(docs_json, render_items);\n",
       "          };\n",
       "          if (document.readyState != \"loading\") fn();\n",
       "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
       "        })();\n",
       "      },\n",
       "      function(Bokeh) {\n",
       "      }\n",
       "    ];\n",
       "  \n",
       "    function run_inline_js() {\n",
       "      \n",
       "      if ((window.Bokeh !== undefined) || (force === true)) {\n",
       "        for (var i = 0; i < inline_js.length; i++) {\n",
       "          inline_js[i](window.Bokeh);\n",
       "        }if (force === true) {\n",
       "          display_loaded();\n",
       "        }} else if (Date.now() < window._bokeh_timeout) {\n",
       "        setTimeout(run_inline_js, 100);\n",
       "      } else if (!window._bokeh_failed_load) {\n",
       "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "        window._bokeh_failed_load = true;\n",
       "      } else if (force !== true) {\n",
       "        var cell = $(document.getElementById(\"418a5452-0f41-427a-8537-ab1314ad1dce\")).parents('.cell').data().cell;\n",
       "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
       "      }\n",
       "  \n",
       "    }\n",
       "  \n",
       "    if (window._bokeh_is_loading === 0) {\n",
       "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
       "      run_inline_js();\n",
       "    } else {\n",
       "      load_libs(js_urls, function() {\n",
       "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "        run_inline_js();\n",
       "      });\n",
       "    }\n",
       "  }(this));\n",
       "</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [57] used -27812.4570 MiB RAM in 0.26s, total RAM usage 43651.40 MiB\n"
     ]
    }
   ],
   "source": [
    "# Airline Retrain Results\n",
    "p = figure(y_axis_label='Accuracy', plot_width=700, plot_height=350, tools=\"pan,wheel_zoom,box_zoom,reset\")\n",
    "l1 = p.line('Year', 'train_acc', legend=' Train AUC', line_color=\"#5975a4\", source=data_cds, line_width=6, line_cap=\"round\")\n",
    "p.line('Year', 'retrain_acc', legend=' Retrain AUC', line_color=\"#a1bae3\", source=data_cds, line_width=6, line_cap=\"round\")\n",
    "l1_hover = HoverTool(renderers=[l1], tooltips=[( 'Train',  '@{train_acc}{0.4f}' ), ( 'Retrain',  '@{retrain_acc}{0.4f}' )], mode='vline')\n",
    "p.add_tools(l1_hover)\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"350\" style=\"width: 700px; height: 350px;\" version=\"1.1\" width=\"700\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"><defs/><g><g/><g transform=\"scale(1,1) translate(0.5,0.5)\"><rect fill=\"#FFFFFF\" height=\"350\" stroke=\"none\" width=\"700\" x=\"0\" y=\"0\"/><rect fill=\"#ffffff\" height=\"350\" stroke=\"none\" width=\"700\" x=\"0\" y=\"0\"/><rect fill=\"#FFFFFF\" height=\"318.9925\" stroke=\"none\" width=\"609.38875\" x=\"60.61125000000001\" y=\"4\"/><rect fill=\"#ffffff\" height=\"318.9925\" stroke=\"none\" width=\"609.38875\" x=\"60.61125000000001\" y=\"4\"/><g><rect fill=\"none\" height=\"318.9925\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\" width=\"609.38875\" x=\"60.61125000000001\" y=\"4\"/></g><g><g><path d=\" M 192 323 L 192 4\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 365 323 L 365 4\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 538 323 L 538 4\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/></g><g><path d=\" M 61 300 L 670 300\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 252 L 670 252\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 204 L 670 204\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 156 L 670 156\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 108 L 670 108\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 60 L 670 60\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/><path d=\" M 61 12 L 670 12\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\"/></g><g><path fill=\"none\" stroke=\"none\"/><path d=\" M 88.31073863635538 18.499659090909063 L 122.93509943180834 21.57992110994155 L 157.5594602272613 49.90755722339122 L 192.1838210227288 80.62963214066508 L 226.80818181818177 102.2717543800386 L 261.4325426136347 106.4468526856715 L 296.0569034090877 141.44180978745203 L 330.68126420454064 154.44819472117342 L 365.3056249999936 181.46126007785006 L 399.92998579544656 218.17753466642603 L 434.5543465908995 203.88215660540027 L 469.178707386367 189.58189003684333 L 503.80306818182 202.63128683593732 L 538.4274289772729 219.99351960512536 L 573.0517897727259 243.6935567083258 L 607.6761505681789 272.69418824346644 L 642.3005113636318 308.4928409090912\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#5975a4\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/></g><g><path fill=\"none\" stroke=\"none\"/><path d=\" M 88.31073863635538 NaN\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/><path fill=\"none\" stroke=\"none\"/><path d=\" M 157.5594602272613 NaN\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/><path fill=\"none\" stroke=\"none\"/><path d=\" M 226.80818181818177 NaN\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/><path fill=\"none\" stroke=\"none\"/><path d=\" M 296.0569034090877 NaN\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/><path fill=\"none\" stroke=\"none\"/><path d=\" M 365.3056249999936 NaN\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/><path fill=\"none\" stroke=\"none\"/><path d=\" M 434.5543465908995 36.24679122154407 L 469.178707386367 40.33262113837645 L 503.80306818182 96.41709672599472 L 538.4274289772729 108.75296459884657 L 573.0517897727259 138.3982147763188 L 607.6761505681789 167.26623972006973 L 642.3005113636318 161.20409579074317\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/></g></g><g><g/><g><path d=\" M 527.328125 14 L 660 14 L 660 77 L 527.328125 77 L 527.328125 14 Z\" fill=\"#ffffff\" fill-opacity=\"0.95\" paint-order=\"fill stroke markers\" stroke=\"#e5e5e5\" stroke-miterlimit=\"10\" stroke-opacity=\"0.5\"/><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"10pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"start\" text-decoration=\"normal\" x=\"562.328125\" y=\"34\"> Train AUC</text><g><path d=\" M 537.328125 34 L 557.328125 34\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#5975a4\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/></g><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"10pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"start\" text-decoration=\"normal\" x=\"562.328125\" y=\"57\"> Retrain AUC</text><g><path d=\" M 537.328125 57 L 557.328125 57\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"#a1bae3\" stroke-linecap=\"round\" stroke-miterlimit=\"10\" stroke-width=\"6\"/></g></g></g><g><g><path d=\" M 61 323 L 670 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 192 329 L 192 321\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 365 329 L 365 321\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 538 329 L 538 321\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 158 327 L 158 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 123 327 L 123 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 88 327 L 88 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 192 327 L 192 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 227 327 L 227 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 261 327 L 261 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 296 327 L 296 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 331 327 L 331 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 365 327 L 365 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 400 327 L 400 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 435 327 L 435 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 469 327 L 469 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 504 327 L 504 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 538 327 L 538 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 573 327 L 573 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 608 327 L 608 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 642 327 L 642 323\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><text dominant-baseline=\"hanging\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"middle\" text-decoration=\"normal\" x=\"192\" y=\"334\">1995</text><text dominant-baseline=\"hanging\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"middle\" text-decoration=\"normal\" x=\"365\" y=\"334\">2000</text><text dominant-baseline=\"hanging\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"middle\" text-decoration=\"normal\" x=\"538\" y=\"334\">2005</text><text dominant-baseline=\"hanging\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"10pt\" font-style=\"italic\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"middle\" text-decoration=\"normal\" x=\"365.3056250000009\" y=\"354.00000000000006\"/></g><g><path d=\" M 61 323 L 61 4\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 300 L 63 300\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 252 L 63 252\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 204 L 63 204\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 156 L 63 156\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 108 L 63 108\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 60 L 63 60\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 55 12 L 63 12\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 310 L 61 310\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 320 L 61 320\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 300 L 61 300\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 291 L 61 291\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 281 L 61 281\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 272 L 61 272\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 262 L 61 262\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 252 L 61 252\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 243 L 61 243\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 233 L 61 233\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 224 L 61 224\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 214 L 61 214\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 204 L 61 204\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 195 L 61 195\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 185 L 61 185\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 175 L 61 175\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 166 L 61 166\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 156 L 61 156\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 147 L 61 147\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 137 L 61 137\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 127 L 61 127\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 118 L 61 118\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 108 L 61 108\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 99 L 61 99\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 89 L 61 89\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 79 L 61 79\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 70 L 61 70\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 60 L 61 60\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 51 L 61 51\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 41 L 61 41\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 31 L 61 31\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 22 L 61 22\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><path d=\" M 57 12 L 61 12\" fill=\"none\" paint-order=\"fill stroke markers\" stroke=\"black\" stroke-miterlimit=\"10\"/><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"300\">0.64</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"252\">0.66</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"204\">0.68</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"156\">0.7</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"108\">0.72</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"60\">0.74</text><text dominant-baseline=\"central\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"8pt\" font-style=\"normal\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"end\" text-decoration=\"normal\" x=\"50\" y=\"12\">0.76</text><g transform=\"translate(17.730000000001745,163.49625000000015) rotate(-90,0,0)\"><text dominant-baseline=\"alphabetic\" fill=\"#444444\" font-family=\"helvetica\" font-size=\"10pt\" font-style=\"italic\" font-weight=\"normal\" stroke=\"none\" text-anchor=\"middle\" text-decoration=\"normal\" x=\"0\" y=\"0\">Accuracy</text><g transform=\"rotate(90,0,0) translate(-17.730000000001745,-163.49625000000015)\"/></g></g></g></g></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In [58] used 0.1133 MiB RAM in 2.66s, total RAM usage 43651.51 MiB\n"
     ]
    }
   ],
   "source": [
    "# SVG: Airline Retrain Results\n",
    "# Save the plot and display it so that is is visible on Github\n",
    "p.output_backend = \"svg\"\n",
    "export_svgs(p, filename=\"airline_retrain.svg\")\n",
    "display(SVG('airline_retrain.svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As it can be seen, the performance is better after retraining. We have found concept drift in this dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python Strata",
   "language": "python",
   "name": "strata"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}