model-selection/model-comparison/pycaret-comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Type Selection with PyCaret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook explains how to use `PyCaret` to build multiple models and compare them to pick the best model for your data.\n",
    "\n",
    "This notebook will build and evaluate a model to predict arrival delay for flights in and out of NYC in 2013.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial uses:\n",
    "* [pandas](https://pandas.pydata.org/docs/)\n",
    "* [statsmodels](https://www.statsmodels.org/stable/index.html)\n",
    "    * [statsmodels.api](https://www.statsmodels.org/stable/api.html#statsmodels-api)\n",
    "* [pycaret](https://pycaret.readthedocs.io/en/latest/index.html)\n",
    "    * [pycaret.regression](https://pycaret.readthedocs.io/en/latest/api/regression.html)\n",
    "    * [pycaret.utils](https://pycaret.readthedocs.io/en/latest/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pycaret.regression as pycr\n",
    "import pycaret.utils as pycu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is from `rdatasets` imported using the Python package `statsmodels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 336776 entries, 0 to 336775\n",
      "Data columns (total 19 columns):\n",
      " #   Column          Non-Null Count   Dtype  \n",
      "---  ------          --------------   -----  \n",
      " 0   year            336776 non-null  int64  \n",
      " 1   month           336776 non-null  int64  \n",
      " 2   day             336776 non-null  int64  \n",
      " 3   dep_time        328521 non-null  float64\n",
      " 4   sched_dep_time  336776 non-null  int64  \n",
      " 5   dep_delay       328521 non-null  float64\n",
      " 6   arr_time        328063 non-null  float64\n",
      " 7   sched_arr_time  336776 non-null  int64  \n",
      " 8   arr_delay       327346 non-null  float64\n",
      " 9   carrier         336776 non-null  object \n",
      " 10  flight          336776 non-null  int64  \n",
      " 11  tailnum         334264 non-null  object \n",
      " 12  origin          336776 non-null  object \n",
      " 13  dest            336776 non-null  object \n",
      " 14  air_time        327346 non-null  float64\n",
      " 15  distance        336776 non-null  int64  \n",
      " 16  hour            336776 non-null  int64  \n",
      " 17  minute          336776 non-null  int64  \n",
      " 18  time_hour       336776 non-null  object \n",
      "dtypes: float64(5), int64(9), object(5)\n",
      "memory usage: 48.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df = sm.datasets.get_rdataset('flights', 'nycflights13').data\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handle null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "year                 0\n",
       "month                0\n",
       "day                  0\n",
       "dep_time          8255\n",
       "sched_dep_time       0\n",
       "dep_delay         8255\n",
       "arr_time          8713\n",
       "sched_arr_time       0\n",
       "arr_delay         9430\n",
       "carrier              0\n",
       "flight               0\n",
       "tailnum           2512\n",
       "origin               0\n",
       "dest                 0\n",
       "air_time          9430\n",
       "distance             0\n",
       "hour                 0\n",
       "minute               0\n",
       "time_hour            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As this model will predict arrival delay, the `Null` values are caused by flights did were cancelled or diverted. These can be excluded from this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the times from floats or ints to hour and minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['arr_hour'] = df.arr_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['arr_minute'] = df.arr_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df['sched_arr_hour'] = df.sched_arr_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['sched_arr_minute'] = df.sched_arr_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df['sched_dep_hour'] = df.sched_dep_time.apply(lambda x: int(np.floor(x/100)))\n",
    "df['sched_dep_minute'] = df.sched_dep_time.apply(lambda x: int(x - np.floor(x/100)*100))\n",
    "df.rename(columns={'hour': 'dep_hour',\n",
    "                   'minute': 'dep_minute'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As `PyCaret` can use large amounts of memory, we will randomly select **100,000** rows for this comparison, reserving the remaining rows as a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain = df.sample(n=100000, random_state=1066)\n",
    "dftest = df.drop(dftrain.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the `PyCaret` environment.  **session_id** is equivalent to **random_state** in `scikit-learn` and allows the experiment to be repeated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_6460c_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Description</th>        <th class=\"col_heading level0 col1\" >Value</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_6460c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_6460c_row0_col0\" class=\"data row0 col0\" >session_id</td>\n",
       "                        <td id=\"T_6460c_row0_col1\" class=\"data row0 col1\" >1066</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_6460c_row1_col0\" class=\"data row1 col0\" >Target</td>\n",
       "                        <td id=\"T_6460c_row1_col1\" class=\"data row1 col1\" >arr_delay</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_6460c_row2_col0\" class=\"data row2 col0\" >Original Data</td>\n",
       "                        <td id=\"T_6460c_row2_col1\" class=\"data row2 col1\" >(100000, 25)</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_6460c_row3_col0\" class=\"data row3 col0\" >Missing Values</td>\n",
       "                        <td id=\"T_6460c_row3_col1\" class=\"data row3 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_6460c_row4_col0\" class=\"data row4 col0\" >Numeric Features</td>\n",
       "                        <td id=\"T_6460c_row4_col1\" class=\"data row4 col1\" >9</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_6460c_row5_col0\" class=\"data row5 col0\" >Categorical Features</td>\n",
       "                        <td id=\"T_6460c_row5_col1\" class=\"data row5 col1\" >6</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "                        <td id=\"T_6460c_row6_col0\" class=\"data row6 col0\" >Ordinal Features</td>\n",
       "                        <td id=\"T_6460c_row6_col1\" class=\"data row6 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "                        <td id=\"T_6460c_row7_col0\" class=\"data row7 col0\" >High Cardinality Features</td>\n",
       "                        <td id=\"T_6460c_row7_col1\" class=\"data row7 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "                        <td id=\"T_6460c_row8_col0\" class=\"data row8 col0\" >High Cardinality Method</td>\n",
       "                        <td id=\"T_6460c_row8_col1\" class=\"data row8 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "                        <td id=\"T_6460c_row9_col0\" class=\"data row9 col0\" >Transformed Train Set</td>\n",
       "                        <td id=\"T_6460c_row9_col1\" class=\"data row9 col1\" >(69999, 162)</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "                        <td id=\"T_6460c_row10_col0\" class=\"data row10 col0\" >Transformed Test Set</td>\n",
       "                        <td id=\"T_6460c_row10_col1\" class=\"data row10 col1\" >(30001, 162)</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "                        <td id=\"T_6460c_row11_col0\" class=\"data row11 col0\" >Shuffle Train-Test</td>\n",
       "                        <td id=\"T_6460c_row11_col1\" class=\"data row11 col1\" >True</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "                        <td id=\"T_6460c_row12_col0\" class=\"data row12 col0\" >Stratify Train-Test</td>\n",
       "                        <td id=\"T_6460c_row12_col1\" class=\"data row12 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
       "                        <td id=\"T_6460c_row13_col0\" class=\"data row13 col0\" >Fold Generator</td>\n",
       "                        <td id=\"T_6460c_row13_col1\" class=\"data row13 col1\" >KFold</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
       "                        <td id=\"T_6460c_row14_col0\" class=\"data row14 col0\" >Fold Number</td>\n",
       "                        <td id=\"T_6460c_row14_col1\" class=\"data row14 col1\" >10</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
       "                        <td id=\"T_6460c_row15_col0\" class=\"data row15 col0\" >CPU Jobs</td>\n",
       "                        <td id=\"T_6460c_row15_col1\" class=\"data row15 col1\" >-1</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
       "                        <td id=\"T_6460c_row16_col0\" class=\"data row16 col0\" >Use GPU</td>\n",
       "                        <td id=\"T_6460c_row16_col1\" class=\"data row16 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
       "                        <td id=\"T_6460c_row17_col0\" class=\"data row17 col0\" >Log Experiment</td>\n",
       "                        <td id=\"T_6460c_row17_col1\" class=\"data row17 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
       "                        <td id=\"T_6460c_row18_col0\" class=\"data row18 col0\" >Experiment Name</td>\n",
       "                        <td id=\"T_6460c_row18_col1\" class=\"data row18 col1\" >reg-default-name</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
       "                        <td id=\"T_6460c_row19_col0\" class=\"data row19 col0\" >USI</td>\n",
       "                        <td id=\"T_6460c_row19_col1\" class=\"data row19 col1\" >c72b</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
       "                        <td id=\"T_6460c_row20_col0\" class=\"data row20 col0\" >Imputation Type</td>\n",
       "                        <td id=\"T_6460c_row20_col1\" class=\"data row20 col1\" >simple</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
       "                        <td id=\"T_6460c_row21_col0\" class=\"data row21 col0\" >Iterative Imputation Iteration</td>\n",
       "                        <td id=\"T_6460c_row21_col1\" class=\"data row21 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
       "                        <td id=\"T_6460c_row22_col0\" class=\"data row22 col0\" >Numeric Imputer</td>\n",
       "                        <td id=\"T_6460c_row22_col1\" class=\"data row22 col1\" >mean</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
       "                        <td id=\"T_6460c_row23_col0\" class=\"data row23 col0\" >Iterative Imputation Numeric Model</td>\n",
       "                        <td id=\"T_6460c_row23_col1\" class=\"data row23 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
       "                        <td id=\"T_6460c_row24_col0\" class=\"data row24 col0\" >Categorical Imputer</td>\n",
       "                        <td id=\"T_6460c_row24_col1\" class=\"data row24 col1\" >constant</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
       "                        <td id=\"T_6460c_row25_col0\" class=\"data row25 col0\" >Iterative Imputation Categorical Model</td>\n",
       "                        <td id=\"T_6460c_row25_col1\" class=\"data row25 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row26\" class=\"row_heading level0 row26\" >26</th>\n",
       "                        <td id=\"T_6460c_row26_col0\" class=\"data row26 col0\" >Unknown Categoricals Handling</td>\n",
       "                        <td id=\"T_6460c_row26_col1\" class=\"data row26 col1\" >least_frequent</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row27\" class=\"row_heading level0 row27\" >27</th>\n",
       "                        <td id=\"T_6460c_row27_col0\" class=\"data row27 col0\" >Normalize</td>\n",
       "                        <td id=\"T_6460c_row27_col1\" class=\"data row27 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row28\" class=\"row_heading level0 row28\" >28</th>\n",
       "                        <td id=\"T_6460c_row28_col0\" class=\"data row28 col0\" >Normalize Method</td>\n",
       "                        <td id=\"T_6460c_row28_col1\" class=\"data row28 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row29\" class=\"row_heading level0 row29\" >29</th>\n",
       "                        <td id=\"T_6460c_row29_col0\" class=\"data row29 col0\" >Transformation</td>\n",
       "                        <td id=\"T_6460c_row29_col1\" class=\"data row29 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row30\" class=\"row_heading level0 row30\" >30</th>\n",
       "                        <td id=\"T_6460c_row30_col0\" class=\"data row30 col0\" >Transformation Method</td>\n",
       "                        <td id=\"T_6460c_row30_col1\" class=\"data row30 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row31\" class=\"row_heading level0 row31\" >31</th>\n",
       "                        <td id=\"T_6460c_row31_col0\" class=\"data row31 col0\" >PCA</td>\n",
       "                        <td id=\"T_6460c_row31_col1\" class=\"data row31 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row32\" class=\"row_heading level0 row32\" >32</th>\n",
       "                        <td id=\"T_6460c_row32_col0\" class=\"data row32 col0\" >PCA Method</td>\n",
       "                        <td id=\"T_6460c_row32_col1\" class=\"data row32 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row33\" class=\"row_heading level0 row33\" >33</th>\n",
       "                        <td id=\"T_6460c_row33_col0\" class=\"data row33 col0\" >PCA Components</td>\n",
       "                        <td id=\"T_6460c_row33_col1\" class=\"data row33 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row34\" class=\"row_heading level0 row34\" >34</th>\n",
       "                        <td id=\"T_6460c_row34_col0\" class=\"data row34 col0\" >Ignore Low Variance</td>\n",
       "                        <td id=\"T_6460c_row34_col1\" class=\"data row34 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row35\" class=\"row_heading level0 row35\" >35</th>\n",
       "                        <td id=\"T_6460c_row35_col0\" class=\"data row35 col0\" >Combine Rare Levels</td>\n",
       "                        <td id=\"T_6460c_row35_col1\" class=\"data row35 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row36\" class=\"row_heading level0 row36\" >36</th>\n",
       "                        <td id=\"T_6460c_row36_col0\" class=\"data row36 col0\" >Rare Level Threshold</td>\n",
       "                        <td id=\"T_6460c_row36_col1\" class=\"data row36 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row37\" class=\"row_heading level0 row37\" >37</th>\n",
       "                        <td id=\"T_6460c_row37_col0\" class=\"data row37 col0\" >Numeric Binning</td>\n",
       "                        <td id=\"T_6460c_row37_col1\" class=\"data row37 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row38\" class=\"row_heading level0 row38\" >38</th>\n",
       "                        <td id=\"T_6460c_row38_col0\" class=\"data row38 col0\" >Remove Outliers</td>\n",
       "                        <td id=\"T_6460c_row38_col1\" class=\"data row38 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row39\" class=\"row_heading level0 row39\" >39</th>\n",
       "                        <td id=\"T_6460c_row39_col0\" class=\"data row39 col0\" >Outliers Threshold</td>\n",
       "                        <td id=\"T_6460c_row39_col1\" class=\"data row39 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row40\" class=\"row_heading level0 row40\" >40</th>\n",
       "                        <td id=\"T_6460c_row40_col0\" class=\"data row40 col0\" >Remove Multicollinearity</td>\n",
       "                        <td id=\"T_6460c_row40_col1\" class=\"data row40 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row41\" class=\"row_heading level0 row41\" >41</th>\n",
       "                        <td id=\"T_6460c_row41_col0\" class=\"data row41 col0\" >Multicollinearity Threshold</td>\n",
       "                        <td id=\"T_6460c_row41_col1\" class=\"data row41 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row42\" class=\"row_heading level0 row42\" >42</th>\n",
       "                        <td id=\"T_6460c_row42_col0\" class=\"data row42 col0\" >Clustering</td>\n",
       "                        <td id=\"T_6460c_row42_col1\" class=\"data row42 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row43\" class=\"row_heading level0 row43\" >43</th>\n",
       "                        <td id=\"T_6460c_row43_col0\" class=\"data row43 col0\" >Clustering Iteration</td>\n",
       "                        <td id=\"T_6460c_row43_col1\" class=\"data row43 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row44\" class=\"row_heading level0 row44\" >44</th>\n",
       "                        <td id=\"T_6460c_row44_col0\" class=\"data row44 col0\" >Polynomial Features</td>\n",
       "                        <td id=\"T_6460c_row44_col1\" class=\"data row44 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row45\" class=\"row_heading level0 row45\" >45</th>\n",
       "                        <td id=\"T_6460c_row45_col0\" class=\"data row45 col0\" >Polynomial Degree</td>\n",
       "                        <td id=\"T_6460c_row45_col1\" class=\"data row45 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row46\" class=\"row_heading level0 row46\" >46</th>\n",
       "                        <td id=\"T_6460c_row46_col0\" class=\"data row46 col0\" >Trignometry Features</td>\n",
       "                        <td id=\"T_6460c_row46_col1\" class=\"data row46 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row47\" class=\"row_heading level0 row47\" >47</th>\n",
       "                        <td id=\"T_6460c_row47_col0\" class=\"data row47 col0\" >Polynomial Threshold</td>\n",
       "                        <td id=\"T_6460c_row47_col1\" class=\"data row47 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row48\" class=\"row_heading level0 row48\" >48</th>\n",
       "                        <td id=\"T_6460c_row48_col0\" class=\"data row48 col0\" >Group Features</td>\n",
       "                        <td id=\"T_6460c_row48_col1\" class=\"data row48 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row49\" class=\"row_heading level0 row49\" >49</th>\n",
       "                        <td id=\"T_6460c_row49_col0\" class=\"data row49 col0\" >Feature Selection</td>\n",
       "                        <td id=\"T_6460c_row49_col1\" class=\"data row49 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row50\" class=\"row_heading level0 row50\" >50</th>\n",
       "                        <td id=\"T_6460c_row50_col0\" class=\"data row50 col0\" >Feature Selection Method</td>\n",
       "                        <td id=\"T_6460c_row50_col1\" class=\"data row50 col1\" >classic</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row51\" class=\"row_heading level0 row51\" >51</th>\n",
       "                        <td id=\"T_6460c_row51_col0\" class=\"data row51 col0\" >Features Selection Threshold</td>\n",
       "                        <td id=\"T_6460c_row51_col1\" class=\"data row51 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row52\" class=\"row_heading level0 row52\" >52</th>\n",
       "                        <td id=\"T_6460c_row52_col0\" class=\"data row52 col0\" >Feature Interaction</td>\n",
       "                        <td id=\"T_6460c_row52_col1\" class=\"data row52 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row53\" class=\"row_heading level0 row53\" >53</th>\n",
       "                        <td id=\"T_6460c_row53_col0\" class=\"data row53 col0\" >Feature Ratio</td>\n",
       "                        <td id=\"T_6460c_row53_col1\" class=\"data row53 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row54\" class=\"row_heading level0 row54\" >54</th>\n",
       "                        <td id=\"T_6460c_row54_col0\" class=\"data row54 col0\" >Interaction Threshold</td>\n",
       "                        <td id=\"T_6460c_row54_col1\" class=\"data row54 col1\" >None</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row55\" class=\"row_heading level0 row55\" >55</th>\n",
       "                        <td id=\"T_6460c_row55_col0\" class=\"data row55 col0\" >Transform Target</td>\n",
       "                        <td id=\"T_6460c_row55_col1\" class=\"data row55 col1\" >False</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_6460c_level0_row56\" class=\"row_heading level0 row56\" >56</th>\n",
       "                        <td id=\"T_6460c_row56_col0\" class=\"data row56 col0\" >Transform Target Method</td>\n",
       "                        <td id=\"T_6460c_row56_col1\" class=\"data row56 col1\" >box-cox</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14a8c9490>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pycaret_experiment = pycr.setup(data=dftrain, target=\"arr_delay\", session_id=1066,\n",
    "                                ignore_features=['flight', 'tailnum', 'time_hour', 'year', 'dep_time', 'sched_dep_time', 'arr_time', 'sched_arr_time', 'dep_delay'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `compare_models` will train about 20 models and show their **MAE**, **MSE**, **RMSE**, **R^2**, **RMSLE**, and **MAPE**.  It will also highlight the best peforming model on each of those metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_1460d_ th {\n",
       "          text-align: left;\n",
       "    }#T_1460d_row0_col0,#T_1460d_row0_col1,#T_1460d_row0_col5,#T_1460d_row0_col6,#T_1460d_row1_col0,#T_1460d_row1_col2,#T_1460d_row1_col3,#T_1460d_row1_col4,#T_1460d_row2_col0,#T_1460d_row2_col1,#T_1460d_row2_col2,#T_1460d_row2_col3,#T_1460d_row2_col4,#T_1460d_row2_col5,#T_1460d_row2_col6,#T_1460d_row3_col0,#T_1460d_row3_col1,#T_1460d_row3_col2,#T_1460d_row3_col3,#T_1460d_row3_col4,#T_1460d_row3_col5,#T_1460d_row3_col6,#T_1460d_row4_col0,#T_1460d_row4_col1,#T_1460d_row4_col2,#T_1460d_row4_col3,#T_1460d_row4_col4,#T_1460d_row4_col5,#T_1460d_row4_col6,#T_1460d_row5_col0,#T_1460d_row5_col1,#T_1460d_row5_col2,#T_1460d_row5_col3,#T_1460d_row5_col4,#T_1460d_row5_col5,#T_1460d_row5_col6,#T_1460d_row6_col0,#T_1460d_row6_col1,#T_1460d_row6_col2,#T_1460d_row6_col3,#T_1460d_row6_col4,#T_1460d_row6_col5,#T_1460d_row6_col6,#T_1460d_row7_col0,#T_1460d_row7_col1,#T_1460d_row7_col2,#T_1460d_row7_col3,#T_1460d_row7_col4,#T_1460d_row7_col5,#T_1460d_row7_col6,#T_1460d_row8_col0,#T_1460d_row8_col1,#T_1460d_row8_col2,#T_1460d_row8_col3,#T_1460d_row8_col4,#T_1460d_row8_col5,#T_1460d_row8_col6,#T_1460d_row9_col0,#T_1460d_row9_col1,#T_1460d_row9_col2,#T_1460d_row9_col3,#T_1460d_row9_col4,#T_1460d_row9_col5,#T_1460d_row9_col6,#T_1460d_row10_col0,#T_1460d_row10_col1,#T_1460d_row10_col2,#T_1460d_row10_col3,#T_1460d_row10_col4,#T_1460d_row10_col5,#T_1460d_row10_col6,#T_1460d_row11_col0,#T_1460d_row11_col1,#T_1460d_row11_col2,#T_1460d_row11_col3,#T_1460d_row11_col4,#T_1460d_row11_col5,#T_1460d_row11_col6,#T_1460d_row12_col0,#T_1460d_row12_col1,#T_1460d_row12_col2,#T_1460d_row12_col3,#T_1460d_row12_col4,#T_1460d_row12_col5,#T_1460d_row12_col6,#T_1460d_row13_col0,#T_1460d_row13_col1,#T_1460d_row13_col2,#T_1460d_row13_col3,#T_1460d_row13_col4,#T_1460d_row13_col5,#T_1460d_row13_col6,#T_1460d_row14_col0,#T_1460d_row14_col1,#T_1460d_row14_col2,#T_1460d_row14_col3,#T_1460d_row14_col4,#T_1460d_row14_col5,#T_1460d_row14_col6,#T_1460d_row15_col0,#T_1460d_row15_col1,#T_1460d_row15_col2,#T_1460d_row15_col3,#T_1460d_row15_col4,#T_1460d_row15_col5,#T_1460d_row15_col6,#T_1460d_row16_col0,#T_1460d_row16_col1,#T_1460d_row16_col2,#T_1460d_row16_col3,#T_1460d_row16_col4,#T_1460d_row16_col5,#T_1460d_row16_col6,#T_1460d_row17_col0,#T_1460d_row17_col1,#T_1460d_row17_col2,#T_1460d_row17_col3,#T_1460d_row17_col4,#T_1460d_row17_col5,#T_1460d_row17_col6{\n",
       "            text-align:  left;\n",
       "            text-align:  left;\n",
       "        }#T_1460d_row0_col2,#T_1460d_row0_col3,#T_1460d_row0_col4,#T_1460d_row1_col1,#T_1460d_row1_col5,#T_1460d_row1_col6{\n",
       "            text-align:  left;\n",
       "            text-align:  left;\n",
       "            background-color:  yellow;\n",
       "        }#T_1460d_row0_col7,#T_1460d_row1_col7,#T_1460d_row2_col7,#T_1460d_row3_col7,#T_1460d_row4_col7,#T_1460d_row5_col7,#T_1460d_row6_col7,#T_1460d_row7_col7,#T_1460d_row9_col7,#T_1460d_row10_col7,#T_1460d_row11_col7,#T_1460d_row12_col7,#T_1460d_row13_col7,#T_1460d_row14_col7,#T_1460d_row15_col7,#T_1460d_row16_col7,#T_1460d_row17_col7{\n",
       "            text-align:  left;\n",
       "            text-align:  left;\n",
       "            background-color:  lightgrey;\n",
       "        }#T_1460d_row8_col7{\n",
       "            text-align:  left;\n",
       "            text-align:  left;\n",
       "            background-color:  yellow;\n",
       "            background-color:  lightgrey;\n",
       "        }</style><table id=\"T_1460d_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Model</th>        <th class=\"col_heading level0 col1\" >MAE</th>        <th class=\"col_heading level0 col2\" >MSE</th>        <th class=\"col_heading level0 col3\" >RMSE</th>        <th class=\"col_heading level0 col4\" >R2</th>        <th class=\"col_heading level0 col5\" >RMSLE</th>        <th class=\"col_heading level0 col6\" >MAPE</th>        <th class=\"col_heading level0 col7\" >TT (Sec)</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_1460d_level0_row0\" class=\"row_heading level0 row0\" >catboost</th>\n",
       "                        <td id=\"T_1460d_row0_col0\" class=\"data row0 col0\" >CatBoost Regressor</td>\n",
       "                        <td id=\"T_1460d_row0_col1\" class=\"data row0 col1\" >1.9788</td>\n",
       "                        <td id=\"T_1460d_row0_col2\" class=\"data row0 col2\" >36.7907</td>\n",
       "                        <td id=\"T_1460d_row0_col3\" class=\"data row0 col3\" >5.5109</td>\n",
       "                        <td id=\"T_1460d_row0_col4\" class=\"data row0 col4\" >0.9818</td>\n",
       "                        <td id=\"T_1460d_row0_col5\" class=\"data row0 col5\" >0.2145</td>\n",
       "                        <td id=\"T_1460d_row0_col6\" class=\"data row0 col6\" >0.1533</td>\n",
       "                        <td id=\"T_1460d_row0_col7\" class=\"data row0 col7\" >4.0700</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row1\" class=\"row_heading level0 row1\" >rf</th>\n",
       "                        <td id=\"T_1460d_row1_col0\" class=\"data row1 col0\" >Random Forest Regressor</td>\n",
       "                        <td id=\"T_1460d_row1_col1\" class=\"data row1 col1\" >1.0965</td>\n",
       "                        <td id=\"T_1460d_row1_col2\" class=\"data row1 col2\" >38.4911</td>\n",
       "                        <td id=\"T_1460d_row1_col3\" class=\"data row1 col3\" >5.6290</td>\n",
       "                        <td id=\"T_1460d_row1_col4\" class=\"data row1 col4\" >0.9809</td>\n",
       "                        <td id=\"T_1460d_row1_col5\" class=\"data row1 col5\" >0.1149</td>\n",
       "                        <td id=\"T_1460d_row1_col6\" class=\"data row1 col6\" >0.0655</td>\n",
       "                        <td id=\"T_1460d_row1_col7\" class=\"data row1 col7\" >25.0300</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row2\" class=\"row_heading level0 row2\" >et</th>\n",
       "                        <td id=\"T_1460d_row2_col0\" class=\"data row2 col0\" >Extra Trees Regressor</td>\n",
       "                        <td id=\"T_1460d_row2_col1\" class=\"data row2 col1\" >1.3357</td>\n",
       "                        <td id=\"T_1460d_row2_col2\" class=\"data row2 col2\" >42.8113</td>\n",
       "                        <td id=\"T_1460d_row2_col3\" class=\"data row2 col3\" >6.1431</td>\n",
       "                        <td id=\"T_1460d_row2_col4\" class=\"data row2 col4\" >0.9788</td>\n",
       "                        <td id=\"T_1460d_row2_col5\" class=\"data row2 col5\" >0.1510</td>\n",
       "                        <td id=\"T_1460d_row2_col6\" class=\"data row2 col6\" >0.0817</td>\n",
       "                        <td id=\"T_1460d_row2_col7\" class=\"data row2 col7\" >35.5990</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row3\" class=\"row_heading level0 row3\" >xgboost</th>\n",
       "                        <td id=\"T_1460d_row3_col0\" class=\"data row3 col0\" >Extreme Gradient Boosting</td>\n",
       "                        <td id=\"T_1460d_row3_col1\" class=\"data row3 col1\" >3.2131</td>\n",
       "                        <td id=\"T_1460d_row3_col2\" class=\"data row3 col2\" >40.0363</td>\n",
       "                        <td id=\"T_1460d_row3_col3\" class=\"data row3 col3\" >6.1862</td>\n",
       "                        <td id=\"T_1460d_row3_col4\" class=\"data row3 col4\" >0.9800</td>\n",
       "                        <td id=\"T_1460d_row3_col5\" class=\"data row3 col5\" >0.3423</td>\n",
       "                        <td id=\"T_1460d_row3_col6\" class=\"data row3 col6\" >0.2805</td>\n",
       "                        <td id=\"T_1460d_row3_col7\" class=\"data row3 col7\" >17.1310</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row4\" class=\"row_heading level0 row4\" >dt</th>\n",
       "                        <td id=\"T_1460d_row4_col0\" class=\"data row4 col0\" >Decision Tree Regressor</td>\n",
       "                        <td id=\"T_1460d_row4_col1\" class=\"data row4 col1\" >1.7987</td>\n",
       "                        <td id=\"T_1460d_row4_col2\" class=\"data row4 col2\" >54.8896</td>\n",
       "                        <td id=\"T_1460d_row4_col3\" class=\"data row4 col3\" >6.8635</td>\n",
       "                        <td id=\"T_1460d_row4_col4\" class=\"data row4 col4\" >0.9728</td>\n",
       "                        <td id=\"T_1460d_row4_col5\" class=\"data row4 col5\" >0.2031</td>\n",
       "                        <td id=\"T_1460d_row4_col6\" class=\"data row4 col6\" >0.1283</td>\n",
       "                        <td id=\"T_1460d_row4_col7\" class=\"data row4 col7\" >0.3770</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row5\" class=\"row_heading level0 row5\" >lightgbm</th>\n",
       "                        <td id=\"T_1460d_row5_col0\" class=\"data row5 col0\" >Light Gradient Boosting Machine</td>\n",
       "                        <td id=\"T_1460d_row5_col1\" class=\"data row5 col1\" >6.2720</td>\n",
       "                        <td id=\"T_1460d_row5_col2\" class=\"data row5 col2\" >165.4157</td>\n",
       "                        <td id=\"T_1460d_row5_col3\" class=\"data row5 col3\" >12.5736</td>\n",
       "                        <td id=\"T_1460d_row5_col4\" class=\"data row5 col4\" >0.9176</td>\n",
       "                        <td id=\"T_1460d_row5_col5\" class=\"data row5 col5\" >0.5248</td>\n",
       "                        <td id=\"T_1460d_row5_col6\" class=\"data row5 col6\" >0.4355</td>\n",
       "                        <td id=\"T_1460d_row5_col7\" class=\"data row5 col7\" >1.2810</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row6\" class=\"row_heading level0 row6\" >gbr</th>\n",
       "                        <td id=\"T_1460d_row6_col0\" class=\"data row6 col0\" >Gradient Boosting Regressor</td>\n",
       "                        <td id=\"T_1460d_row6_col1\" class=\"data row6 col1\" >14.3364</td>\n",
       "                        <td id=\"T_1460d_row6_col2\" class=\"data row6 col2\" >561.7050</td>\n",
       "                        <td id=\"T_1460d_row6_col3\" class=\"data row6 col3\" >23.6655</td>\n",
       "                        <td id=\"T_1460d_row6_col4\" class=\"data row6 col4\" >0.7183</td>\n",
       "                        <td id=\"T_1460d_row6_col5\" class=\"data row6 col5\" >1.1623</td>\n",
       "                        <td id=\"T_1460d_row6_col6\" class=\"data row6 col6\" >0.8105</td>\n",
       "                        <td id=\"T_1460d_row6_col7\" class=\"data row6 col7\" >5.7960</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row7\" class=\"row_heading level0 row7\" >br</th>\n",
       "                        <td id=\"T_1460d_row7_col0\" class=\"data row7 col0\" >Bayesian Ridge</td>\n",
       "                        <td id=\"T_1460d_row7_col1\" class=\"data row7 col1\" >25.1405</td>\n",
       "                        <td id=\"T_1460d_row7_col2\" class=\"data row7 col2\" >1707.8873</td>\n",
       "                        <td id=\"T_1460d_row7_col3\" class=\"data row7 col3\" >41.3148</td>\n",
       "                        <td id=\"T_1460d_row7_col4\" class=\"data row7 col4\" >0.1422</td>\n",
       "                        <td id=\"T_1460d_row7_col5\" class=\"data row7 col5\" >1.3191</td>\n",
       "                        <td id=\"T_1460d_row7_col6\" class=\"data row7 col6\" >1.8122</td>\n",
       "                        <td id=\"T_1460d_row7_col7\" class=\"data row7 col7\" >0.9350</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row8\" class=\"row_heading level0 row8\" >ridge</th>\n",
       "                        <td id=\"T_1460d_row8_col0\" class=\"data row8 col0\" >Ridge Regression</td>\n",
       "                        <td id=\"T_1460d_row8_col1\" class=\"data row8 col1\" >25.1445</td>\n",
       "                        <td id=\"T_1460d_row8_col2\" class=\"data row8 col2\" >1708.1058</td>\n",
       "                        <td id=\"T_1460d_row8_col3\" class=\"data row8 col3\" >41.3174</td>\n",
       "                        <td id=\"T_1460d_row8_col4\" class=\"data row8 col4\" >0.1421</td>\n",
       "                        <td id=\"T_1460d_row8_col5\" class=\"data row8 col5\" >1.3159</td>\n",
       "                        <td id=\"T_1460d_row8_col6\" class=\"data row8 col6\" >1.8226</td>\n",
       "                        <td id=\"T_1460d_row8_col7\" class=\"data row8 col7\" >0.0650</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row9\" class=\"row_heading level0 row9\" >lr</th>\n",
       "                        <td id=\"T_1460d_row9_col0\" class=\"data row9 col0\" >Linear Regression</td>\n",
       "                        <td id=\"T_1460d_row9_col1\" class=\"data row9 col1\" >25.1495</td>\n",
       "                        <td id=\"T_1460d_row9_col2\" class=\"data row9 col2\" >1708.5206</td>\n",
       "                        <td id=\"T_1460d_row9_col3\" class=\"data row9 col3\" >41.3225</td>\n",
       "                        <td id=\"T_1460d_row9_col4\" class=\"data row9 col4\" >0.1418</td>\n",
       "                        <td id=\"T_1460d_row9_col5\" class=\"data row9 col5\" >1.3159</td>\n",
       "                        <td id=\"T_1460d_row9_col6\" class=\"data row9 col6\" >1.8245</td>\n",
       "                        <td id=\"T_1460d_row9_col7\" class=\"data row9 col7\" >0.4370</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row10\" class=\"row_heading level0 row10\" >knn</th>\n",
       "                        <td id=\"T_1460d_row10_col0\" class=\"data row10 col0\" >K Neighbors Regressor</td>\n",
       "                        <td id=\"T_1460d_row10_col1\" class=\"data row10 col1\" >21.6627</td>\n",
       "                        <td id=\"T_1460d_row10_col2\" class=\"data row10 col2\" >1763.1355</td>\n",
       "                        <td id=\"T_1460d_row10_col3\" class=\"data row10 col3\" >41.9758</td>\n",
       "                        <td id=\"T_1460d_row10_col4\" class=\"data row10 col4\" >0.1145</td>\n",
       "                        <td id=\"T_1460d_row10_col5\" class=\"data row10 col5\" >1.1372</td>\n",
       "                        <td id=\"T_1460d_row10_col6\" class=\"data row10 col6\" >1.4348</td>\n",
       "                        <td id=\"T_1460d_row10_col7\" class=\"data row10 col7\" >0.8130</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row11\" class=\"row_heading level0 row11\" >en</th>\n",
       "                        <td id=\"T_1460d_row11_col0\" class=\"data row11 col0\" >Elastic Net</td>\n",
       "                        <td id=\"T_1460d_row11_col1\" class=\"data row11 col1\" >26.3483</td>\n",
       "                        <td id=\"T_1460d_row11_col2\" class=\"data row11 col2\" >1830.8023</td>\n",
       "                        <td id=\"T_1460d_row11_col3\" class=\"data row11 col3\" >42.7765</td>\n",
       "                        <td id=\"T_1460d_row11_col4\" class=\"data row11 col4\" >0.0804</td>\n",
       "                        <td id=\"T_1460d_row11_col5\" class=\"data row11 col5\" >1.4243</td>\n",
       "                        <td id=\"T_1460d_row11_col6\" class=\"data row11 col6\" >1.6788</td>\n",
       "                        <td id=\"T_1460d_row11_col7\" class=\"data row11 col7\" >1.6150</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row12\" class=\"row_heading level0 row12\" >lasso</th>\n",
       "                        <td id=\"T_1460d_row12_col0\" class=\"data row12 col0\" >Lasso Regression</td>\n",
       "                        <td id=\"T_1460d_row12_col1\" class=\"data row12 col1\" >26.3723</td>\n",
       "                        <td id=\"T_1460d_row12_col2\" class=\"data row12 col2\" >1831.8645</td>\n",
       "                        <td id=\"T_1460d_row12_col3\" class=\"data row12 col3\" >42.7890</td>\n",
       "                        <td id=\"T_1460d_row12_col4\" class=\"data row12 col4\" >0.0799</td>\n",
       "                        <td id=\"T_1460d_row12_col5\" class=\"data row12 col5\" >1.4279</td>\n",
       "                        <td id=\"T_1460d_row12_col6\" class=\"data row12 col6\" >1.6800</td>\n",
       "                        <td id=\"T_1460d_row12_col7\" class=\"data row12 col7\" >1.3760</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row13\" class=\"row_heading level0 row13\" >omp</th>\n",
       "                        <td id=\"T_1460d_row13_col0\" class=\"data row13 col0\" >Orthogonal Matching Pursuit</td>\n",
       "                        <td id=\"T_1460d_row13_col1\" class=\"data row13 col1\" >26.8935</td>\n",
       "                        <td id=\"T_1460d_row13_col2\" class=\"data row13 col2\" >1840.2330</td>\n",
       "                        <td id=\"T_1460d_row13_col3\" class=\"data row13 col3\" >42.8878</td>\n",
       "                        <td id=\"T_1460d_row13_col4\" class=\"data row13 col4\" >0.0755</td>\n",
       "                        <td id=\"T_1460d_row13_col5\" class=\"data row13 col5\" >1.3864</td>\n",
       "                        <td id=\"T_1460d_row13_col6\" class=\"data row13 col6\" >1.6620</td>\n",
       "                        <td id=\"T_1460d_row13_col7\" class=\"data row13 col7\" >0.0680</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row14\" class=\"row_heading level0 row14\" >huber</th>\n",
       "                        <td id=\"T_1460d_row14_col0\" class=\"data row14 col0\" >Huber Regressor</td>\n",
       "                        <td id=\"T_1460d_row14_col1\" class=\"data row14 col1\" >24.3238</td>\n",
       "                        <td id=\"T_1460d_row14_col2\" class=\"data row14 col2\" >1938.0253</td>\n",
       "                        <td id=\"T_1460d_row14_col3\" class=\"data row14 col3\" >44.0110</td>\n",
       "                        <td id=\"T_1460d_row14_col4\" class=\"data row14 col4\" >0.0265</td>\n",
       "                        <td id=\"T_1460d_row14_col5\" class=\"data row14 col5\" >1.5449</td>\n",
       "                        <td id=\"T_1460d_row14_col6\" class=\"data row14 col6\" >1.2221</td>\n",
       "                        <td id=\"T_1460d_row14_col7\" class=\"data row14 col7\" >5.4420</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row15\" class=\"row_heading level0 row15\" >llar</th>\n",
       "                        <td id=\"T_1460d_row15_col0\" class=\"data row15 col0\" >Lasso Least Angle Regression</td>\n",
       "                        <td id=\"T_1460d_row15_col1\" class=\"data row15 col1\" >27.7529</td>\n",
       "                        <td id=\"T_1460d_row15_col2\" class=\"data row15 col2\" >1990.9916</td>\n",
       "                        <td id=\"T_1460d_row15_col3\" class=\"data row15 col3\" >44.6098</td>\n",
       "                        <td id=\"T_1460d_row15_col4\" class=\"data row15 col4\" >-0.0001</td>\n",
       "                        <td id=\"T_1460d_row15_col5\" class=\"data row15 col5\" >1.2494</td>\n",
       "                        <td id=\"T_1460d_row15_col6\" class=\"data row15 col6\" >1.4905</td>\n",
       "                        <td id=\"T_1460d_row15_col7\" class=\"data row15 col7\" >0.3390</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row16\" class=\"row_heading level0 row16\" >par</th>\n",
       "                        <td id=\"T_1460d_row16_col0\" class=\"data row16 col0\" >Passive Aggressive Regressor</td>\n",
       "                        <td id=\"T_1460d_row16_col1\" class=\"data row16 col1\" >36.9246</td>\n",
       "                        <td id=\"T_1460d_row16_col2\" class=\"data row16 col2\" >3204.7370</td>\n",
       "                        <td id=\"T_1460d_row16_col3\" class=\"data row16 col3\" >54.3478</td>\n",
       "                        <td id=\"T_1460d_row16_col4\" class=\"data row16 col4\" >-0.6131</td>\n",
       "                        <td id=\"T_1460d_row16_col5\" class=\"data row16 col5\" >1.4722</td>\n",
       "                        <td id=\"T_1460d_row16_col6\" class=\"data row16 col6\" >3.3425</td>\n",
       "                        <td id=\"T_1460d_row16_col7\" class=\"data row16 col7\" >0.2210</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1460d_level0_row17\" class=\"row_heading level0 row17\" >ada</th>\n",
       "                        <td id=\"T_1460d_row17_col0\" class=\"data row17 col0\" >AdaBoost Regressor</td>\n",
       "                        <td id=\"T_1460d_row17_col1\" class=\"data row17 col1\" >74.5467</td>\n",
       "                        <td id=\"T_1460d_row17_col2\" class=\"data row17 col2\" >7082.7512</td>\n",
       "                        <td id=\"T_1460d_row17_col3\" class=\"data row17 col3\" >82.5292</td>\n",
       "                        <td id=\"T_1460d_row17_col4\" class=\"data row17 col4\" >-2.5762</td>\n",
       "                        <td id=\"T_1460d_row17_col5\" class=\"data row17 col5\" >1.8155</td>\n",
       "                        <td id=\"T_1460d_row17_col6\" class=\"data row17 col6\" >9.5337</td>\n",
       "                        <td id=\"T_1460d_row17_col7\" class=\"data row17 col7\" >9.5170</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14745c610>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "best = pycr.compare_models(sort='RMSE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `create_model` with one of the types above, will create the model that can then be used like any other model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "#T_34d9b_row10_col0,#T_34d9b_row10_col1,#T_34d9b_row10_col2,#T_34d9b_row10_col3,#T_34d9b_row10_col4,#T_34d9b_row10_col5{\n",
       "            background:  yellow;\n",
       "        }</style><table id=\"T_34d9b_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >MAE</th>        <th class=\"col_heading level0 col1\" >MSE</th>        <th class=\"col_heading level0 col2\" >RMSE</th>        <th class=\"col_heading level0 col3\" >R2</th>        <th class=\"col_heading level0 col4\" >RMSLE</th>        <th class=\"col_heading level0 col5\" >MAPE</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_34d9b_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_34d9b_row0_col0\" class=\"data row0 col0\" >2.0394</td>\n",
       "                        <td id=\"T_34d9b_row0_col1\" class=\"data row0 col1\" >95.0439</td>\n",
       "                        <td id=\"T_34d9b_row0_col2\" class=\"data row0 col2\" >9.7490</td>\n",
       "                        <td id=\"T_34d9b_row0_col3\" class=\"data row0 col3\" >0.9536</td>\n",
       "                        <td id=\"T_34d9b_row0_col4\" class=\"data row0 col4\" >0.2097</td>\n",
       "                        <td id=\"T_34d9b_row0_col5\" class=\"data row0 col5\" >0.1461</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_34d9b_row1_col0\" class=\"data row1 col0\" >1.9653</td>\n",
       "                        <td id=\"T_34d9b_row1_col1\" class=\"data row1 col1\" >13.1927</td>\n",
       "                        <td id=\"T_34d9b_row1_col2\" class=\"data row1 col2\" >3.6322</td>\n",
       "                        <td id=\"T_34d9b_row1_col3\" class=\"data row1 col3\" >0.9936</td>\n",
       "                        <td id=\"T_34d9b_row1_col4\" class=\"data row1 col4\" >0.2171</td>\n",
       "                        <td id=\"T_34d9b_row1_col5\" class=\"data row1 col5\" >0.1557</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_34d9b_row2_col0\" class=\"data row2 col0\" >2.0037</td>\n",
       "                        <td id=\"T_34d9b_row2_col1\" class=\"data row2 col1\" >12.6137</td>\n",
       "                        <td id=\"T_34d9b_row2_col2\" class=\"data row2 col2\" >3.5516</td>\n",
       "                        <td id=\"T_34d9b_row2_col3\" class=\"data row2 col3\" >0.9933</td>\n",
       "                        <td id=\"T_34d9b_row2_col4\" class=\"data row2 col4\" >0.2239</td>\n",
       "                        <td id=\"T_34d9b_row2_col5\" class=\"data row2 col5\" >0.1662</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_34d9b_row3_col0\" class=\"data row3 col0\" >2.0038</td>\n",
       "                        <td id=\"T_34d9b_row3_col1\" class=\"data row3 col1\" >17.1477</td>\n",
       "                        <td id=\"T_34d9b_row3_col2\" class=\"data row3 col2\" >4.1410</td>\n",
       "                        <td id=\"T_34d9b_row3_col3\" class=\"data row3 col3\" >0.9914</td>\n",
       "                        <td id=\"T_34d9b_row3_col4\" class=\"data row3 col4\" >0.2165</td>\n",
       "                        <td id=\"T_34d9b_row3_col5\" class=\"data row3 col5\" >0.1585</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_34d9b_row4_col0\" class=\"data row4 col0\" >1.8106</td>\n",
       "                        <td id=\"T_34d9b_row4_col1\" class=\"data row4 col1\" >9.8252</td>\n",
       "                        <td id=\"T_34d9b_row4_col2\" class=\"data row4 col2\" >3.1345</td>\n",
       "                        <td id=\"T_34d9b_row4_col3\" class=\"data row4 col3\" >0.9948</td>\n",
       "                        <td id=\"T_34d9b_row4_col4\" class=\"data row4 col4\" >0.2090</td>\n",
       "                        <td id=\"T_34d9b_row4_col5\" class=\"data row4 col5\" >0.1422</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_34d9b_row5_col0\" class=\"data row5 col0\" >1.9573</td>\n",
       "                        <td id=\"T_34d9b_row5_col1\" class=\"data row5 col1\" >37.8786</td>\n",
       "                        <td id=\"T_34d9b_row5_col2\" class=\"data row5 col2\" >6.1546</td>\n",
       "                        <td id=\"T_34d9b_row5_col3\" class=\"data row5 col3\" >0.9814</td>\n",
       "                        <td id=\"T_34d9b_row5_col4\" class=\"data row5 col4\" >0.2032</td>\n",
       "                        <td id=\"T_34d9b_row5_col5\" class=\"data row5 col5\" >0.1530</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "                        <td id=\"T_34d9b_row6_col0\" class=\"data row6 col0\" >2.0721</td>\n",
       "                        <td id=\"T_34d9b_row6_col1\" class=\"data row6 col1\" >93.5713</td>\n",
       "                        <td id=\"T_34d9b_row6_col2\" class=\"data row6 col2\" >9.6732</td>\n",
       "                        <td id=\"T_34d9b_row6_col3\" class=\"data row6 col3\" >0.9533</td>\n",
       "                        <td id=\"T_34d9b_row6_col4\" class=\"data row6 col4\" >0.2080</td>\n",
       "                        <td id=\"T_34d9b_row6_col5\" class=\"data row6 col5\" >0.1536</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "                        <td id=\"T_34d9b_row7_col0\" class=\"data row7 col0\" >1.9391</td>\n",
       "                        <td id=\"T_34d9b_row7_col1\" class=\"data row7 col1\" >15.2617</td>\n",
       "                        <td id=\"T_34d9b_row7_col2\" class=\"data row7 col2\" >3.9066</td>\n",
       "                        <td id=\"T_34d9b_row7_col3\" class=\"data row7 col3\" >0.9926</td>\n",
       "                        <td id=\"T_34d9b_row7_col4\" class=\"data row7 col4\" >0.2142</td>\n",
       "                        <td id=\"T_34d9b_row7_col5\" class=\"data row7 col5\" >0.1474</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "                        <td id=\"T_34d9b_row8_col0\" class=\"data row8 col0\" >2.0233</td>\n",
       "                        <td id=\"T_34d9b_row8_col1\" class=\"data row8 col1\" >62.9052</td>\n",
       "                        <td id=\"T_34d9b_row8_col2\" class=\"data row8 col2\" >7.9313</td>\n",
       "                        <td id=\"T_34d9b_row8_col3\" class=\"data row8 col3\" >0.9699</td>\n",
       "                        <td id=\"T_34d9b_row8_col4\" class=\"data row8 col4\" >0.2108</td>\n",
       "                        <td id=\"T_34d9b_row8_col5\" class=\"data row8 col5\" >0.1473</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "                        <td id=\"T_34d9b_row9_col0\" class=\"data row9 col0\" >1.9735</td>\n",
       "                        <td id=\"T_34d9b_row9_col1\" class=\"data row9 col1\" >10.4671</td>\n",
       "                        <td id=\"T_34d9b_row9_col2\" class=\"data row9 col2\" >3.2353</td>\n",
       "                        <td id=\"T_34d9b_row9_col3\" class=\"data row9 col3\" >0.9942</td>\n",
       "                        <td id=\"T_34d9b_row9_col4\" class=\"data row9 col4\" >0.2328</td>\n",
       "                        <td id=\"T_34d9b_row9_col5\" class=\"data row9 col5\" >0.1628</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row10\" class=\"row_heading level0 row10\" >Mean</th>\n",
       "                        <td id=\"T_34d9b_row10_col0\" class=\"data row10 col0\" >1.9788</td>\n",
       "                        <td id=\"T_34d9b_row10_col1\" class=\"data row10 col1\" >36.7907</td>\n",
       "                        <td id=\"T_34d9b_row10_col2\" class=\"data row10 col2\" >5.5109</td>\n",
       "                        <td id=\"T_34d9b_row10_col3\" class=\"data row10 col3\" >0.9818</td>\n",
       "                        <td id=\"T_34d9b_row10_col4\" class=\"data row10 col4\" >0.2145</td>\n",
       "                        <td id=\"T_34d9b_row10_col5\" class=\"data row10 col5\" >0.1533</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_34d9b_level0_row11\" class=\"row_heading level0 row11\" >SD</th>\n",
       "                        <td id=\"T_34d9b_row11_col0\" class=\"data row11 col0\" >0.0679</td>\n",
       "                        <td id=\"T_34d9b_row11_col1\" class=\"data row11 col1\" >32.7283</td>\n",
       "                        <td id=\"T_34d9b_row11_col2\" class=\"data row11 col2\" >2.5338</td>\n",
       "                        <td id=\"T_34d9b_row11_col3\" class=\"data row11 col3\" >0.0160</td>\n",
       "                        <td id=\"T_34d9b_row11_col4\" class=\"data row11 col4\" >0.0082</td>\n",
       "                        <td id=\"T_34d9b_row11_col5\" class=\"data row11 col5\" >0.0073</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14a478ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "catboost = pycr.create_model('catboost')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate this model on the hold_out sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_fff23_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Model</th>        <th class=\"col_heading level0 col1\" >MAE</th>        <th class=\"col_heading level0 col2\" >MSE</th>        <th class=\"col_heading level0 col3\" >RMSE</th>        <th class=\"col_heading level0 col4\" >R2</th>        <th class=\"col_heading level0 col5\" >RMSLE</th>        <th class=\"col_heading level0 col6\" >MAPE</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_fff23_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_fff23_row0_col0\" class=\"data row0 col0\" >CatBoost Regressor</td>\n",
       "                        <td id=\"T_fff23_row0_col1\" class=\"data row0 col1\" >1.9158</td>\n",
       "                        <td id=\"T_fff23_row0_col2\" class=\"data row0 col2\" >30.7283</td>\n",
       "                        <td id=\"T_fff23_row0_col3\" class=\"data row0 col3\" >5.5433</td>\n",
       "                        <td id=\"T_fff23_row0_col4\" class=\"data row0 col4\" >0.9849</td>\n",
       "                        <td id=\"T_fff23_row0_col5\" class=\"data row0 col5\" >0.2069</td>\n",
       "                        <td id=\"T_fff23_row0_col6\" class=\"data row0 col6\" >0.1498</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14a465b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict_result = pycr.predict_model(catboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model is built using the default hyperparameters.  The model with tuned hyperparameters can be found using `tune_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "#T_2aecd_row10_col0,#T_2aecd_row10_col1,#T_2aecd_row10_col2,#T_2aecd_row10_col3,#T_2aecd_row10_col4,#T_2aecd_row10_col5{\n",
       "            background:  yellow;\n",
       "        }</style><table id=\"T_2aecd_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >MAE</th>        <th class=\"col_heading level0 col1\" >MSE</th>        <th class=\"col_heading level0 col2\" >RMSE</th>        <th class=\"col_heading level0 col3\" >R2</th>        <th class=\"col_heading level0 col4\" >RMSLE</th>        <th class=\"col_heading level0 col5\" >MAPE</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_2aecd_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_2aecd_row0_col0\" class=\"data row0 col0\" >2.3507</td>\n",
       "                        <td id=\"T_2aecd_row0_col1\" class=\"data row0 col1\" >116.4995</td>\n",
       "                        <td id=\"T_2aecd_row0_col2\" class=\"data row0 col2\" >10.7935</td>\n",
       "                        <td id=\"T_2aecd_row0_col3\" class=\"data row0 col3\" >0.9431</td>\n",
       "                        <td id=\"T_2aecd_row0_col4\" class=\"data row0 col4\" >0.2424</td>\n",
       "                        <td id=\"T_2aecd_row0_col5\" class=\"data row0 col5\" >0.1784</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_2aecd_row1_col0\" class=\"data row1 col0\" >2.5108</td>\n",
       "                        <td id=\"T_2aecd_row1_col1\" class=\"data row1 col1\" >22.7701</td>\n",
       "                        <td id=\"T_2aecd_row1_col2\" class=\"data row1 col2\" >4.7718</td>\n",
       "                        <td id=\"T_2aecd_row1_col3\" class=\"data row1 col3\" >0.9890</td>\n",
       "                        <td id=\"T_2aecd_row1_col4\" class=\"data row1 col4\" >0.2792</td>\n",
       "                        <td id=\"T_2aecd_row1_col5\" class=\"data row1 col5\" >0.2172</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_2aecd_row2_col0\" class=\"data row2 col0\" >2.2563</td>\n",
       "                        <td id=\"T_2aecd_row2_col1\" class=\"data row2 col1\" >26.8256</td>\n",
       "                        <td id=\"T_2aecd_row2_col2\" class=\"data row2 col2\" >5.1793</td>\n",
       "                        <td id=\"T_2aecd_row2_col3\" class=\"data row2 col3\" >0.9858</td>\n",
       "                        <td id=\"T_2aecd_row2_col4\" class=\"data row2 col4\" >0.2707</td>\n",
       "                        <td id=\"T_2aecd_row2_col5\" class=\"data row2 col5\" >0.1974</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_2aecd_row3_col0\" class=\"data row3 col0\" >2.4275</td>\n",
       "                        <td id=\"T_2aecd_row3_col1\" class=\"data row3 col1\" >47.8128</td>\n",
       "                        <td id=\"T_2aecd_row3_col2\" class=\"data row3 col2\" >6.9147</td>\n",
       "                        <td id=\"T_2aecd_row3_col3\" class=\"data row3 col3\" >0.9761</td>\n",
       "                        <td id=\"T_2aecd_row3_col4\" class=\"data row3 col4\" >0.2687</td>\n",
       "                        <td id=\"T_2aecd_row3_col5\" class=\"data row3 col5\" >0.2114</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_2aecd_row4_col0\" class=\"data row4 col0\" >2.2783</td>\n",
       "                        <td id=\"T_2aecd_row4_col1\" class=\"data row4 col1\" >13.9257</td>\n",
       "                        <td id=\"T_2aecd_row4_col2\" class=\"data row4 col2\" >3.7317</td>\n",
       "                        <td id=\"T_2aecd_row4_col3\" class=\"data row4 col3\" >0.9927</td>\n",
       "                        <td id=\"T_2aecd_row4_col4\" class=\"data row4 col4\" >0.2747</td>\n",
       "                        <td id=\"T_2aecd_row4_col5\" class=\"data row4 col5\" >0.2036</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_2aecd_row5_col0\" class=\"data row5 col0\" >2.3571</td>\n",
       "                        <td id=\"T_2aecd_row5_col1\" class=\"data row5 col1\" >50.3552</td>\n",
       "                        <td id=\"T_2aecd_row5_col2\" class=\"data row5 col2\" >7.0961</td>\n",
       "                        <td id=\"T_2aecd_row5_col3\" class=\"data row5 col3\" >0.9752</td>\n",
       "                        <td id=\"T_2aecd_row5_col4\" class=\"data row5 col4\" >0.2491</td>\n",
       "                        <td id=\"T_2aecd_row5_col5\" class=\"data row5 col5\" >0.1948</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "                        <td id=\"T_2aecd_row6_col0\" class=\"data row6 col0\" >2.5718</td>\n",
       "                        <td id=\"T_2aecd_row6_col1\" class=\"data row6 col1\" >173.7631</td>\n",
       "                        <td id=\"T_2aecd_row6_col2\" class=\"data row6 col2\" >13.1819</td>\n",
       "                        <td id=\"T_2aecd_row6_col3\" class=\"data row6 col3\" >0.9133</td>\n",
       "                        <td id=\"T_2aecd_row6_col4\" class=\"data row6 col4\" >0.2706</td>\n",
       "                        <td id=\"T_2aecd_row6_col5\" class=\"data row6 col5\" >0.2126</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "                        <td id=\"T_2aecd_row7_col0\" class=\"data row7 col0\" >2.4011</td>\n",
       "                        <td id=\"T_2aecd_row7_col1\" class=\"data row7 col1\" >53.7416</td>\n",
       "                        <td id=\"T_2aecd_row7_col2\" class=\"data row7 col2\" >7.3309</td>\n",
       "                        <td id=\"T_2aecd_row7_col3\" class=\"data row7 col3\" >0.9740</td>\n",
       "                        <td id=\"T_2aecd_row7_col4\" class=\"data row7 col4\" >0.2814</td>\n",
       "                        <td id=\"T_2aecd_row7_col5\" class=\"data row7 col5\" >0.2061</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "                        <td id=\"T_2aecd_row8_col0\" class=\"data row8 col0\" >2.6607</td>\n",
       "                        <td id=\"T_2aecd_row8_col1\" class=\"data row8 col1\" >53.3262</td>\n",
       "                        <td id=\"T_2aecd_row8_col2\" class=\"data row8 col2\" >7.3025</td>\n",
       "                        <td id=\"T_2aecd_row8_col3\" class=\"data row8 col3\" >0.9744</td>\n",
       "                        <td id=\"T_2aecd_row8_col4\" class=\"data row8 col4\" >0.2780</td>\n",
       "                        <td id=\"T_2aecd_row8_col5\" class=\"data row8 col5\" >0.2101</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "                        <td id=\"T_2aecd_row9_col0\" class=\"data row9 col0\" >2.2036</td>\n",
       "                        <td id=\"T_2aecd_row9_col1\" class=\"data row9 col1\" >14.4401</td>\n",
       "                        <td id=\"T_2aecd_row9_col2\" class=\"data row9 col2\" >3.8000</td>\n",
       "                        <td id=\"T_2aecd_row9_col3\" class=\"data row9 col3\" >0.9920</td>\n",
       "                        <td id=\"T_2aecd_row9_col4\" class=\"data row9 col4\" >0.2585</td>\n",
       "                        <td id=\"T_2aecd_row9_col5\" class=\"data row9 col5\" >0.1939</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row10\" class=\"row_heading level0 row10\" >Mean</th>\n",
       "                        <td id=\"T_2aecd_row10_col0\" class=\"data row10 col0\" >2.4018</td>\n",
       "                        <td id=\"T_2aecd_row10_col1\" class=\"data row10 col1\" >57.3460</td>\n",
       "                        <td id=\"T_2aecd_row10_col2\" class=\"data row10 col2\" >7.0102</td>\n",
       "                        <td id=\"T_2aecd_row10_col3\" class=\"data row10 col3\" >0.9716</td>\n",
       "                        <td id=\"T_2aecd_row10_col4\" class=\"data row10 col4\" >0.2673</td>\n",
       "                        <td id=\"T_2aecd_row10_col5\" class=\"data row10 col5\" >0.2025</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_2aecd_level0_row11\" class=\"row_heading level0 row11\" >SD</th>\n",
       "                        <td id=\"T_2aecd_row11_col0\" class=\"data row11 col0\" >0.1374</td>\n",
       "                        <td id=\"T_2aecd_row11_col1\" class=\"data row11 col1\" >48.0356</td>\n",
       "                        <td id=\"T_2aecd_row11_col2\" class=\"data row11 col2\" >2.8640</td>\n",
       "                        <td id=\"T_2aecd_row11_col3\" class=\"data row11 col3\" >0.0237</td>\n",
       "                        <td id=\"T_2aecd_row11_col4\" class=\"data row11 col4\" >0.0125</td>\n",
       "                        <td id=\"T_2aecd_row11_col5\" class=\"data row11 col5\" >0.0110</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14a500f50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "catboost = pycr.tune_model(catboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate this model on the hold_out sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_de6bf_\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Model</th>        <th class=\"col_heading level0 col1\" >MAE</th>        <th class=\"col_heading level0 col2\" >MSE</th>        <th class=\"col_heading level0 col3\" >RMSE</th>        <th class=\"col_heading level0 col4\" >R2</th>        <th class=\"col_heading level0 col5\" >RMSLE</th>        <th class=\"col_heading level0 col6\" >MAPE</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_de6bf_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_de6bf_row0_col0\" class=\"data row0 col0\" >CatBoost Regressor</td>\n",
       "                        <td id=\"T_de6bf_row0_col1\" class=\"data row0 col1\" >2.3154</td>\n",
       "                        <td id=\"T_de6bf_row0_col2\" class=\"data row0 col2\" >48.6610</td>\n",
       "                        <td id=\"T_de6bf_row0_col3\" class=\"data row0 col3\" >6.9757</td>\n",
       "                        <td id=\"T_de6bf_row0_col4\" class=\"data row0 col4\" >0.9761</td>\n",
       "                        <td id=\"T_de6bf_row0_col5\" class=\"data row0 col5\" >0.2608</td>\n",
       "                        <td id=\"T_de6bf_row0_col6\" class=\"data row0 col6\" >0.1978</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x14710ddd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tuned_result = pycr.predict_model(catboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finalize the model for deployment by fitting the model onto all of the data including the hold-out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_catboost = pycr.finalize_model(catboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use this final model to predict on the observations not sampled above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pycr.predict_model(final_catboost, data=dftest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the **R^2** for these predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9822"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pycu.check_metric(predictions.arr_delay, predictions.Label, 'R2')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}