h2o-py/demos/H2O_tutorial_eeg_eyestate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# H2O Tutorial: EEG Eye State Classification\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Erin LeDell\n",
    "\n",
    "Contact: erin@h2o.ai\n",
    "\n",
    "This tutorial steps through a quick introduction to H2O's Python API. The goal of this tutorial is to introduce through a complete example H2O's capabilities from Python. \n",
    "\n",
    "Most of the functionality for a [Pandas](http://pandas.pydata.org/) `DataFrame` is exactly the same syntax for an `H2OFrame`, so if you are comfortable with Pandas, data frame manipulation will come naturally to you in H2O.  The modeling syntax in the H2O Python API may also remind you of [scikit-learn](http://scikit-learn.org/stable/).  \n",
    "\n",
    "References: [H2O Python API documentation](http://h2o-release.s3.amazonaws.com/h2o/latest_stable_Pydoc.html) and [H2O general documentation](http://docs.h2o.ai)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install H2O in Python\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "This tutorial assumes you have Python 2.7 installed.  The `h2o` Python package has a few dependencies which can be installed using [pip](http://pip.readthedocs.org/en/stable/installing/). The packages that are required are (which also have their own dependencies):\n",
    "```bash\n",
    "pip install requests\n",
    "pip install tabulate\n",
    "pip install scikit-learn \n",
    "```\n",
    "If you have any problems (for example, installing the `scikit-learn` package), check out [this page](https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/product/howto/FAQ.md#python) for tips.\n",
    "\n",
    "### Install h2o\n",
    "\n",
    "Once the dependencies are installed, you can install H2O.  We will use the [latest stable](http://h2o-release.s3.amazonaws.com/h2o/latest_stable.html) version of the `h2o` package, which is currently \"Tibshirani-8.\"  The installation instructions are on the \"Install in Python\" tab on [this page](http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/index.html).\n",
    "\n",
    "```bash\n",
    "# The following command removes the H2O module for Python (if it already exists).\n",
    "pip uninstall h2o\n",
    "\n",
    "# Next, use pip to install this version of the H2O Python module.\n",
    "pip install http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/Python/h2o-3.6.0.8-py2.py3-none-any.whl\n",
    "```\n",
    "\n",
    "For reference, the Python documentation for the latest stable release of H2O is [here](http://h2o-release.s3.amazonaws.com/h2o/latest_stable_Pydoc.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start up an H2O cluster\n",
    "\n",
    "In a Python terminal, we can import the `h2o` package and start up an H2O cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321. connected.\n"
     ]
    }
   ],
   "source": [
    "import h2o\n",
    "\n",
    "# Start an H2O Cluster on your local machine\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you already have an H2O cluster running that you'd like to connect to (for example, in a multi-node Hadoop environment), then you can specify the IP and port of that cluster as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This will not actually do anything since it's a fake IP address\n",
    "# h2o.init(ip=\"123.45.67.89\", port=54321)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download EEG Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code downloads a copy of the [EEG Eye State](http://archive.ics.uci.edu/ml/datasets/EEG+Eye+State#) dataset.  All data is from one continuous EEG measurement with the [Emotiv EEG Neuroheadset](https://emotiv.com/epoc.php). The duration of the measurement was 117 seconds. The eye state was detected via a camera during the EEG measurement and added later manually to the file after analysing the video frames. '1' indicates the eye-closed and '0' the eye-open state. All values are in chronological order with the first measured value at the top of the data.\n",
    "\n",
    "![Emotiv Headset](http://dissociatedpress.com/wp-content/uploads/2013/03/emotiv-490.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can import the data directly into H2O using the `import_file` method in the Python API.  The import path can be a URL, a local path, a path to an HDFS file, or a file on Amazon S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Parse Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "#csv_url = \"http://www.stat.berkeley.edu/~ledell/data/eeg_eyestate_splits.csv\"\n",
    "csv_url = \"https://h2o-public-test-data.s3.amazonaws.com/smalldata/eeg/eeg_eyestate_splits.csv\"\n",
    "data = h2o.import_file(csv_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Data\n",
    "Once we have loaded the data, let's take a quick look.  First the dimension of the frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14980, 16)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's take a look at the top of the frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">    AF3</th><th style=\"text-align: right;\">     F7</th><th style=\"text-align: right;\">     F3</th><th style=\"text-align: right;\">    FC5</th><th style=\"text-align: right;\">     T7</th><th style=\"text-align: right;\">     P7</th><th style=\"text-align: right;\">     O1</th><th style=\"text-align: right;\">     O2</th><th style=\"text-align: right;\">     P8</th><th style=\"text-align: right;\">     T8</th><th style=\"text-align: right;\">    FC6</th><th style=\"text-align: right;\">     F4</th><th style=\"text-align: right;\">     F8</th><th style=\"text-align: right;\">    AF4</th><th style=\"text-align: right;\">  eyeDetection</th><th>split  </th></tr>\n",
       "<tr><td style=\"text-align: right;\">4329.23</td><td style=\"text-align: right;\">4009.23</td><td style=\"text-align: right;\">4289.23</td><td style=\"text-align: right;\">4148.21</td><td style=\"text-align: right;\">4350.26</td><td style=\"text-align: right;\">4586.15</td><td style=\"text-align: right;\">4096.92</td><td style=\"text-align: right;\">4641.03</td><td style=\"text-align: right;\">4222.05</td><td style=\"text-align: right;\">4238.46</td><td style=\"text-align: right;\">4211.28</td><td style=\"text-align: right;\">4280.51</td><td style=\"text-align: right;\">4635.9 </td><td style=\"text-align: right;\">4393.85</td><td style=\"text-align: right;\">             0</td><td>valid  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4324.62</td><td style=\"text-align: right;\">4004.62</td><td style=\"text-align: right;\">4293.85</td><td style=\"text-align: right;\">4148.72</td><td style=\"text-align: right;\">4342.05</td><td style=\"text-align: right;\">4586.67</td><td style=\"text-align: right;\">4097.44</td><td style=\"text-align: right;\">4638.97</td><td style=\"text-align: right;\">4210.77</td><td style=\"text-align: right;\">4226.67</td><td style=\"text-align: right;\">4207.69</td><td style=\"text-align: right;\">4279.49</td><td style=\"text-align: right;\">4632.82</td><td style=\"text-align: right;\">4384.1 </td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4327.69</td><td style=\"text-align: right;\">4006.67</td><td style=\"text-align: right;\">4295.38</td><td style=\"text-align: right;\">4156.41</td><td style=\"text-align: right;\">4336.92</td><td style=\"text-align: right;\">4583.59</td><td style=\"text-align: right;\">4096.92</td><td style=\"text-align: right;\">4630.26</td><td style=\"text-align: right;\">4207.69</td><td style=\"text-align: right;\">4222.05</td><td style=\"text-align: right;\">4206.67</td><td style=\"text-align: right;\">4282.05</td><td style=\"text-align: right;\">4628.72</td><td style=\"text-align: right;\">4389.23</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4328.72</td><td style=\"text-align: right;\">4011.79</td><td style=\"text-align: right;\">4296.41</td><td style=\"text-align: right;\">4155.9 </td><td style=\"text-align: right;\">4343.59</td><td style=\"text-align: right;\">4582.56</td><td style=\"text-align: right;\">4097.44</td><td style=\"text-align: right;\">4630.77</td><td style=\"text-align: right;\">4217.44</td><td style=\"text-align: right;\">4235.38</td><td style=\"text-align: right;\">4210.77</td><td style=\"text-align: right;\">4287.69</td><td style=\"text-align: right;\">4632.31</td><td style=\"text-align: right;\">4396.41</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">4011.79</td><td style=\"text-align: right;\">4292.31</td><td style=\"text-align: right;\">4151.28</td><td style=\"text-align: right;\">4347.69</td><td style=\"text-align: right;\">4586.67</td><td style=\"text-align: right;\">4095.9 </td><td style=\"text-align: right;\">4627.69</td><td style=\"text-align: right;\">4210.77</td><td style=\"text-align: right;\">4244.1 </td><td style=\"text-align: right;\">4212.82</td><td style=\"text-align: right;\">4288.21</td><td style=\"text-align: right;\">4632.82</td><td style=\"text-align: right;\">4398.46</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4321.03</td><td style=\"text-align: right;\">4004.62</td><td style=\"text-align: right;\">4284.1 </td><td style=\"text-align: right;\">4153.33</td><td style=\"text-align: right;\">4345.64</td><td style=\"text-align: right;\">4587.18</td><td style=\"text-align: right;\">4093.33</td><td style=\"text-align: right;\">4616.92</td><td style=\"text-align: right;\">4202.56</td><td style=\"text-align: right;\">4232.82</td><td style=\"text-align: right;\">4209.74</td><td style=\"text-align: right;\">4281.03</td><td style=\"text-align: right;\">4628.21</td><td style=\"text-align: right;\">4389.74</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4319.49</td><td style=\"text-align: right;\">4001.03</td><td style=\"text-align: right;\">4280.51</td><td style=\"text-align: right;\">4151.79</td><td style=\"text-align: right;\">4343.59</td><td style=\"text-align: right;\">4584.62</td><td style=\"text-align: right;\">4089.74</td><td style=\"text-align: right;\">4615.9 </td><td style=\"text-align: right;\">4212.31</td><td style=\"text-align: right;\">4226.67</td><td style=\"text-align: right;\">4201.03</td><td style=\"text-align: right;\">4269.74</td><td style=\"text-align: right;\">4625.13</td><td style=\"text-align: right;\">4378.46</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4325.64</td><td style=\"text-align: right;\">4006.67</td><td style=\"text-align: right;\">4278.46</td><td style=\"text-align: right;\">4143.08</td><td style=\"text-align: right;\">4344.1 </td><td style=\"text-align: right;\">4583.08</td><td style=\"text-align: right;\">4087.18</td><td style=\"text-align: right;\">4614.87</td><td style=\"text-align: right;\">4205.64</td><td style=\"text-align: right;\">4230.26</td><td style=\"text-align: right;\">4195.9 </td><td style=\"text-align: right;\">4266.67</td><td style=\"text-align: right;\">4622.05</td><td style=\"text-align: right;\">4380.51</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">4010.77</td><td style=\"text-align: right;\">4276.41</td><td style=\"text-align: right;\">4139.49</td><td style=\"text-align: right;\">4345.13</td><td style=\"text-align: right;\">4584.1 </td><td style=\"text-align: right;\">4091.28</td><td style=\"text-align: right;\">4608.21</td><td style=\"text-align: right;\">4187.69</td><td style=\"text-align: right;\">4229.74</td><td style=\"text-align: right;\">4202.05</td><td style=\"text-align: right;\">4273.85</td><td style=\"text-align: right;\">4627.18</td><td style=\"text-align: right;\">4389.74</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">4011.28</td><td style=\"text-align: right;\">4276.92</td><td style=\"text-align: right;\">4142.05</td><td style=\"text-align: right;\">4344.1 </td><td style=\"text-align: right;\">4582.56</td><td style=\"text-align: right;\">4092.82</td><td style=\"text-align: right;\">4608.72</td><td style=\"text-align: right;\">4194.36</td><td style=\"text-align: right;\">4228.72</td><td style=\"text-align: right;\">4212.82</td><td style=\"text-align: right;\">4277.95</td><td style=\"text-align: right;\">4637.44</td><td style=\"text-align: right;\">4393.33</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first 14 columns are numeric values that represent EEG measurements from the headset.  The \"eyeDetection\" column is the response.  There is an additional column called \"split\" that was added (by me) in order to specify partitions of the data (so we can easily benchmark against other tools outside of H2O using the same splits).  I randomly divided the dataset into three partitions: train (60%), valid (%20) and test (20%) and marked which split each row belongs to in the \"split\" column.\n",
    "\n",
    "Let's take a look at the column names.  The data contains derived features from the medical images of the tumors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'AF3',\n",
       " u'F7',\n",
       " u'F3',\n",
       " u'FC5',\n",
       " u'T7',\n",
       " u'P7',\n",
       " u'O1',\n",
       " u'O2',\n",
       " u'P8',\n",
       " u'T8',\n",
       " u'FC6',\n",
       " u'F4',\n",
       " u'F8',\n",
       " u'AF4',\n",
       " u'eyeDetection',\n",
       " u'split']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select a subset of the columns to look at, typical Pandas indexing applies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">    AF3</th><th style=\"text-align: right;\">  eyeDetection</th><th>split  </th></tr>\n",
       "<tr><td style=\"text-align: right;\">4329.23</td><td style=\"text-align: right;\">             0</td><td>valid  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4324.62</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4327.69</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4328.72</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4321.03</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4319.49</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4325.64</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">             0</td><td>test   </td></tr>\n",
       "<tr><td style=\"text-align: right;\">4326.15</td><td style=\"text-align: right;\">             0</td><td>train  </td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['AF3', 'eyeDetection', 'split']\n",
    "data[columns].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's select a single column, for example -- the response column, and look at the data more closely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  eyeDetection</th></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = 'eyeDetection'\n",
    "data[y]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like a binary response, but let's validate that assumption:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  C1</th></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   1</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't specify the column types when you import the file, H2O makes a guess at what your column types are.  If there are 0's and 1's in a column, H2O will automatically parse that as numeric by default.  \n",
    "\n",
    "Therefore, we should convert the response column to a more efficient \"enum\" representation -- in this case it is a categorial variable with two levels, 0 and 1.  If the only column in my data that is categorical is the response, I typically don't bother specifying the column type during the parse, and instead use this one-liner to convert it aftewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data[y] = data[y].asfactor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check that there are two levels in our response column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].nlevels()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can query the categorical \"levels\" as well ('0' and '1' stand for \"eye open\" and \"eye closed\") to see what they are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['0', '1']]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].levels()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may want to check if there are any missing values, so let's look for NAs in our dataset.  For tree-based methods like GBM and RF, H2O handles missing feature values automatically, so it's not a problem if we are missing certain feature values.  However, it is always a good idea to check to make sure that you are not missing any of the training labels.  \n",
    "\n",
    "To figure out which, if any, values are missing, we can use the `isna` method on the diagnosis column.  The columns in an H2O Frame are also H2O Frames themselves, so all the methods that apply to a Frame also apply to a single column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  C1</th><th style=\"text-align: right;\">  C2</th><th style=\"text-align: right;\">  C3</th><th style=\"text-align: right;\">  C4</th><th style=\"text-align: right;\">  C5</th><th style=\"text-align: right;\">  C6</th><th style=\"text-align: right;\">  C7</th><th style=\"text-align: right;\">  C8</th><th style=\"text-align: right;\">  C9</th><th style=\"text-align: right;\">  C10</th><th style=\"text-align: right;\">  C11</th><th style=\"text-align: right;\">  C12</th><th style=\"text-align: right;\">  C13</th><th style=\"text-align: right;\">  C14</th><th style=\"text-align: right;\">  C15</th><th style=\"text-align: right;\">  C16</th></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">   0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td><td style=\"text-align: right;\">    0</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  C1</th></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   0</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].isna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `isna` method doesn't directly answer the question, \"Does the response column contain any NAs?\", rather it returns a 0 if that cell is not missing (Is NA? FALSE == 0) and a 1 if it is missing (Is NA? TRUE == 1).  So if there are no missing values, then summing over the whole column should produce a summand equal to 0.0.  Let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, no missing labels.  :-)\n",
    "\n",
    "Out of curiosity, let's see if there is any missing data in this frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sum is still zero, so there are no missing values in any of the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next thing I may wonder about in a binary classification problem is the distribution of the response in the training data.  Is one of the two outcomes under-represented in the training set?  Many real datasets have what's called an \"imbalanace\" problem, where one of the classes has far fewer training examples than the other class.  Let's take a look at the distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">  eyeDetection</th><th style=\"text-align: right;\">  Count</th></tr>\n",
       "<tr><td style=\"text-align: right;\">             0</td><td style=\"text-align: right;\">   8257</td></tr>\n",
       "<tr><td style=\"text-align: right;\">             1</td><td style=\"text-align: right;\">   6723</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[y].table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, the data is not exactly evenly distributed between the two classes -- there are more 0's than 1's in the dataset.  However, this level of imbalance shouldn't be much of an issue for the machine learning algos.  (We will revisit this later in the modeling section below).\n",
    "\n",
    "Let's calculate the percentage that each class represents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th style=\"text-align: right;\">   Count</th></tr>\n",
       "<tr><td style=\"text-align: right;\">0.551202</td></tr>\n",
       "<tr><td style=\"text-align: right;\">0.448798</td></tr>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = data.shape[0]  # Total number of training samples\n",
    "data[y].table()['Count']/n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split H2O Frame into a train and test set\n",
    "\n",
    "So far we have explored the original dataset (all rows).  For the machine learning portion of this tutorial, we will break the dataset into three parts: a training set, validation set and a test set.\n",
    "\n",
    "If you want H2O to do the splitting for you, you can use the `split_frame` method.  However, we have explicit splits that we want (for reproducibility reasons), so we can just subset the Frame to get the partitions we want.  \n",
    "\n",
    "Subset the `data` H2O Frame on the \"split\" column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8988, 16)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = data[data['split']==\"train\"]\n",
    "train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2996, 16)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid = data[data['split']==\"valid\"]\n",
    "valid.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2996, 16)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = data[data['split']==\"test\"]\n",
    "test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning in H2O\n",
    "\n",
    "We will do a quick demo of the H2O software using a Gradient Boosting Machine (GBM).  The goal of this problem is to train a model to predict eye state (open vs closed) from EEG data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and Test a GBM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import H2O GBM:\n",
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first create a `model` object of class, `\"H2OGradientBoostingEstimator\"`.  This does not actually do any training, it just sets the model up for training by specifying model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = H2OGradientBoostingEstimator(distribution='bernoulli',\n",
    "                                    ntrees=100,\n",
    "                                    max_depth=4,\n",
    "                                    learn_rate=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify the predictor set and response\n",
    "\n",
    "The `model` object, like all H2O estimator objects, has a `train` method, which will actually perform model training.  At this step we specify the training and (optionally) a validation set, along with the response and predictor variables.\n",
    "\n",
    "The `x` argument should be a list of predictor names in the training frame, and `y` specifies the response column.  We have already set `y = \"eyeDetector\"` above, but we still need to specify `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'AF3',\n",
       " u'F7',\n",
       " u'F3',\n",
       " u'FC5',\n",
       " u'T7',\n",
       " u'P7',\n",
       " u'O1',\n",
       " u'O2',\n",
       " u'P8',\n",
       " u'T8',\n",
       " u'FC6',\n",
       " u'F4',\n",
       " u'F8',\n",
       " u'AF4',\n",
       " u'eyeDetection',\n",
       " u'split']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = list(train.columns)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'AF3',\n",
       " u'F7',\n",
       " u'F3',\n",
       " u'FC5',\n",
       " u'T7',\n",
       " u'P7',\n",
       " u'O1',\n",
       " u'O2',\n",
       " u'P8',\n",
       " u'T8',\n",
       " u'FC6',\n",
       " u'F4']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del x[12:14]  #Remove the 13th and 14th columns, 'eyeDetection' and 'split'\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have specified `x` and `y`, we can train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "gbm Model Build Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "model.train(x=x, y=y, training_frame=train, validation_frame=valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect Model\n",
    "\n",
    "The type of results shown when you print a model, are determined by the following:\n",
    "- Model class of the estimator (e.g. GBM, RF, GLM, DL)\n",
    "- The type of machine learning problem (e.g. binary classification, multiclass classification, regression)\n",
    "- The data you specify (e.g. `training_frame` only, `training_frame` and `validation_frame`, or `training_frame` and `nfolds`)\n",
    "\n",
    "Below, we see a GBM Model Summary, as well as training and validation metrics since we supplied a `validation_frame`.  Since this a binary classification task, we are shown the relevant performance metrics, which inclues: MSE, R^2, LogLoss, AUC and Gini.  Also, we are shown a Confusion Matrix, where the threshold for classification is chosen automatically (by H2O) as the threshold which maximizes the F1 score.\n",
    "\n",
    "The scoring history is also printed, which shows the performance metrics over some increment such as \"number of trees\" in the case of GBM and RF.\n",
    "\n",
    "Lastly, for tree-based methods (GBM and RF), we also print variable importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OGradientBoostingEstimator :  Gradient Boosting Machine\n",
      "Model Key:  GBM_model_python_1448559565749_9080\n",
      "\n",
      "Model Summary:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>number_of_trees</b></td>\n",
       "<td><b>model_size_in_bytes</b></td>\n",
       "<td><b>min_depth</b></td>\n",
       "<td><b>max_depth</b></td>\n",
       "<td><b>mean_depth</b></td>\n",
       "<td><b>min_leaves</b></td>\n",
       "<td><b>max_leaves</b></td>\n",
       "<td><b>mean_leaves</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>100.0</td>\n",
       "<td>23614.0</td>\n",
       "<td>4.0</td>\n",
       "<td>4.0</td>\n",
       "<td>4.0</td>\n",
       "<td>10.0</td>\n",
       "<td>16.0</td>\n",
       "<td>14.9</td></tr></table></div>"
      ],
      "text/plain": [
       "    number_of_trees    model_size_in_bytes    min_depth    max_depth    mean_depth    min_leaves    max_leaves    mean_leaves\n",
       "--  -----------------  ---------------------  -----------  -----------  ------------  ------------  ------------  -------------\n",
       "    100                23614                  4            4            4             10            16            14.9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsBinomial: gbm\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.114026790434\n",
      "R^2: 0.539835211\n",
      "LogLoss: 0.376005292812\n",
      "AUC: 0.936370388939\n",
      "Gini: 0.872740777878\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.43076103173:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>0</b></td>\n",
       "<td><b>1</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>0</td>\n",
       "<td>4102.0</td>\n",
       "<td>814.0</td>\n",
       "<td>0.1656</td>\n",
       "<td> (814.0/4916.0)</td></tr>\n",
       "<tr><td>1</td>\n",
       "<td>534.0</td>\n",
       "<td>3538.0</td>\n",
       "<td>0.1311</td>\n",
       "<td> (534.0/4072.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>4636.0</td>\n",
       "<td>4352.0</td>\n",
       "<td>0.15</td>\n",
       "<td> (1348.0/8988.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       0     1     Error    Rate\n",
       "-----  ----  ----  -------  ---------------\n",
       "0      4102  814   0.1656   (814.0/4916.0)\n",
       "1      534   3538  0.1311   (534.0/4072.0)\n",
       "Total  4636  4352  0.15     (1348.0/8988.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.4307610</td>\n",
       "<td>0.8399810</td>\n",
       "<td>225.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.2903699</td>\n",
       "<td>0.8927355</td>\n",
       "<td>280.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5822403</td>\n",
       "<td>0.8685376</td>\n",
       "<td>167.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5002179</td>\n",
       "<td>0.8529150</td>\n",
       "<td>199.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9870341</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.5002179</td>\n",
       "<td>0.7030709</td>\n",
       "<td>199.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.4473262</td>\n",
       "<td>0.8492677</td>\n",
       "<td>218.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.430761     0.839981  225\n",
       "max f2                      0.29037      0.892735  280\n",
       "max f0point5                0.58224      0.868538  167\n",
       "max accuracy                0.500218     0.852915  199\n",
       "max precision               0.987034     1         0\n",
       "max absolute_MCC            0.500218     0.703071  199\n",
       "max min_per_class_accuracy  0.447326     0.849268  218"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 45.30 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.9206586</td>\n",
       "<td>0.0500668</td>\n",
       "<td>1.0</td>\n",
       "<td>1.0</td>\n",
       "<td>0.1105108</td>\n",
       "<td>0.1105108</td>\n",
       "<td>2.2072692</td>\n",
       "<td>2.2072692</td>\n",
       "<td>120.7269155</td>\n",
       "<td>120.7269155</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.8721988</td>\n",
       "<td>0.1000223</td>\n",
       "<td>0.9955457</td>\n",
       "<td>0.9977753</td>\n",
       "<td>0.1097741</td>\n",
       "<td>0.2202849</td>\n",
       "<td>2.1974372</td>\n",
       "<td>2.2023587</td>\n",
       "<td>119.7437221</td>\n",
       "<td>120.2358657</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.8154129</td>\n",
       "<td>0.1500890</td>\n",
       "<td>0.9888889</td>\n",
       "<td>0.9948110</td>\n",
       "<td>0.1092829</td>\n",
       "<td>0.3295678</td>\n",
       "<td>2.1827439</td>\n",
       "<td>2.1958156</td>\n",
       "<td>118.2743942</td>\n",
       "<td>119.5815572</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.7587556</td>\n",
       "<td>0.2000445</td>\n",
       "<td>0.9599109</td>\n",
       "<td>0.9860957</td>\n",
       "<td>0.1058448</td>\n",
       "<td>0.4354126</td>\n",
       "<td>2.1187818</td>\n",
       "<td>2.1765785</td>\n",
       "<td>111.8781750</td>\n",
       "<td>117.6578538</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.7006297</td>\n",
       "<td>0.25</td>\n",
       "<td>0.9020045</td>\n",
       "<td>0.9692924</td>\n",
       "<td>0.0994597</td>\n",
       "<td>0.5348723</td>\n",
       "<td>1.9909666</td>\n",
       "<td>2.1394892</td>\n",
       "<td>99.0966610</td>\n",
       "<td>113.9489195</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.6450366</td>\n",
       "<td>0.3000668</td>\n",
       "<td>0.8644444</td>\n",
       "<td>0.9517983</td>\n",
       "<td>0.0955305</td>\n",
       "<td>0.6304028</td>\n",
       "<td>1.9080616</td>\n",
       "<td>2.1008750</td>\n",
       "<td>90.8061559</td>\n",
       "<td>110.0875017</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.5826161</td>\n",
       "<td>0.3500223</td>\n",
       "<td>0.7260579</td>\n",
       "<td>0.9195804</td>\n",
       "<td>0.0800589</td>\n",
       "<td>0.7104617</td>\n",
       "<td>1.6026052</td>\n",
       "<td>2.0297615</td>\n",
       "<td>60.2605222</td>\n",
       "<td>102.9761496</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.5183693</td>\n",
       "<td>0.3999777</td>\n",
       "<td>0.5924276</td>\n",
       "<td>0.8787204</td>\n",
       "<td>0.0653242</td>\n",
       "<td>0.7757859</td>\n",
       "<td>1.3076472</td>\n",
       "<td>1.9395725</td>\n",
       "<td>30.7647206</td>\n",
       "<td>93.9572534</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.4640757</td>\n",
       "<td>0.4500445</td>\n",
       "<td>0.5222222</td>\n",
       "<td>0.8390606</td>\n",
       "<td>0.0577112</td>\n",
       "<td>0.8334971</td>\n",
       "<td>1.1526850</td>\n",
       "<td>1.8520325</td>\n",
       "<td>15.2685003</td>\n",
       "<td>85.2032512</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4124356</td>\n",
       "<td>0.5</td>\n",
       "<td>0.4365256</td>\n",
       "<td>0.7988429</td>\n",
       "<td>0.0481336</td>\n",
       "<td>0.8816306</td>\n",
       "<td>0.9635295</td>\n",
       "<td>1.7632613</td>\n",
       "<td>-3.6470480</td>\n",
       "<td>76.3261297</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.3623212</td>\n",
       "<td>0.5499555</td>\n",
       "<td>0.3385301</td>\n",
       "<td>0.7570301</td>\n",
       "<td>0.0373281</td>\n",
       "<td>0.9189587</td>\n",
       "<td>0.7472270</td>\n",
       "<td>1.6709693</td>\n",
       "<td>-25.2773025</td>\n",
       "<td>67.0969286</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.3185648</td>\n",
       "<td>0.6000223</td>\n",
       "<td>0.2666667</td>\n",
       "<td>0.7161135</td>\n",
       "<td>0.0294695</td>\n",
       "<td>0.9484283</td>\n",
       "<td>0.5886051</td>\n",
       "<td>1.5806552</td>\n",
       "<td>-41.1394892</td>\n",
       "<td>58.0655197</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.2778815</td>\n",
       "<td>0.6499777</td>\n",
       "<td>0.1826281</td>\n",
       "<td>0.6751113</td>\n",
       "<td>0.0201375</td>\n",
       "<td>0.9685658</td>\n",
       "<td>0.4031093</td>\n",
       "<td>1.4901523</td>\n",
       "<td>-59.6890711</td>\n",
       "<td>49.0152268</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.2374738</td>\n",
       "<td>0.6999332</td>\n",
       "<td>0.1469933</td>\n",
       "<td>0.6374185</td>\n",
       "<td>0.0162083</td>\n",
       "<td>0.9847741</td>\n",
       "<td>0.3244538</td>\n",
       "<td>1.4069543</td>\n",
       "<td>-67.5546182</td>\n",
       "<td>40.6954270</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.2014388</td>\n",
       "<td>0.75</td>\n",
       "<td>0.0644444</td>\n",
       "<td>0.5991693</td>\n",
       "<td>0.0071218</td>\n",
       "<td>0.9918959</td>\n",
       "<td>0.1422462</td>\n",
       "<td>1.3225278</td>\n",
       "<td>-85.7753766</td>\n",
       "<td>32.2527832</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>0.1656835</td>\n",
       "<td>0.7999555</td>\n",
       "<td>0.0356347</td>\n",
       "<td>0.5639777</td>\n",
       "<td>0.0039293</td>\n",
       "<td>0.9958251</td>\n",
       "<td>0.0786555</td>\n",
       "<td>1.2448507</td>\n",
       "<td>-92.1344529</td>\n",
       "<td>24.4850685</td></tr>\n",
       "<tr><td></td>\n",
       "<td>17</td>\n",
       "<td>0.1326734</td>\n",
       "<td>0.8499110</td>\n",
       "<td>0.0200445</td>\n",
       "<td>0.5320068</td>\n",
       "<td>0.0022102</td>\n",
       "<td>0.9980354</td>\n",
       "<td>0.0442437</td>\n",
       "<td>1.1742822</td>\n",
       "<td>-95.5756298</td>\n",
       "<td>17.4282216</td></tr>\n",
       "<tr><td></td>\n",
       "<td>18</td>\n",
       "<td>0.1036424</td>\n",
       "<td>0.8999777</td>\n",
       "<td>0.0088889</td>\n",
       "<td>0.5029052</td>\n",
       "<td>0.0009823</td>\n",
       "<td>0.9990177</td>\n",
       "<td>0.0196202</td>\n",
       "<td>1.1100471</td>\n",
       "<td>-98.0379830</td>\n",
       "<td>11.0047092</td></tr>\n",
       "<tr><td></td>\n",
       "<td>19</td>\n",
       "<td>0.0758269</td>\n",
       "<td>0.9499332</td>\n",
       "<td>0.0044543</td>\n",
       "<td>0.4766924</td>\n",
       "<td>0.0004912</td>\n",
       "<td>0.9995088</td>\n",
       "<td>0.0098319</td>\n",
       "<td>1.0521885</td>\n",
       "<td>-99.0168066</td>\n",
       "<td>5.2188506</td></tr>\n",
       "<tr><td></td>\n",
       "<td>20</td>\n",
       "<td>0.0081794</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0044444</td>\n",
       "<td>0.4530485</td>\n",
       "<td>0.0004912</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0098101</td>\n",
       "<td>1.0</td>\n",
       "<td>-99.0189915</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    lower_threshold    cumulative_data_fraction    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    lift        cumulative_lift    gain      cumulative_gain\n",
       "--  -------  -----------------  --------------------------  ---------------  --------------------------  --------------  -------------------------  ----------  -----------------  --------  -----------------\n",
       "    1        0.920659           0.0500668                   1                1                           0.110511        0.110511                   2.20727     2.20727            120.727   120.727\n",
       "    2        0.872199           0.100022                    0.995546         0.997775                    0.109774        0.220285                   2.19744     2.20236            119.744   120.236\n",
       "    3        0.815413           0.150089                    0.988889         0.994811                    0.109283        0.329568                   2.18274     2.19582            118.274   119.582\n",
       "    4        0.758756           0.200045                    0.959911         0.986096                    0.105845        0.435413                   2.11878     2.17658            111.878   117.658\n",
       "    5        0.70063            0.25                        0.902004         0.969292                    0.0994597       0.534872                   1.99097     2.13949            99.0967   113.949\n",
       "    6        0.645037           0.300067                    0.864444         0.951798                    0.0955305       0.630403                   1.90806     2.10088            90.8062   110.088\n",
       "    7        0.582616           0.350022                    0.726058         0.91958                     0.0800589       0.710462                   1.60261     2.02976            60.2605   102.976\n",
       "    8        0.518369           0.399978                    0.592428         0.87872                     0.0653242       0.775786                   1.30765     1.93957            30.7647   93.9573\n",
       "    9        0.464076           0.450045                    0.522222         0.839061                    0.0577112       0.833497                   1.15269     1.85203            15.2685   85.2033\n",
       "    10       0.412436           0.5                         0.436526         0.798843                    0.0481336       0.881631                   0.96353     1.76326            -3.64705  76.3261\n",
       "    11       0.362321           0.549955                    0.33853          0.75703                     0.0373281       0.918959                   0.747227    1.67097            -25.2773  67.0969\n",
       "    12       0.318565           0.600022                    0.266667         0.716113                    0.0294695       0.948428                   0.588605    1.58066            -41.1395  58.0655\n",
       "    13       0.277881           0.649978                    0.182628         0.675111                    0.0201375       0.968566                   0.403109    1.49015            -59.6891  49.0152\n",
       "    14       0.237474           0.699933                    0.146993         0.637419                    0.0162083       0.984774                   0.324454    1.40695            -67.5546  40.6954\n",
       "    15       0.201439           0.75                        0.0644444        0.599169                    0.00712181      0.991896                   0.142246    1.32253            -85.7754  32.2528\n",
       "    16       0.165683           0.799955                    0.0356347        0.563978                    0.00392927      0.995825                   0.0786555   1.24485            -92.1345  24.4851\n",
       "    17       0.132673           0.849911                    0.0200445        0.532007                    0.00221022      0.998035                   0.0442437   1.17428            -95.5756  17.4282\n",
       "    18       0.103642           0.899978                    0.00888889       0.502905                    0.000982318     0.999018                   0.0196202   1.11005            -98.038   11.0047\n",
       "    19       0.0758269          0.949933                    0.00445434       0.476692                    0.000491159     0.999509                   0.00983193  1.05219            -99.0168  5.21885\n",
       "    20       0.00817945         1                           0.00444444       0.453049                    0.000491159     1                          0.00981009  1                  -99.019   0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsBinomial: gbm\n",
      "** Reported on validation data. **\n",
      "\n",
      "MSE: 0.124121459821\n",
      "R^2: 0.499326493922\n",
      "LogLoss: 0.400023227684\n",
      "AUC: 0.917514329947\n",
      "Gini: 0.835028659894\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.460577730095:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>0</b></td>\n",
       "<td><b>1</b></td>\n",
       "<td><b>Error</b></td>\n",
       "<td><b>Rate</b></td></tr>\n",
       "<tr><td>0</td>\n",
       "<td>1364.0</td>\n",
       "<td>271.0</td>\n",
       "<td>0.1657</td>\n",
       "<td> (271.0/1635.0)</td></tr>\n",
       "<tr><td>1</td>\n",
       "<td>223.0</td>\n",
       "<td>1138.0</td>\n",
       "<td>0.1639</td>\n",
       "<td> (223.0/1361.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>1587.0</td>\n",
       "<td>1409.0</td>\n",
       "<td>0.1649</td>\n",
       "<td> (494.0/2996.0)</td></tr></table></div>"
      ],
      "text/plain": [
       "       0     1     Error    Rate\n",
       "-----  ----  ----  -------  --------------\n",
       "0      1364  271   0.1657   (271.0/1635.0)\n",
       "1      223   1138  0.1639   (223.0/1361.0)\n",
       "Total  1587  1409  0.1649   (494.0/2996.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>metric</b></td>\n",
       "<td><b>threshold</b></td>\n",
       "<td><b>value</b></td>\n",
       "<td><b>idx</b></td></tr>\n",
       "<tr><td>max f1</td>\n",
       "<td>0.4605777</td>\n",
       "<td>0.8216606</td>\n",
       "<td>211.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.3288904</td>\n",
       "<td>0.8816658</td>\n",
       "<td>269.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5924794</td>\n",
       "<td>0.8415575</td>\n",
       "<td>158.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4749281</td>\n",
       "<td>0.8354473</td>\n",
       "<td>205.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9728563</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_MCC</td>\n",
       "<td>0.4620595</td>\n",
       "<td>0.6691432</td>\n",
       "<td>210.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.4605777</td>\n",
       "<td>0.8342508</td>\n",
       "<td>211.0</td></tr></table></div>"
      ],
      "text/plain": [
       "metric                      threshold    value     idx\n",
       "--------------------------  -----------  --------  -----\n",
       "max f1                      0.460578     0.821661  211\n",
       "max f2                      0.32889      0.881666  269\n",
       "max f0point5                0.592479     0.841558  158\n",
       "max accuracy                0.474928     0.835447  205\n",
       "max precision               0.972856     1         0\n",
       "max absolute_MCC            0.46206      0.669143  210\n",
       "max min_per_class_accuracy  0.460578     0.834251  211"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 45.43 %\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>group</b></td>\n",
       "<td><b>lower_threshold</b></td>\n",
       "<td><b>cumulative_data_fraction</b></td>\n",
       "<td><b>response_rate</b></td>\n",
       "<td><b>cumulative_response_rate</b></td>\n",
       "<td><b>capture_rate</b></td>\n",
       "<td><b>cumulative_capture_rate</b></td>\n",
       "<td><b>lift</b></td>\n",
       "<td><b>cumulative_lift</b></td>\n",
       "<td><b>gain</b></td>\n",
       "<td><b>cumulative_gain</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>1</td>\n",
       "<td>0.9213965</td>\n",
       "<td>0.0500668</td>\n",
       "<td>1.0</td>\n",
       "<td>1.0</td>\n",
       "<td>0.1102131</td>\n",
       "<td>0.1102131</td>\n",
       "<td>2.2013226</td>\n",
       "<td>2.2013226</td>\n",
       "<td>120.1322557</td>\n",
       "<td>120.1322557</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2</td>\n",
       "<td>0.8773782</td>\n",
       "<td>0.1001335</td>\n",
       "<td>0.9933333</td>\n",
       "<td>0.9966667</td>\n",
       "<td>0.1094783</td>\n",
       "<td>0.2196914</td>\n",
       "<td>2.1866471</td>\n",
       "<td>2.1939848</td>\n",
       "<td>118.6647073</td>\n",
       "<td>119.3984815</td></tr>\n",
       "<tr><td></td>\n",
       "<td>3</td>\n",
       "<td>0.8248667</td>\n",
       "<td>0.1502003</td>\n",
       "<td>0.9866667</td>\n",
       "<td>0.9933333</td>\n",
       "<td>0.1087436</td>\n",
       "<td>0.3284350</td>\n",
       "<td>2.1719716</td>\n",
       "<td>2.1866471</td>\n",
       "<td>117.1971590</td>\n",
       "<td>118.6647073</td></tr>\n",
       "<tr><td></td>\n",
       "<td>4</td>\n",
       "<td>0.7540418</td>\n",
       "<td>0.2002670</td>\n",
       "<td>0.92</td>\n",
       "<td>0.975</td>\n",
       "<td>0.1013960</td>\n",
       "<td>0.4298310</td>\n",
       "<td>2.0252168</td>\n",
       "<td>2.1462895</td>\n",
       "<td>102.5216752</td>\n",
       "<td>114.6289493</td></tr>\n",
       "<tr><td></td>\n",
       "<td>5</td>\n",
       "<td>0.7023105</td>\n",
       "<td>0.25</td>\n",
       "<td>0.8590604</td>\n",
       "<td>0.9519359</td>\n",
       "<td>0.0940485</td>\n",
       "<td>0.5238795</td>\n",
       "<td>1.8910690</td>\n",
       "<td>2.0955180</td>\n",
       "<td>89.1069042</td>\n",
       "<td>109.5518001</td></tr>\n",
       "<tr><td></td>\n",
       "<td>6</td>\n",
       "<td>0.6471993</td>\n",
       "<td>0.3000668</td>\n",
       "<td>0.7666667</td>\n",
       "<td>0.9210234</td>\n",
       "<td>0.0844967</td>\n",
       "<td>0.6083762</td>\n",
       "<td>1.6876806</td>\n",
       "<td>2.0274695</td>\n",
       "<td>68.7680627</td>\n",
       "<td>102.7469496</td></tr>\n",
       "<tr><td></td>\n",
       "<td>7</td>\n",
       "<td>0.5924509</td>\n",
       "<td>0.3501335</td>\n",
       "<td>0.7066667</td>\n",
       "<td>0.8903718</td>\n",
       "<td>0.0778839</td>\n",
       "<td>0.6862601</td>\n",
       "<td>1.5556013</td>\n",
       "<td>1.9599955</td>\n",
       "<td>55.5601274</td>\n",
       "<td>95.9995489</td></tr>\n",
       "<tr><td></td>\n",
       "<td>8</td>\n",
       "<td>0.5395983</td>\n",
       "<td>0.4002003</td>\n",
       "<td>0.6133333</td>\n",
       "<td>0.8557131</td>\n",
       "<td>0.0675974</td>\n",
       "<td>0.7538575</td>\n",
       "<td>1.3501445</td>\n",
       "<td>1.8837005</td>\n",
       "<td>35.0144502</td>\n",
       "<td>88.3700537</td></tr>\n",
       "<tr><td></td>\n",
       "<td>9</td>\n",
       "<td>0.4826484</td>\n",
       "<td>0.4499332</td>\n",
       "<td>0.5167785</td>\n",
       "<td>0.8182493</td>\n",
       "<td>0.0565760</td>\n",
       "<td>0.8104335</td>\n",
       "<td>1.1375962</td>\n",
       "<td>1.8012305</td>\n",
       "<td>13.7596221</td>\n",
       "<td>80.1230549</td></tr>\n",
       "<tr><td></td>\n",
       "<td>10</td>\n",
       "<td>0.4285123</td>\n",
       "<td>0.5</td>\n",
       "<td>0.4066667</td>\n",
       "<td>0.7770360</td>\n",
       "<td>0.0448200</td>\n",
       "<td>0.8552535</td>\n",
       "<td>0.8952045</td>\n",
       "<td>1.7105070</td>\n",
       "<td>-10.4795494</td>\n",
       "<td>71.0506980</td></tr>\n",
       "<tr><td></td>\n",
       "<td>11</td>\n",
       "<td>0.3800505</td>\n",
       "<td>0.5500668</td>\n",
       "<td>0.42</td>\n",
       "<td>0.7445388</td>\n",
       "<td>0.0462895</td>\n",
       "<td>0.9015430</td>\n",
       "<td>0.9245555</td>\n",
       "<td>1.6389701</td>\n",
       "<td>-7.5444526</td>\n",
       "<td>63.8970132</td></tr>\n",
       "<tr><td></td>\n",
       "<td>12</td>\n",
       "<td>0.3377836</td>\n",
       "<td>0.6001335</td>\n",
       "<td>0.3066667</td>\n",
       "<td>0.7080089</td>\n",
       "<td>0.0337987</td>\n",
       "<td>0.9353417</td>\n",
       "<td>0.6750723</td>\n",
       "<td>1.5585560</td>\n",
       "<td>-32.4927749</td>\n",
       "<td>55.8555959</td></tr>\n",
       "<tr><td></td>\n",
       "<td>13</td>\n",
       "<td>0.2898491</td>\n",
       "<td>0.6498665</td>\n",
       "<td>0.1946309</td>\n",
       "<td>0.6687211</td>\n",
       "<td>0.0213079</td>\n",
       "<td>0.9566495</td>\n",
       "<td>0.4284453</td>\n",
       "<td>1.4720709</td>\n",
       "<td>-57.1554670</td>\n",
       "<td>47.2070862</td></tr>\n",
       "<tr><td></td>\n",
       "<td>14</td>\n",
       "<td>0.2490383</td>\n",
       "<td>0.6999332</td>\n",
       "<td>0.1</td>\n",
       "<td>0.6280401</td>\n",
       "<td>0.0110213</td>\n",
       "<td>0.9676708</td>\n",
       "<td>0.2201323</td>\n",
       "<td>1.3825187</td>\n",
       "<td>-77.9867744</td>\n",
       "<td>38.2518745</td></tr>\n",
       "<tr><td></td>\n",
       "<td>15</td>\n",
       "<td>0.2117511</td>\n",
       "<td>0.75</td>\n",
       "<td>0.0866667</td>\n",
       "<td>0.5919003</td>\n",
       "<td>0.0095518</td>\n",
       "<td>0.9772226</td>\n",
       "<td>0.1907813</td>\n",
       "<td>1.3029635</td>\n",
       "<td>-80.9218712</td>\n",
       "<td>30.2963507</td></tr>\n",
       "<tr><td></td>\n",
       "<td>16</td>\n",
       "<td>0.1752189</td>\n",
       "<td>0.8000668</td>\n",
       "<td>0.1</td>\n",
       "<td>0.5611181</td>\n",
       "<td>0.0110213</td>\n",
       "<td>0.9882439</td>\n",
       "<td>0.2201323</td>\n",
       "<td>1.2352019</td>\n",
       "<td>-77.9867744</td>\n",
       "<td>23.5201852</td></tr>\n",
       "<tr><td></td>\n",
       "<td>17</td>\n",
       "<td>0.1445499</td>\n",
       "<td>0.8497997</td>\n",
       "<td>0.0469799</td>\n",
       "<td>0.5310291</td>\n",
       "<td>0.0051433</td>\n",
       "<td>0.9933872</td>\n",
       "<td>0.1034178</td>\n",
       "<td>1.1689663</td>\n",
       "<td>-89.6582162</td>\n",
       "<td>16.8966260</td></tr>\n",
       "<tr><td></td>\n",
       "<td>18</td>\n",
       "<td>0.1085791</td>\n",
       "<td>0.8998665</td>\n",
       "<td>0.0266667</td>\n",
       "<td>0.5029674</td>\n",
       "<td>0.0029390</td>\n",
       "<td>0.9963262</td>\n",
       "<td>0.0587019</td>\n",
       "<td>1.1071934</td>\n",
       "<td>-94.1298065</td>\n",
       "<td>10.7193393</td></tr>\n",
       "<tr><td></td>\n",
       "<td>19</td>\n",
       "<td>0.0788782</td>\n",
       "<td>0.9499332</td>\n",
       "<td>0.02</td>\n",
       "<td>0.4775123</td>\n",
       "<td>0.0022043</td>\n",
       "<td>0.9985305</td>\n",
       "<td>0.0440265</td>\n",
       "<td>1.0511586</td>\n",
       "<td>-95.5973549</td>\n",
       "<td>5.1158593</td></tr>\n",
       "<tr><td></td>\n",
       "<td>20</td>\n",
       "<td>0.0087775</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0133333</td>\n",
       "<td>0.4542724</td>\n",
       "<td>0.0014695</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0293510</td>\n",
       "<td>1.0</td>\n",
       "<td>-97.0649033</td>\n",
       "<td>0.0</td></tr></table></div>"
      ],
      "text/plain": [
       "    group    lower_threshold    cumulative_data_fraction    response_rate    cumulative_response_rate    capture_rate    cumulative_capture_rate    lift       cumulative_lift    gain      cumulative_gain\n",
       "--  -------  -----------------  --------------------------  ---------------  --------------------------  --------------  -------------------------  ---------  -----------------  --------  -----------------\n",
       "    1        0.921396           0.0500668                   1                1                           0.110213        0.110213                   2.20132    2.20132            120.132   120.132\n",
       "    2        0.877378           0.100134                    0.993333         0.996667                    0.109478        0.219691                   2.18665    2.19398            118.665   119.398\n",
       "    3        0.824867           0.1502                      0.986667         0.993333                    0.108744        0.328435                   2.17197    2.18665            117.197   118.665\n",
       "    4        0.754042           0.200267                    0.92             0.975                       0.101396        0.429831                   2.02522    2.14629            102.522   114.629\n",
       "    5        0.70231            0.25                        0.85906          0.951936                    0.0940485       0.52388                    1.89107    2.09552            89.1069   109.552\n",
       "    6        0.647199           0.300067                    0.766667         0.921023                    0.0844967       0.608376                   1.68768    2.02747            68.7681   102.747\n",
       "    7        0.592451           0.350134                    0.706667         0.890372                    0.0778839       0.68626                    1.5556     1.96               55.5601   95.9995\n",
       "    8        0.539598           0.4002                      0.613333         0.855713                    0.0675974       0.753857                   1.35014    1.8837             35.0145   88.3701\n",
       "    9        0.482648           0.449933                    0.516779         0.818249                    0.056576        0.810434                   1.1376     1.80123            13.7596   80.1231\n",
       "    10       0.428512           0.5                         0.406667         0.777036                    0.04482         0.855253                   0.895205   1.71051            -10.4795  71.0507\n",
       "    11       0.38005            0.550067                    0.42             0.744539                    0.0462895       0.901543                   0.924555   1.63897            -7.54445  63.897\n",
       "    12       0.337784           0.600134                    0.306667         0.708009                    0.0337987       0.935342                   0.675072   1.55856            -32.4928  55.8556\n",
       "    13       0.289849           0.649866                    0.194631         0.668721                    0.0213079       0.95665                    0.428445   1.47207            -57.1555  47.2071\n",
       "    14       0.249038           0.699933                    0.1              0.62804                     0.0110213       0.967671                   0.220132   1.38252            -77.9868  38.2519\n",
       "    15       0.211751           0.75                        0.0866667        0.5919                      0.0095518       0.977223                   0.190781   1.30296            -80.9219  30.2964\n",
       "    16       0.175219           0.800067                    0.1              0.561118                    0.0110213       0.988244                   0.220132   1.2352             -77.9868  23.5202\n",
       "    17       0.14455            0.8498                      0.0469799        0.531029                    0.00514328      0.993387                   0.103418   1.16897            -89.6582  16.8966\n",
       "    18       0.108579           0.899866                    0.0266667        0.502967                    0.00293902      0.996326                   0.0587019  1.10719            -94.1298  10.7193\n",
       "    19       0.0788782          0.949933                    0.02             0.477512                    0.00220426      0.99853                    0.0440265  1.05116            -95.5974  5.11586\n",
       "    20       0.00877746         1                           0.0133333        0.454272                    0.00146951      1                          0.029351   1                  -97.0649  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Scoring History:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b></b></td>\n",
       "<td><b>timestamp</b></td>\n",
       "<td><b>duration</b></td>\n",
       "<td><b>number_of_trees</b></td>\n",
       "<td><b>training_MSE</b></td>\n",
       "<td><b>training_logloss</b></td>\n",
       "<td><b>training_AUC</b></td>\n",
       "<td><b>training_lift</b></td>\n",
       "<td><b>training_classification_error</b></td>\n",
       "<td><b>validation_MSE</b></td>\n",
       "<td><b>validation_logloss</b></td>\n",
       "<td><b>validation_AUC</b></td>\n",
       "<td><b>validation_lift</b></td>\n",
       "<td><b>validation_classification_error</b></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:17</td>\n",
       "<td> 0.036 sec</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2393135</td>\n",
       "<td>0.6716245</td>\n",
       "<td>0.7312101</td>\n",
       "<td>1.9746189</td>\n",
       "<td>0.3159769</td>\n",
       "<td>0.2393579</td>\n",
       "<td>0.6717168</td>\n",
       "<td>0.7310525</td>\n",
       "<td>2.0087068</td>\n",
       "<td>0.3197597</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:17</td>\n",
       "<td> 0.071 sec</td>\n",
       "<td>2.0</td>\n",
       "<td>0.2324178</td>\n",
       "<td>0.6576643</td>\n",
       "<td>0.7498719</td>\n",
       "<td>2.0431475</td>\n",
       "<td>0.3172007</td>\n",
       "<td>0.2326440</td>\n",
       "<td>0.6581388</td>\n",
       "<td>0.7446153</td>\n",
       "<td>2.0202614</td>\n",
       "<td>0.3147530</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:18</td>\n",
       "<td> 0.102 sec</td>\n",
       "<td>3.0</td>\n",
       "<td>0.2261150</td>\n",
       "<td>0.6447969</td>\n",
       "<td>0.7735294</td>\n",
       "<td>2.0350459</td>\n",
       "<td>0.3147530</td>\n",
       "<td>0.2265954</td>\n",
       "<td>0.6458099</td>\n",
       "<td>0.7649803</td>\n",
       "<td>2.0103402</td>\n",
       "<td>0.2973965</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:18</td>\n",
       "<td> 0.138 sec</td>\n",
       "<td>4.0</td>\n",
       "<td>0.2205965</td>\n",
       "<td>0.6333786</td>\n",
       "<td>0.7768265</td>\n",
       "<td>2.0432395</td>\n",
       "<td>0.2890521</td>\n",
       "<td>0.2210435</td>\n",
       "<td>0.6343441</td>\n",
       "<td>0.7709237</td>\n",
       "<td>2.0264511</td>\n",
       "<td>0.3034045</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:18</td>\n",
       "<td> 0.183 sec</td>\n",
       "<td>5.0</td>\n",
       "<td>0.2152810</td>\n",
       "<td>0.6223737</td>\n",
       "<td>0.7890473</td>\n",
       "<td>2.1408557</td>\n",
       "<td>0.2900534</td>\n",
       "<td>0.2157338</td>\n",
       "<td>0.6233813</td>\n",
       "<td>0.7850207</td>\n",
       "<td>2.0890102</td>\n",
       "<td>0.3017356</td></tr>\n",
       "<tr><td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:21</td>\n",
       "<td> 3.451 sec</td>\n",
       "<td>34.0</td>\n",
       "<td>0.1554295</td>\n",
       "<td>0.4845459</td>\n",
       "<td>0.8784078</td>\n",
       "<td>2.2023641</td>\n",
       "<td>0.2209613</td>\n",
       "<td>0.1583969</td>\n",
       "<td>0.4916962</td>\n",
       "<td>0.8697198</td>\n",
       "<td>2.2013226</td>\n",
       "<td>0.2166222</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:21</td>\n",
       "<td> 3.612 sec</td>\n",
       "<td>35.0</td>\n",
       "<td>0.1541826</td>\n",
       "<td>0.4815032</td>\n",
       "<td>0.8802874</td>\n",
       "<td>2.2024603</td>\n",
       "<td>0.2212951</td>\n",
       "<td>0.1571958</td>\n",
       "<td>0.4887622</td>\n",
       "<td>0.8718223</td>\n",
       "<td>2.2013226</td>\n",
       "<td>0.2192924</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:21</td>\n",
       "<td> 3.770 sec</td>\n",
       "<td>36.0</td>\n",
       "<td>0.1533055</td>\n",
       "<td>0.4793517</td>\n",
       "<td>0.8821833</td>\n",
       "<td>2.2024073</td>\n",
       "<td>0.2170672</td>\n",
       "<td>0.1562774</td>\n",
       "<td>0.4865299</td>\n",
       "<td>0.8738704</td>\n",
       "<td>2.2013226</td>\n",
       "<td>0.2176235</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:21</td>\n",
       "<td> 3.927 sec</td>\n",
       "<td>37.0</td>\n",
       "<td>0.1523871</td>\n",
       "<td>0.4767990</td>\n",
       "<td>0.8825759</td>\n",
       "<td>2.2024073</td>\n",
       "<td>0.2159546</td>\n",
       "<td>0.1554071</td>\n",
       "<td>0.4841040</td>\n",
       "<td>0.8742059</td>\n",
       "<td>2.2013226</td>\n",
       "<td>0.2152870</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2015-11-26 10:25:22</td>\n",
       "<td> 4.811 sec</td>\n",
       "<td>100.0</td>\n",
       "<td>0.1140268</td>\n",
       "<td>0.3760053</td>\n",
       "<td>0.9363704</td>\n",
       "<td>2.2072692</td>\n",
       "<td>0.1499777</td>\n",
       "<td>0.1241215</td>\n",
       "<td>0.4000232</td>\n",
       "<td>0.9175143</td>\n",
       "<td>2.2013226</td>\n",
       "<td>0.1648865</td></tr></table></div>"
      ],
      "text/plain": [
       "     timestamp            duration    number_of_trees    training_MSE    training_logloss    training_AUC    training_lift    training_classification_error    validation_MSE    validation_logloss    validation_AUC    validation_lift    validation_classification_error\n",
       "---  -------------------  ----------  -----------------  --------------  ------------------  --------------  ---------------  -------------------------------  ----------------  --------------------  ----------------  -----------------  ---------------------------------\n",
       "     2015-11-26 10:25:17  0.036 sec   1.0                0.239313478555  0.67162447955       0.7312101158    1.97461891487    0.315976858033                   0.23935789076     0.671716831898        0.73105245064     2.00870683321      0.319759679573\n",
       "     2015-11-26 10:25:17  0.071 sec   2.0                0.232417836559  0.657664348685      0.749871865014  2.04314747295    0.317200712061                   0.232644025245    0.658138849088        0.744615287824    2.02026139425      0.314753004005\n",
       "     2015-11-26 10:25:18  0.102 sec   3.0                0.226115035209  0.644796855934      0.773529405006  2.03504592494    0.314753004005                   0.226595449596    0.645809907388        0.76498032792     2.01034023465      0.297396528705\n",
       "     2015-11-26 10:25:18  0.138 sec   4.0                0.22059653579   0.633378626899      0.77682647056   2.04323946031    0.289052069426                   0.221043535916    0.634344116339        0.770923745133    2.02645113887      0.303404539386\n",
       "     2015-11-26 10:25:18  0.183 sec   5.0                0.215281001153  0.622373733072      0.78904732612   2.140855747      0.290053404539                   0.215733846033    0.623381272811        0.785020683209    2.08901018159      0.30173564753\n",
       "---  ---                  ---         ---                ---             ---                 ---             ---              ---                              ---               ---                   ---               ---                ---\n",
       "     2015-11-26 10:25:21  3.451 sec   34.0               0.155429504881  0.484545906145      0.878407766189  2.20236411264    0.220961281709                   0.158396887188    0.491696211598        0.869719827344    2.20132255694      0.216622162884\n",
       "     2015-11-26 10:25:21  3.612 sec   35.0               0.154182571062  0.481503232691      0.880287379049  2.20246028994    0.22129506008                    0.157195752374    0.488762184948        0.871822301914    2.20132255694      0.219292389853\n",
       "     2015-11-26 10:25:21  3.770 sec   36.0               0.153305469979  0.479351735359      0.88218327729   2.20240732887    0.217067200712                   0.156277394093    0.486529947139        0.873870400205    2.20132255694      0.217623497997\n",
       "     2015-11-26 10:25:21  3.927 sec   37.0               0.152387113107  0.476799007874      0.88257592485   2.20240732887    0.215954606142                   0.155407142822    0.484104023468        0.874205870391    2.20132255694      0.215287049399\n",
       "     2015-11-26 10:25:22  4.811 sec   100.0              0.114026790434  0.376005292812      0.936370388939  2.20726915521    0.149977748109                   0.124121459821    0.400023227684        0.917514329947    2.20132255694      0.164886515354"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Variable Importances:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>variable</b></td>\n",
       "<td><b>relative_importance</b></td>\n",
       "<td><b>scaled_importance</b></td>\n",
       "<td><b>percentage</b></td></tr>\n",
       "<tr><td>P7</td>\n",
       "<td>1308.6181641</td>\n",
       "<td>1.0</td>\n",
       "<td>0.2080290</td></tr>\n",
       "<tr><td>O1</td>\n",
       "<td>1043.3781738</td>\n",
       "<td>0.7973129</td>\n",
       "<td>0.1658642</td></tr>\n",
       "<tr><td>F7</td>\n",
       "<td>835.8756714</td>\n",
       "<td>0.6387468</td>\n",
       "<td>0.1328779</td></tr>\n",
       "<tr><td>AF3</td>\n",
       "<td>730.2410889</td>\n",
       "<td>0.5580246</td>\n",
       "<td>0.1160853</td></tr>\n",
       "<tr><td>F4</td>\n",
       "<td>465.6864014</td>\n",
       "<td>0.3558612</td>\n",
       "<td>0.0740295</td></tr>\n",
       "<tr><td>O2</td>\n",
       "<td>465.3877869</td>\n",
       "<td>0.3556330</td>\n",
       "<td>0.0739820</td></tr>\n",
       "<tr><td>T8</td>\n",
       "<td>340.6835938</td>\n",
       "<td>0.2603384</td>\n",
       "<td>0.0541580</td></tr>\n",
       "<tr><td>FC6</td>\n",
       "<td>282.6287231</td>\n",
       "<td>0.2159749</td>\n",
       "<td>0.0449291</td></tr>\n",
       "<tr><td>FC5</td>\n",
       "<td>249.6704864</td>\n",
       "<td>0.1907894</td>\n",
       "<td>0.0396897</td></tr>\n",
       "<tr><td>F3</td>\n",
       "<td>238.9603577</td>\n",
       "<td>0.1826051</td>\n",
       "<td>0.0379872</td></tr>\n",
       "<tr><td>T7</td>\n",
       "<td>233.7027283</td>\n",
       "<td>0.1785874</td>\n",
       "<td>0.0371514</td></tr>\n",
       "<tr><td>P8</td>\n",
       "<td>95.7217865</td>\n",
       "<td>0.0731472</td>\n",
       "<td>0.0152167</td></tr></table></div>"
      ],
      "text/plain": [
       "variable    relative_importance    scaled_importance    percentage\n",
       "----------  ---------------------  -------------------  ------------\n",
       "P7          1308.62                1                    0.208029\n",
       "O1          1043.38                0.797313             0.165864\n",
       "F7          835.876                0.638747             0.132878\n",
       "AF3         730.241                0.558025             0.116085\n",
       "F4          465.686                0.355861             0.0740295\n",
       "O2          465.388                0.355633             0.073982\n",
       "T8          340.684                0.260338             0.054158\n",
       "FC6         282.629                0.215975             0.0449291\n",
       "FC5         249.67                 0.190789             0.0396897\n",
       "F3          238.96                 0.182605             0.0379872\n",
       "T7          233.703                0.178587             0.0371514\n",
       "P8          95.7218                0.0731472            0.0152167"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Performance on a Test Set\n",
    "\n",
    "Once a model has been trained, you can also use it to make predictions on a test set.  In the case above, we just ran the model once, so our validation set (passed as `validation_frame`), could have also served as a \"test set.\"  We technically have already created test set predictions and evaluated test set performance.  \n",
    "\n",
    "However, when performing model selection over a variety of model parameters, it is common for users to train a variety of models (using different parameters) using the training set, `train`, and a validation set, `valid`.  Once the user selects the best model (based on validation set performance), the true test of model performance is performed by making a final set of predictions on the held-out (never been used before) test set, `test`.\n",
    "\n",
    "You can use the `model_performance` method to generate predictions on a new dataset.  The results are stored in an object of class, `\"H2OBinomialModelMetrics\"`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'h2o.model.metrics_base.H2OBinomialModelMetrics'>\n"
     ]
    }
   ],
   "source": [
    "perf = model.model_performance(test)\n",
    "print(perf.__class__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual model performance metrics can be extracted using methods like `auc` and `mse`.  In the case of binary classification, we may be most interested in evaluating test set Area Under the ROC Curve (AUC).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91671733144306"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf.auc()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12372290870105287"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf.mse()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validated Performance\n",
    "\n",
    "To perform k-fold cross-validation, you use the same code as above, but you specify `nfolds` as an integer greater than 1, or add a \"fold_column\" to your H2O Frame which indicates a fold ID for each row.\n",
    "\n",
    "Unless you have a specific reason to manually assign the observations to folds, you will find it easiest to simply use the `nfolds` argument.\n",
    "\n",
    "When performing cross-validation, you can still pass a `validation_frame`, but you can also choose to use the original dataset that contains all the rows.  We will cross-validate a model below using the original H2O Frame which is called `data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "gbm Model Build Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "cvmodel = H2OGradientBoostingEstimator(distribution='bernoulli',\n",
    "                                       ntrees=100,\n",
    "                                       max_depth=4,\n",
    "                                       learn_rate=0.1,\n",
    "                                       nfolds=5)\n",
    "\n",
    "cvmodel.train(x=x, y=y, training_frame=data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time around, we will simply pull the training and cross-validation metrics out of the model.  To do so, you use the `auc` method again, and you can specify `train` or `xval` as `True` to get the correct metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.926208136139\n",
      "0.909288088259\n"
     ]
    }
   ],
   "source": [
    "print(cvmodel.auc(train=True))\n",
    "print(cvmodel.auc(xval=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search\n",
    "\n",
    "One way of evaluting models with different parameters is to perform a grid search over a set of parameter values.  For example, in GBM, here are three model parameters that may be useful to search over:\n",
    "- `ntrees`: Number of trees\n",
    "- `max_depth`: Maximum depth of a tree\n",
    "- `learn_rate`: Learning rate in the GBM\n",
    "\n",
    "We will define a grid as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ntrees_opt = [5,50,100]\n",
    "max_depth_opt = [2,3,5]\n",
    "learn_rate_opt = [0.1,0.2]\n",
    "\n",
    "hyper_params = {'ntrees': ntrees_opt, \n",
    "                'max_depth': max_depth_opt,\n",
    "                'learn_rate': learn_rate_opt}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an `\"H2OGridSearch\"` object by specifying the algorithm (GBM) and the hyper parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from h2o.grid.grid_search import H2OGridSearch\n",
    "\n",
    "gs = H2OGridSearch(H2OGradientBoostingEstimator, hyper_params = hyper_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An `\"H2OGridSearch\"` object also has a `train` method, which is used to train all the models in the grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "gbm Grid Build Progress: [##################################################] 100%\n"
     ]
    }
   ],
   "source": [
    "gs.train(x=x, y=y, training_frame=train, validation_frame=valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Grid Search Results for H2OGradientBoostingEstimator:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>Model Id</b></td>\n",
       "<td><b>Hyperparameters: [learn_rate, ntrees, max_depth]</b></td>\n",
       "<td><b>mse</b></td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_17</td>\n",
       "<td>[0.2, 100, 5]</td>\n",
       "<td>0.0511374</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_16</td>\n",
       "<td>[0.2, 50, 5]</td>\n",
       "<td>0.0825649</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_8</td>\n",
       "<td>[0.1, 100, 5]</td>\n",
       "<td>0.0827864</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_7</td>\n",
       "<td>[0.1, 50, 5]</td>\n",
       "<td>0.1148579</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_14</td>\n",
       "<td>[0.2, 100, 3]</td>\n",
       "<td>0.1183549</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_13</td>\n",
       "<td>[0.2, 50, 3]</td>\n",
       "<td>0.1433345</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_5</td>\n",
       "<td>[0.1, 100, 3]</td>\n",
       "<td>0.1446235</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_4</td>\n",
       "<td>[0.1, 50, 3]</td>\n",
       "<td>0.1671745</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_11</td>\n",
       "<td>[0.2, 100, 2]</td>\n",
       "<td>0.1718563</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_15</td>\n",
       "<td>[0.2, 5, 5]</td>\n",
       "<td>0.1796775</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_10</td>\n",
       "<td>[0.2, 50, 2]</td>\n",
       "<td>0.1876036</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_2</td>\n",
       "<td>[0.1, 100, 2]</td>\n",
       "<td>0.1879522</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_1</td>\n",
       "<td>[0.1, 50, 2]</td>\n",
       "<td>0.2024828</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_6</td>\n",
       "<td>[0.1, 5, 5]</td>\n",
       "<td>0.2042339</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_12</td>\n",
       "<td>[0.2, 5, 3]</td>\n",
       "<td>0.2126602</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_3</td>\n",
       "<td>[0.1, 5, 3]</td>\n",
       "<td>0.2264710</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_9</td>\n",
       "<td>[0.2, 5, 2]</td>\n",
       "<td>0.2276228</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_0</td>\n",
       "<td>[0.1, 5, 2]</td>\n",
       "<td>0.2355396</td></tr></table></div>"
      ],
      "text/plain": [
       "Model Id                                                 Hyperparameters: [learn_rate, ntrees, max_depth]    mse\n",
       "-------------------------------------------------------  --------------------------------------------------  ---------\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_17  [0.2, 100, 5]                                       0.0511374\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_16  [0.2, 50, 5]                                        0.0825649\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_8   [0.1, 100, 5]                                       0.0827864\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_7   [0.1, 50, 5]                                        0.114858\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_14  [0.2, 100, 3]                                       0.118355\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_13  [0.2, 50, 3]                                        0.143335\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_5   [0.1, 100, 3]                                       0.144623\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_4   [0.1, 50, 3]                                        0.167174\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_11  [0.2, 100, 2]                                       0.171856\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_15  [0.2, 5, 5]                                         0.179677\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_10  [0.2, 50, 2]                                        0.187604\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_2   [0.1, 100, 2]                                       0.187952\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_1   [0.1, 50, 2]                                        0.202483\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_6   [0.1, 5, 5]                                         0.204234\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_12  [0.2, 5, 3]                                         0.21266\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_3   [0.1, 5, 3]                                         0.226471\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_9   [0.2, 5, 2]                                         0.227623\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_0   [0.1, 5, 2]                                         0.23554"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(gs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Grid Search Results for H2OGradientBoostingEstimator:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td><b>Model Id</b></td>\n",
       "<td><b>Hyperparameters: [learn_rate, ntrees, max_depth]</b></td>\n",
       "<td><b>auc(valid=True)</b></td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_17</td>\n",
       "<td>[0.2, 100, 5]</td>\n",
       "<td>0.9604258</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_8</td>\n",
       "<td>[0.1, 100, 5]</td>\n",
       "<td>0.9422169</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_16</td>\n",
       "<td>[0.2, 50, 5]</td>\n",
       "<td>0.9417059</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_7</td>\n",
       "<td>[0.1, 50, 5]</td>\n",
       "<td>0.9169205</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_14</td>\n",
       "<td>[0.2, 100, 3]</td>\n",
       "<td>0.9134707</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_13</td>\n",
       "<td>[0.2, 50, 3]</td>\n",
       "<td>0.8833912</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_5</td>\n",
       "<td>[0.1, 100, 3]</td>\n",
       "<td>0.8803587</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_4</td>\n",
       "<td>[0.1, 50, 3]</td>\n",
       "<td>0.8480756</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_15</td>\n",
       "<td>[0.2, 5, 5]</td>\n",
       "<td>0.8447013</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_6</td>\n",
       "<td>[0.1, 5, 5]</td>\n",
       "<td>0.8227407</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_11</td>\n",
       "<td>[0.2, 100, 2]</td>\n",
       "<td>0.8102571</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_2</td>\n",
       "<td>[0.1, 100, 2]</td>\n",
       "<td>0.7821284</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_10</td>\n",
       "<td>[0.2, 50, 2]</td>\n",
       "<td>0.7818671</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_12</td>\n",
       "<td>[0.2, 5, 3]</td>\n",
       "<td>0.7646862</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_1</td>\n",
       "<td>[0.1, 50, 2]</td>\n",
       "<td>0.7548347</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_3</td>\n",
       "<td>[0.1, 5, 3]</td>\n",
       "<td>0.7282790</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_9</td>\n",
       "<td>[0.2, 5, 2]</td>\n",
       "<td>0.6927028</td></tr>\n",
       "<tr><td>Grid_GBM_py_17_model_python_1448559565749_9958_model_0</td>\n",
       "<td>[0.1, 5, 2]</td>\n",
       "<td>0.6761445</td></tr></table></div>"
      ],
      "text/plain": [
       "Model Id                                                 Hyperparameters: [learn_rate, ntrees, max_depth]    auc(valid=True)\n",
       "-------------------------------------------------------  --------------------------------------------------  -----------------\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_17  [0.2, 100, 5]                                       0.960426\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_8   [0.1, 100, 5]                                       0.942217\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_16  [0.2, 50, 5]                                        0.941706\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_7   [0.1, 50, 5]                                        0.91692\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_14  [0.2, 100, 3]                                       0.913471\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_13  [0.2, 50, 3]                                        0.883391\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_5   [0.1, 100, 3]                                       0.880359\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_4   [0.1, 50, 3]                                        0.848076\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_15  [0.2, 5, 5]                                         0.844701\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_6   [0.1, 5, 5]                                         0.822741\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_11  [0.2, 100, 2]                                       0.810257\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_2   [0.1, 100, 2]                                       0.782128\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_10  [0.2, 50, 2]                                        0.781867\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_12  [0.2, 5, 3]                                         0.764686\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_1   [0.1, 50, 2]                                        0.754835\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_3   [0.1, 5, 3]                                         0.728279\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_9   [0.2, 5, 2]                                         0.692703\n",
       "Grid_GBM_py_17_model_python_1448559565749_9958_model_0   [0.1, 5, 2]                                         0.676145"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# print out the auc for all of the models\n",
    "auc_table = gs.sort_by('auc(valid=True)',increasing=False)\n",
    "print(auc_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"best\" model in terms of validation set AUC is listed first in auc_table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9894042107804035"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model = h2o.get_model(auc_table['Model Id'][0])\n",
    "best_model.auc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we may want to do is generate predictions on the test set using the \"best\" model, and evaluate the test set AUC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9609710824540837"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_perf = best_model.model_performance(test)\n",
    "best_perf.auc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test set AUC is approximately 0.96.  Not bad!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}