Tutorial notebooks/Nested Cross Validation Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "Hyperparameters tuning requires nested cross validation. The idea is that during hyperparameters tuning using GridSearchCV, the data used to find the optimal hyperparameters cannot be used to estimate the performance of them, a separate hold out set is necessary to assess their performance in an unbiased manner. \n",
    "\n",
    "In [Scikit-Learn's nested cross validation](http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html) example, the code showed how to nest a GridSearchCV into a cross_val_score for easy nested cross validation performance estimation. However, one drawback in using this methodology is that you won't see the \"winning\" hyperparameters in each of the inner cross validation loop. \n",
    "\n",
    "This notebook shows \"side-by-side\" the SKLearn method and a longer method which exposes the \"winning\" hyper parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV,KFold,cross_val_predict,cross_val_score,StratifiedKFold\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "from collections import Counter\n",
    "from sklearn.metrics import classification_report,accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data set, we will be using the Iris data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()\n",
    "features = iris.data\n",
    "target = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the inner and outer cross validation loops to be used in the SKLearn example and the \"long\" way example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Following kf is the outer loop\n",
    "outer_kf = StratifiedKFold(n_splits=8,shuffle=True,random_state=1)\n",
    "inner_kf = StratifiedKFold(n_splits=3,shuffle=True,random_state=2)\n",
    "model = SVC()\n",
    "params = {'kernel':['rbf','linear'],'C':[1,10]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scikit-Learn example from the link in the Introduction. In this method, you won't be able to see the winning hyperparameters in each of the inner GridSearchCV loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non nested best score: 0.98\n",
      "Nested scores: [ 0.95238095  0.95238095  1.          0.94444444  0.94444444  1.          1.\n",
      "  0.94444444]\n",
      "Nested score mean: 0.967261904762\n"
     ]
    }
   ],
   "source": [
    "clf = GridSearchCV(estimator=model,param_grid=params,cv=inner_kf)\n",
    "clf.fit(features,target)\n",
    "print 'Non nested best score:',clf.best_score_\n",
    "\n",
    "nested_score = cross_val_score(clf,features,target,cv=outer_kf)\n",
    "print 'Nested scores:',nested_score\n",
    "print 'Nested score mean:',nested_score.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Long way of doing nested loops, but you can see the winning hyperparameters for each inner GridSearchCV loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "({'kernel': 'rbf', 'C': 1}, 0.95238095238095233, 0.97674418604651159)\n",
      "({'kernel': 'rbf', 'C': 1}, 0.95238095238095233, 0.98449612403100772)\n",
      "({'kernel': 'linear', 'C': 1}, 1.0, 0.97727272727272729)\n",
      "({'kernel': 'linear', 'C': 1}, 0.94444444444444442, 0.98484848484848486)\n",
      "({'kernel': 'linear', 'C': 1}, 0.94444444444444442, 0.98484848484848486)\n",
      "({'kernel': 'linear', 'C': 1}, 1.0, 0.98484848484848486)\n",
      "({'kernel': 'rbf', 'C': 1}, 1.0, 0.96212121212121215)\n",
      "({'kernel': 'rbf', 'C': 1}, 0.94444444444444442, 0.96969696969696972)\n",
      "Mean of outer loop accuracy score: 0.967261904762\n"
     ]
    }
   ],
   "source": [
    "outer_loop_accuracy_scores = []\n",
    "inner_loop_won_params = []\n",
    "inner_loop_accuracy_scores = []\n",
    "\n",
    "# Looping through the outer loop, feeding each training set into a GSCV as the inner loop\n",
    "for train_index,test_index in outer_kf.split(features,target):\n",
    "    \n",
    "    GSCV = GridSearchCV(estimator=model,param_grid=params,cv=inner_kf)\n",
    "    \n",
    "    # GSCV is looping through the training data to find the best parameters. This is the inner loop\n",
    "    GSCV.fit(features[train_index],target[train_index])\n",
    "    \n",
    "    # The best hyper parameters from GSCV is now being tested on the unseen outer loop test data.\n",
    "    pred = GSCV.predict(features[test_index])\n",
    "    \n",
    "    # Appending the \"winning\" hyper parameters and their associated accuracy score\n",
    "    inner_loop_won_params.append(GSCV.best_params_)\n",
    "    outer_loop_accuracy_scores.append(accuracy_score(target[test_index],pred))\n",
    "    inner_loop_accuracy_scores.append(GSCV.best_score_)\n",
    "\n",
    "for i in zip(inner_loop_won_params,outer_loop_accuracy_scores,inner_loop_accuracy_scores):\n",
    "    print i\n",
    "\n",
    "print 'Mean of outer loop accuracy score:',np.mean(outer_loop_accuracy_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the mean outer loop accuracy score in the longer method is identical to the nested score mean in the SKLearn method. The nested scores in both methods are also identical. However, in the longer method, you can see the winning hyperparameters.\n",
    "\n",
    "Furthermore, the inner loop accuracy score (acquired through the GSCV.best_score_ method) are in general better than the outer loop accuracy scores, this shows that without using nested CV, your estimation for the generalization error might be too optimistic."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}