notebooks/04.3_measuring_prediction_performance.ipynb

{
 "metadata": {
  "name": "04C_measuring_prediction_performance"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Measuring prediction performance"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here we will discuss how to use **validation sets** to get a better measure of\n",
      "performance for a classifier."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Using the K-neighbors classifier"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here we'll continue to look at the digits data, but we'll switch to the\n",
      "K-Neighbors classifier.  The K-neighbors classifier is an instance-based\n",
      "classifier.  The K-neighbors classifier predicts the label of\n",
      "an unknown point based on the labels of the *K* nearest points in the\n",
      "parameter space."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Get the data\n",
      "from sklearn.datasets import load_digits\n",
      "digits = load_digits()\n",
      "X = digits.data\n",
      "y = digits.target"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Instantiate and train the classifier\n",
      "from sklearn.neighbors import KNeighborsClassifier\n",
      "clf = KNeighborsClassifier(n_neighbors=1)\n",
      "clf.fit(X, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Check the results using metrics\n",
      "from sklearn import metrics\n",
      "y_pred = clf.predict(X)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print metrics.confusion_matrix(y_pred, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Apparently, we've found a perfect classifier!  But this is misleading\n",
      "for the reasons we saw before: the classifier essentially \"memorizes\"\n",
      "all the samples it has already seen.  To really test how well this\n",
      "algorithm does, we need to try some samples it *hasn't* yet seen.\n",
      "\n",
      "This problem can also occur with regression models. In the following we fit an other instance-based model named \"decision tree\" to the Boston Housing price dataset we introduced previously:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pylab inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import load_boston\n",
      "from sklearn.tree import DecisionTreeRegressor\n",
      "\n",
      "data = load_boston()\n",
      "clf = DecisionTreeRegressor().fit(data.data, data.target)\n",
      "predicted = clf.predict(data.data)\n",
      "expected = data.target\n",
      "\n",
      "plt.scatter(expected, predicted)\n",
      "plt.plot([0, 50], [0, 50], '--k')\n",
      "plt.axis('tight')\n",
      "plt.xlabel('True price ($1000s)')\n",
      "plt.ylabel('Predicted price ($1000s)')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here again the predictions are seemingly perfect as the model was able to perfectly memorize the training set."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "A Better Approach: Using a validation set"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Learning the parameters of a prediction function and testing it on the\n",
      "same data is a methodological mistake: a model that would just repeat\n",
      "the labels of the samples that it has just seen would have a perfect\n",
      "score but would fail to predict anything useful on yet-unseen data.\n",
      "\n",
      "To avoid over-fitting, we have to define two different sets:\n",
      "\n",
      "- a training set X_train, y_train which is used for learning the parameters of a predictive model\n",
      "- a testing set X_test, y_test which is used for evaluating the fitted predictive model\n",
      "\n",
      "In scikit-learn such a random split can be quickly computed with the\n",
      "`train_test_split` helper function.  It can be used this way:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import cross_validation\n",
      "X = digits.data\n",
      "y = digits.target\n",
      "\n",
      "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.25, random_state=0)\n",
      "\n",
      "print X.shape, X_train.shape, X_test.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we train on the training data, and test on the testing data:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train)\n",
      "y_pred = clf.predict(X_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print metrics.confusion_matrix(y_test, y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print metrics.classification_report(y_test, y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The averaged f1-score is often used as a convenient measure of the\n",
      "overall performance of an algorithm.  It appears in the bottom row\n",
      "of the classification report; it can also be accessed directly:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "metrics.f1_score(y_test, y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The over-fitting we saw previously can be quantified by computing the\n",
      "f1-score on the training data itself:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "metrics.f1_score(y_train, clf.predict(X_train))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Validation with a Regression Model"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "These validation metrics also work in the case of regression models.  Here we'll use\n",
      "a Gradient-boosted regression tree, which is a meta-estimator which makes use of the\n",
      "``DecisionTreeRegressor`` we showed above.  We'll start by doing the train-test split\n",
      "as we did with the classification case:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = load_boston()\n",
      "X = data.data\n",
      "y = data.target\n",
      "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.25, random_state=0)\n",
      "\n",
      "print X.shape, X_train.shape, X_test.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we'll compute the training and testing error using the Decision Tree that\n",
      "we saw before:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "est = DecisionTreeRegressor().fit(X_train, y_train)\n",
      "\n",
      "print \"validation:\", metrics.explained_variance_score(y_test, est.predict(X_test))\n",
      "print \"training:\", metrics.explained_variance_score(y_train, est.predict(X_train))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This large spread between validation and training error is characteristic\n",
      "of a **high variance** model.  Decision trees are not entirely useless,\n",
      "however: by combining many individual decision trees within ensemble\n",
      "estimators such as Gradient Boosted Trees or Random Forests, we can get\n",
      "much better performance:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.ensemble import GradientBoostingRegressor\n",
      "est = GradientBoostingRegressor().fit(X_train, y_train)\n",
      "\n",
      "print \"validation:\", metrics.explained_variance_score(y_test, est.predict(X_test))\n",
      "print \"training:\", metrics.explained_variance_score(y_train, est.predict(X_train))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This model is still over-fitting the data, but not by as much as the single tree."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Exercise: Model Selection via Validation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In the previous notebook, we saw Gaussian Naive Bayes classification of the digits.\n",
      "Here we saw K-neighbors classification of the digits.  We've also seen support vector\n",
      "machine classification of digits.  Now that we have these\n",
      "validation tools in place, we can ask quantitatively which of the three estimators\n",
      "works best for the digits dataset.\n",
      "\n",
      "Take a moment and determine the answers to these questions for the digits dataset:\n",
      "\n",
      "- With the default hyper-parameters for each estimator, which gives the best f1 score\n",
      "  on the **validation set**?  Recall that hyperparameters are the parameters set when\n",
      "  you instantiate the classifier: for example, the ``n_neighbors`` in\n",
      "\n",
      "          clf = KNeighborsClassifier(n_neighbors=1)\n",
      "\n",
      "  To use the default value, simply leave them unspecified.\n",
      "- For each classifier, which value for the hyperparameters gives the best results for\n",
      "  the digits data?  For ``LinearSVC``, use ``loss='l2'`` and ``loss='l1'``.  For\n",
      "  ``KNeighborsClassifier`` use ``n_neighbors`` between 1 and 10.  Note that ``GaussianNB``\n",
      "  does not have any adjustable hyperparameters.\n",
      "- Bonus: do the same exercise on the Iris data rather than the Digits data.  Does the\n",
      "  same classifier/hyperparameter combination win out in this case?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.svm import LinearSVC\n",
      "from sklearn.naive_bayes import GaussianNB\n",
      "from sklearn.neighbors import KNeighborsClassifier\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Solution"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load solutions/04C_validation_exercise.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}