Books/Machine Learning Mastery With Python - Jason Brownlee/Chapter10.MachineLearning AlgorithmPerformanceMetrics.ipynb

{
  "cells": [
    {
      "metadata": {
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "### 10.1 Algorithm Evaluation Metrics\n\nVarious different algorithm evaluation metrics are demonstrated for both classification and regression type machine learning problems.\nA caveat in these recipes is the cross validation.cross val score function1 used to report the performance in each recipe.\n"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.2 Classification Metrics\n- Classification Accuracy. \n- Logarithmic Loss.\n- Area Under ROC Curve. \n- Confusion Matrix. \n- Classification Report."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.2.1 Classification Accuracy\nClassification accuracy is the number of correct predictions made as a ratio of all predictions made. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Classification Accuracy\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LogisticRegression\nurl = \"http://ftp.ics.uci.edu/pub/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data\"\nnames=[\"preg\", \"plas\", \"pres\", \"skin\", \"test\", \"mass\", \"pedi\", \"age\", \"class\"] \ndataframe = pandas.read_csv(url, names=names)\narray = dataframe.values\nX = array[:,0:8]\nY = array[:,8]\nnum_folds = 10\nnum_instances = len(X)\nseed = 7\nkfold = cross_validation.KFold(n=num_instances, n_folds=num_folds, random_state=seed) \nmodel = LogisticRegression()\nscoring = 'accuracy'\nresults = cross_validation.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) \nprint(\"Accuracy: %.3f (%.3f)\" % (results.mean(), results.std()))",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Accuracy: 0.770 (0.048)\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.2.2 Logarithmic Loss\nLogarithmic loss (or logloss) is a performance metric for evaluating the predictions of probabilities of membership to a given class. \n The scalar probability between 0 and 1 can be seen as a measure of confidence for a prediction by an algorithm. Predictions that are correct or incorrect are rewarded or punished proportionally to the confidence of the prediction. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Classification LogLoss\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LogisticRegression\nurl = \"http://ftp.ics.uci.edu/pub/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data\"\nnames=[\"preg\", \"plas\", \"pres\", \"skin\", \"test\", \"mass\", \"pedi\", \"age\", \"class\"] \ndataframe = pandas.read_csv(url, names=names)\narray = dataframe.values\nX = array[:,0:8]\nY = array[:,8]\nnum_folds = 10\nnum_instances = len(X)\nseed = 7\nkfold = cross_validation.KFold(n=num_instances, n_folds=num_folds, random_state=seed) \nmodel = LogisticRegression()\nscoring = 'log_loss'\nresults = cross_validation.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) \nprint(\"Logloss: %.3f (%.3f)\" % (results.mean(), results.std()))",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n",
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": "Logloss: -0.493 (0.047)\n",
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n",
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Smaller logloss is better with 0 representing a perfect logloss. "
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.2.3 Area Under ROC Curve\nThe AUC represents a model’s ability to discriminate between positive and negative classes. \nAn area of 1.0 represents a model that made all predictions perfectly. An area of 0.5 represents a model that is as good as random. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Classification ROC AUC\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LogisticRegression\nurl = \"http://ftp.ics.uci.edu/pub/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data\"\nnames=[\"preg\", \"plas\", \"pres\", \"skin\", \"test\", \"mass\", \"pedi\", \"age\", \"class\"]\ndataframe = pandas.read_csv(url, names=names)\narray = dataframe.values\nX = array[:,0:8]\nY = array[:,8]\nnum_folds = 10\nnum_instances = len(X)\nseed = 7\nkfold = cross_validation.KFold(n=num_instances, n_folds=num_folds, random_state=seed) \nmodel = LogisticRegression()\nscoring = 'roc_auc'\nresults = cross_validation.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) \nprint(\"AUC: %.3f (%.3f)\" % (results.mean(), results.std()))",
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": "AUC: 0.824 (0.041)\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "### 10.2.4 Confusion Matrix\nThe confusion matrix is a handy presentation of the accuracy of a model with two or more classes."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Classification Confusion Matrix\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix\nurl = \"http://ftp.ics.uci.edu/pub/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data\"\nnames=[\"preg\", \"plas\", \"pres\", \"skin\", \"test\", \"mass\", \"pedi\", \"age\", \"class\"]\ndataframe = pandas.read_csv(url, names=names)\narray = dataframe.values\nX = array[:,0:8]\nY = array[:,8]\ntest_size = 0.33\nseed = 7\nX_train, X_test, Y_train, Y_test = cross_validation.train_test_split(X, Y, test_size=test_size, random_state=seed)\nmodel = LogisticRegression()\nmodel.fit(X_train, Y_train)\npredicted = model.predict(X_test)\nmatrix = confusion_matrix(Y_test, predicted)\nprint(matrix)",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": "[[141  21]\n [ 41  51]]\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "you can see that the majority of the predictions fall on the diagonal line of the matrix"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.2.5 Classification Report\nThe scikit-learn library provides a convenience report when working on classification problems to give you a quick idea of the accuracy of a model using a number of measures.\nThe classification report() function displays the precision, recall, F1-score and support for each class. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Classification Report\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import classification_report\nurl = \"http://ftp.ics.uci.edu/pub/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data\"\nnames=[\"preg\", \"plas\", \"pres\", \"skin\", \"test\", \"mass\", \"pedi\", \"age\", \"class\"]\ndataframe = pandas.read_csv(url, names=names)\narray = dataframe.values\nX = array[:,0:8]\nY = array[:,8]\ntest_size = 0.33\nseed = 7\nX_train, X_test, Y_train, Y_test = cross_validation.train_test_split(X, Y,\n    test_size=test_size, random_state=seed)\nmodel = LogisticRegression()\nmodel.fit(X_train, Y_train)\npredicted = model.predict(X_test)\nreport = classification_report(Y_test, predicted)\nprint(report)",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": "             precision    recall  f1-score   support\n\n        0.0       0.77      0.87      0.82       162\n        1.0       0.71      0.55      0.62        92\n\navg / total       0.75      0.76      0.75       254\n\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 10.3 Regression Metrics\n- Mean Absolute Error. \n- Mean Squared Error. \n- R2.\n\n#### 10.3.1 Mean Absolute Error\nThe Mean Absolute Error (or MAE) is the sum of the absolute differences between predictions and actual values. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Cross Validation Regression MAE\nimport pandas\nfrom sklearn import cross_validation\nfrom sklearn.linear_model import LinearRegression\nurl = \"https://goo.gl/sXleFv\"\nnames=[\"CRIM\", \"ZN\", \"INDUS\", \"CHAS\", \"NOX\", \"RM\", \"AGE\", \"DIS\", \"RAD\", \"TAX\", \"PTRATIO\",\"B\", \"LSTAT\", \"MEDV\"]\ndataframe = pandas.read_csv(url, delim_whitespace=True, names=names)\narray = dataframe.values\nX = array[:,0:13]\nY = array[:,13]\nnum_folds = 10\nnum_instances = len(X)\nseed = 7\nkfold = cross_validation.KFold(n=num_instances, n_folds=num_folds, random_state=seed) \nmodel = LinearRegression()\nscoring = 'mean_absolute_error'\nresults = cross_validation.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) \nprint(\"MAE: %.3f (%.3f)\" % (results.mean(), results.std()))",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": "MAE: -4.005 (2.084)\n",
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": "/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/metrics/scorer.py:100: DeprecationWarning: Scoring method mean_absolute_error was renamed to neg_mean_absolute_error in version 0.18 and will be removed in 0.20.\n  sample_weight=sample_weight)\n",
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "A value of 0 indicates no error or perfect predictions. Like logloss, this metric is inverted by the cross val score() function."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### 10.3.2 Mean Squared Error\nThe Mean Squared Error (or MSE) is much like the mean absolute error in that it provides a gross idea of the magnitude of error. Taking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation. \n#### 10.3.3 R2 Metric"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python36",
      "display_name": "Python 3.6",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}