EvaluatingModels/evaluation_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Evaluation-Metrics-For-Classification\" data-toc-modified-id=\"Evaluation-Metrics-For-Classification-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Evaluation Metrics For Classification</a></span><ul class=\"toc-item\"><li><span><a href=\"#Precision-&amp;-Recall\" data-toc-modified-id=\"Precision-&amp;-Recall-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Precision &amp; Recall</a></span><ul class=\"toc-item\"><li><span><a href=\"#Precision\" data-toc-modified-id=\"Precision-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span>Precision</a></span></li><li><span><a href=\"#Recall\" data-toc-modified-id=\"Recall-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span>Recall</a></span></li><li><span><a href=\"#Comparing\" data-toc-modified-id=\"Comparing-1.1.3\"><span class=\"toc-item-num\">1.1.3&nbsp;&nbsp;</span>Comparing</a></span></li></ul></li><li><span><a href=\"#Accuracy\" data-toc-modified-id=\"Accuracy-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Accuracy</a></span></li><li><span><a href=\"#$F_1$-Score\" data-toc-modified-id=\"$F_1$-Score-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>$F_1$ Score</a></span></li><li><span><a href=\"#$F_\\beta$-Score\" data-toc-modified-id=\"$F_\\beta$-Score-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>$F_\\beta$ Score</a></span></li></ul></li><li><span><a href=\"#Confusion-Matrix\" data-toc-modified-id=\"Confusion-Matrix-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Confusion Matrix</a></span><ul class=\"toc-item\"><li><span><a href=\"#Examples\" data-toc-modified-id=\"Examples-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Examples</a></span><ul class=\"toc-item\"><li><span><a href=\"#Simple\" data-toc-modified-id=\"Simple-2.1.1\"><span class=\"toc-item-num\">2.1.1&nbsp;&nbsp;</span>Simple</a></span></li><li><span><a href=\"#Multiple-Classes\" data-toc-modified-id=\"Multiple-Classes-2.1.2\"><span class=\"toc-item-num\">2.1.2&nbsp;&nbsp;</span>Multiple Classes</a></span></li><li><span><a href=\"#American-Sign-Language-Classification-Example\" data-toc-modified-id=\"American-Sign-Language-Classification-Example-2.1.3\"><span class=\"toc-item-num\">2.1.3&nbsp;&nbsp;</span>American Sign Language Classification Example</a></span></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Evaluation Metrics For Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "How do we measure how good our classification?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Precision & Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "An inverse relationship"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='images/precisionrecall.png'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Precision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$$ Precision = \\frac{True\\ Pos}{All\\ Selected\\ Pos}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$$ Recall = \\frac{True\\ Pos}{All\\ Pos}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Comparing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "High Recall: \"Everyone is positive!!!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "High Percision: \"Don't want to make guess; only if near 100% certain!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$$ Accuracy = \\frac{True\\ Pos + True\\ Neg}{All\\ Observations}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## $F_1$ Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Harmonic mean: punishes extreme values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$$F_1 = \\frac{2 \\cdot Precision \\cdot Recall}{Precision + Recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## $F_\\beta$ Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "As $\\beta$ increases, **recall** is more important"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "$$F_\\beta =  \\frac{(1+\\beta^2) \\cdot Precision \\cdot Recall}{\\beta^2 \\cdot Precision + Recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- Type 1 Errors: False Positive\n",
    "- Type 2 Errors: False Negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='images/precisionrecall.png'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Confusion matrix:\n",
    "- diagonals are true positives\n",
    "- y-axis (usually the true label)\n",
    "    + False negatives when read horizontal\n",
    "    + False positives when read vertically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='https://raw.githubusercontent.com/learn-co-students/dsc-3-27-06-confusion-matrices-online-ds-sp-000/master/index_files/index_7_0.png'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Multiple Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='https://raw.githubusercontent.com/learn-co-students/dsc-3-27-06-confusion-matrices-online-ds-sp-000/master/index_files/index_9_0.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### American Sign Language Classification Example\n",
    "\n",
    "https://github.com/MrGeislinger/ASLTransalation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='images/vgg16_confusionMatrix.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src='https://raw.githubusercontent.com/MrGeislinger/ASLTransalation/master/fingerspelling/paper/images/resnet50_confusionMatrix.png?token=ACE4BRYOSBINDWBWH3UOJDK43V3EO' />"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}