Document/Document.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyCM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version : 1.2\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:justify;\">\n",
    "PyCM is a multi-class confusion matrix library written in Python that supports both input data vectors and direct matrix, and a proper tool for post-classification model evaluation that supports most classes and overall statistics parameters.\t\n",
    "PyCM is the swiss-army knife of confusion matrices, targeted mainly at data scientists that need a broad array of metrics for predictive models and an accurate evaluation of large variety of classifiers.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center;\">\n",
    "    <img src=\"../Otherfiles/block_diagram.jpg\">\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Source Code\n",
    "- Download [Version 1.2](https://github.com/sepandhaghighi/pycm/archive/v1.2.zip) or [Latest Source ](https://github.com/sepandhaghighi/pycm/archive/dev.zip)\n",
    "- Run `pip install -r requirements.txt` or `pip3 install -r requirements.txt` (Need root access)\n",
    "- Run `python3 setup.py install` or `python setup.py install` (Need root access)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyPI\n",
    "\n",
    "\n",
    "- Check [Python Packaging User Guide](https://packaging.python.org/installing/)     \n",
    "- Run `pip install pycm --upgrade` or `pip3 install pycm --upgrade` (Need root access)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Easy Install\n",
    "\n",
    "- Run `easy_install --upgrade pycm` (Need root access)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From Vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pycm import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]\n",
    "y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = ConfusionMatrix(y_actu, y_pred,digit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  digit (the number of digits to the right of the decimal point in a number) is new in version 0.6 (default valaue : 5)\n",
    "* Only for print and save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pycm.ConfusionMatrix(classes: [0, 1, 2])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.actual_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.predict_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ACC': {0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334},\n",
       " 'BM': {0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652},\n",
       " 'DOR': {0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998},\n",
       " 'ERR': {0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663},\n",
       " 'F0.5': {0: 0.6521739130434783,\n",
       "  1: 0.45454545454545453,\n",
       "  2: 0.5769230769230769},\n",
       " 'F1': {0: 0.75, 1: 0.4, 2: 0.5454545454545454},\n",
       " 'F2': {0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449},\n",
       " 'FDR': {0: 0.4, 1: 0.5, 2: 0.4},\n",
       " 'FN': {0: 0, 1: 2, 2: 3},\n",
       " 'FNR': {0: 0.0, 1: 0.6666666666666667, 2: 0.5},\n",
       " 'FOR': {0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286},\n",
       " 'FP': {0: 2, 1: 1, 2: 2},\n",
       " 'FPR': {0: 0.2222222222222222,\n",
       "  1: 0.11111111111111116,\n",
       "  2: 0.33333333333333337},\n",
       " 'G': {0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661},\n",
       " 'J': {0: 0.6, 1: 0.25, 2: 0.375},\n",
       " 'LR+': {0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998},\n",
       " 'LR-': {0: 0.0, 1: 0.7500000000000001, 2: 0.75},\n",
       " 'MCC': {0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033},\n",
       " 'MK': {0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126},\n",
       " 'N': {0: 9, 1: 9, 2: 6},\n",
       " 'NPV': {0: 1.0, 1: 0.8, 2: 0.5714285714285714},\n",
       " 'P': {0: 3, 1: 3, 2: 6},\n",
       " 'POP': {0: 12, 1: 12, 2: 12},\n",
       " 'PPV': {0: 0.6, 1: 0.5, 2: 0.6},\n",
       " 'PRE': {0: 0.25, 1: 0.25, 2: 0.5},\n",
       " 'RACC': {0: 0.10416666666666667,\n",
       "  1: 0.041666666666666664,\n",
       "  2: 0.20833333333333334},\n",
       " 'RACCU': {0: 0.1111111111111111,\n",
       "  1: 0.04340277777777778,\n",
       "  2: 0.21006944444444442},\n",
       " 'TN': {0: 7, 1: 8, 2: 4},\n",
       " 'TNR': {0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666},\n",
       " 'TON': {0: 7, 1: 10, 2: 7},\n",
       " 'TOP': {0: 5, 1: 2, 2: 5},\n",
       " 'TP': {0: 3, 1: 1, 2: 3},\n",
       " 'TPR': {0: 1.0, 1: 0.3333333333333333, 2: 0.5}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.class_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  cm.statistic_result in prev versions (<0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'95% CI': (0.30438856248221097, 0.8622781041844558),\n",
       " 'Bennett_S': 0.37500000000000006,\n",
       " 'Chi-Squared': 6.6,\n",
       " 'Chi-Squared DF': 4,\n",
       " 'Conditional Entropy': 0.9591479170272448,\n",
       " 'Cramer_V': 0.5244044240850757,\n",
       " 'Cross Entropy': 1.5935164295556343,\n",
       " 'Gwet_AC1': 0.3893129770992367,\n",
       " 'Hamming Loss': 0.41666666666666663,\n",
       " 'Joint Entropy': 2.4591479170272446,\n",
       " 'KL Divergence': 0.09351642955563438,\n",
       " 'Kappa': 0.35483870967741943,\n",
       " 'Kappa 95% CI': (-0.07707577422109269, 0.7867531935759315),\n",
       " 'Kappa No Prevalence': 0.16666666666666674,\n",
       " 'Kappa Standard Error': 0.2203645326012817,\n",
       " 'Kappa Unbiased': 0.34426229508196726,\n",
       " 'Lambda A': 0.16666666666666666,\n",
       " 'Lambda B': 0.42857142857142855,\n",
       " 'Mutual Information': 0.5242078379544426,\n",
       " 'NIR': 0.5,\n",
       " 'Overall_ACC': 0.5833333333333334,\n",
       " 'Overall_J': (1.225, 0.4083333333333334),\n",
       " 'Overall_RACC': 0.3541666666666667,\n",
       " 'Overall_RACCU': 0.3645833333333333,\n",
       " 'P-Value': 0.38720703125,\n",
       " 'PPV_Macro': 0.5666666666666668,\n",
       " 'PPV_Micro': 0.5833333333333334,\n",
       " 'Phi-Squared': 0.5499999999999999,\n",
       " 'Reference Entropy': 1.5,\n",
       " 'Response Entropy': 1.4833557549816874,\n",
       " 'Scott_PI': 0.34426229508196726,\n",
       " 'Standard Error': 0.14231876063832777,\n",
       " 'Strength_Of_Agreement(Altman)': 'Fair',\n",
       " 'Strength_Of_Agreement(Cicchetti)': 'Poor',\n",
       " 'Strength_Of_Agreement(Fleiss)': 'Poor',\n",
       " 'Strength_Of_Agreement(Landis and Koch)': 'Fair',\n",
       " 'TPR_Macro': 0.611111111111111,\n",
       " 'TPR_Micro': 0.5833333333333334,\n",
       " 'Zero-one Loss': 5}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.overall_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])\n",
    "y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = ConfusionMatrix(y_actu, y_pred,digit=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pycm.ConfusionMatrix(classes: [0, 1, 2])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  numpy.array support in version>0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Direct CM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm2 = ConfusionMatrix(matrix={0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}},digit=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pycm.ConfusionMatrix(classes: [0, 1, 2])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm2.actual_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm2.predict_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm2.classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ACC': {0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334},\n",
       " 'BM': {0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652},\n",
       " 'DOR': {0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998},\n",
       " 'ERR': {0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663},\n",
       " 'F0.5': {0: 0.6521739130434783,\n",
       "  1: 0.45454545454545453,\n",
       "  2: 0.5769230769230769},\n",
       " 'F1': {0: 0.75, 1: 0.4, 2: 0.5454545454545454},\n",
       " 'F2': {0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449},\n",
       " 'FDR': {0: 0.4, 1: 0.5, 2: 0.4},\n",
       " 'FN': {0: 0, 1: 2, 2: 3},\n",
       " 'FNR': {0: 0.0, 1: 0.6666666666666667, 2: 0.5},\n",
       " 'FOR': {0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286},\n",
       " 'FP': {0: 2, 1: 1, 2: 2},\n",
       " 'FPR': {0: 0.2222222222222222,\n",
       "  1: 0.11111111111111116,\n",
       "  2: 0.33333333333333337},\n",
       " 'G': {0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661},\n",
       " 'J': {0: 0.6, 1: 0.25, 2: 0.375},\n",
       " 'LR+': {0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998},\n",
       " 'LR-': {0: 0.0, 1: 0.7500000000000001, 2: 0.75},\n",
       " 'MCC': {0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033},\n",
       " 'MK': {0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126},\n",
       " 'N': {0: 9, 1: 9, 2: 6},\n",
       " 'NPV': {0: 1.0, 1: 0.8, 2: 0.5714285714285714},\n",
       " 'P': {0: 3, 1: 3, 2: 6},\n",
       " 'POP': {0: 12, 1: 12, 2: 12},\n",
       " 'PPV': {0: 0.6, 1: 0.5, 2: 0.6},\n",
       " 'PRE': {0: 0.25, 1: 0.25, 2: 0.5},\n",
       " 'RACC': {0: 0.10416666666666667,\n",
       "  1: 0.041666666666666664,\n",
       "  2: 0.20833333333333334},\n",
       " 'RACCU': {0: 0.1111111111111111,\n",
       "  1: 0.04340277777777778,\n",
       "  2: 0.21006944444444442},\n",
       " 'TN': {0: 7, 1: 8, 2: 4},\n",
       " 'TNR': {0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666},\n",
       " 'TON': {0: 7, 1: 10, 2: 7},\n",
       " 'TOP': {0: 5, 1: 2, 2: 5},\n",
       " 'TP': {0: 3, 1: 1, 2: 3},\n",
       " 'TPR': {0: 1.0, 1: 0.3333333333333333, 2: 0.5}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm2.class_stat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'95% CI': (0.30438856248221097, 0.8622781041844558),\n",
       " 'Bennett_S': 0.37500000000000006,\n",
       " 'Chi-Squared': 6.6,\n",
       " 'Chi-Squared DF': 4,\n",
       " 'Conditional Entropy': 0.9591479170272448,\n",
       " 'Cramer_V': 0.5244044240850757,\n",
       " 'Cross Entropy': 1.5935164295556343,\n",
       " 'Gwet_AC1': 0.3893129770992367,\n",
       " 'Hamming Loss': 0.41666666666666663,\n",
       " 'Joint Entropy': 2.4591479170272446,\n",
       " 'KL Divergence': 0.09351642955563438,\n",
       " 'Kappa': 0.35483870967741943,\n",
       " 'Kappa 95% CI': (-0.07707577422109269, 0.7867531935759315),\n",
       " 'Kappa No Prevalence': 0.16666666666666674,\n",
       " 'Kappa Standard Error': 0.2203645326012817,\n",
       " 'Kappa Unbiased': 0.34426229508196726,\n",
       " 'Lambda A': 0.16666666666666666,\n",
       " 'Lambda B': 0.42857142857142855,\n",
       " 'Mutual Information': 0.5242078379544426,\n",
       " 'NIR': 0.5,\n",
       " 'Overall_ACC': 0.5833333333333334,\n",
       " 'Overall_J': (1.225, 0.4083333333333334),\n",
       " 'Overall_RACC': 0.3541666666666667,\n",
       " 'Overall_RACCU': 0.3645833333333333,\n",
       " 'P-Value': 0.38720703125,\n",
       " 'PPV_Macro': 0.5666666666666668,\n",
       " 'PPV_Micro': 0.5833333333333334,\n",
       " 'Phi-Squared': 0.5499999999999999,\n",
       " 'Reference Entropy': 1.5,\n",
       " 'Response Entropy': 1.4833557549816874,\n",
       " 'Scott_PI': 0.34426229508196726,\n",
       " 'Standard Error': 0.14231876063832777,\n",
       " 'Strength_Of_Agreement(Altman)': 'Fair',\n",
       " 'Strength_Of_Agreement(Cicchetti)': 'Poor',\n",
       " 'Strength_Of_Agreement(Fleiss)': 'Poor',\n",
       " 'Strength_Of_Agreement(Landis and Koch)': 'Fair',\n",
       " 'TPR_Macro': 0.611111111111111,\n",
       " 'TPR_Micro': 0.5833333333333334,\n",
       " 'Zero-one Loss': 5}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.overall_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1\n",
    "* In direct matrix mode actual_vector and predict_vector are empty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`threshold` is added in `Version 0.9` for real value prediction.\t\t\t\n",
    "\t\t\t\t\t\t\n",
    "For more information visit [Example3](http://www.shaghighi.ir/pycm/doc/Example3.html \"Example3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load From File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`file` is added in `Version 0.9.5` in order to load saved confusion matrix with `.obj` format generated by `save_obj` method.\n",
    "\n",
    "For more information visit [Example4](http://www.shaghighi.ir/pycm/doc/Example4.html \"Example4\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.9.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sample_weight` is added in `Version 1.2`\n",
    "\n",
    "For more information visit [Example5](http://www.shaghighi.ir/pycm/doc/Example5.html \"Example5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transpose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`transpose` is added in `Version 1.2` in order to transpose input matrix (only in `Direct CM` mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = ConfusionMatrix(matrix={0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}},digit=5,transpose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0    1    2    \n",
      "Actual\n",
      "0                3    0    2    \n",
      "1                0    1    1    \n",
      "2                0    2    3    \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Online Help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`online_help` function is added in `Version 1.1` in order to open each statistics definition in web browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "\n",
    ">>> from pycm import online_help\n",
    ">>> online_help(\"J\")\n",
    ">>> online_help(\"Strength_Of_Agreement(Landis and Koch)\")\n",
    ">>> online_help(2)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* list of items are available by calling `online_help()` (without argument)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please choose one parameter : \n",
      "\n",
      "Example : online_help(\"J\") or online_help(2)\n",
      "\n",
      "1-95% CI\n",
      "2-ACC\n",
      "3-BM\n",
      "4-Bennett_S\n",
      "5-Chi-Squared\n",
      "6-Chi-Squared DF\n",
      "7-Conditional Entropy\n",
      "8-Cramer_V\n",
      "9-Cross Entropy\n",
      "10-DOR\n",
      "11-ERR\n",
      "12-F0.5\n",
      "13-F1\n",
      "14-F2\n",
      "15-FDR\n",
      "16-FN\n",
      "17-FNR\n",
      "18-FOR\n",
      "19-FP\n",
      "20-FPR\n",
      "21-G\n",
      "22-Gwet_AC1\n",
      "23-Hamming Loss\n",
      "24-J\n",
      "25-Joint Entropy\n",
      "26-KL Divergence\n",
      "27-Kappa\n",
      "28-Kappa 95% CI\n",
      "29-Kappa No Prevalence\n",
      "30-Kappa Standard Error\n",
      "31-Kappa Unbiased\n",
      "32-LR+\n",
      "33-LR-\n",
      "34-Lambda A\n",
      "35-Lambda B\n",
      "36-MCC\n",
      "37-MK\n",
      "38-Mutual Information\n",
      "39-N\n",
      "40-NIR\n",
      "41-NPV\n",
      "42-Overall_ACC\n",
      "43-Overall_J\n",
      "44-Overall_RACC\n",
      "45-Overall_RACCU\n",
      "46-P\n",
      "47-P-Value\n",
      "48-POP\n",
      "49-PPV\n",
      "50-PPV_Macro\n",
      "51-PPV_Micro\n",
      "52-PRE\n",
      "53-Phi-Squared\n",
      "54-RACC\n",
      "55-RACCU\n",
      "56-Reference Entropy\n",
      "57-Response Entropy\n",
      "58-Scott_PI\n",
      "59-Standard Error\n",
      "60-Strength_Of_Agreement(Altman)\n",
      "61-Strength_Of_Agreement(Cicchetti)\n",
      "62-Strength_Of_Agreement(Fleiss)\n",
      "63-Strength_Of_Agreement(Landis and Koch)\n",
      "64-TN\n",
      "65-TNR\n",
      "66-TON\n",
      "67-TOP\n",
      "68-TP\n",
      "69-TPR\n",
      "70-TPR_Macro\n",
      "71-TPR_Micro\n",
      "72-Zero-one Loss\n"
     ]
    }
   ],
   "source": [
    "online_help()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acceptable Data Types\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. `actual_vector` : python `list` or numpy `array` of any stringable objects\n",
    "2. `predict_vector` : python `list` or numpy `array` of any stringable objects\n",
    "3. `matrix` : `dict`\n",
    "4. `digit`: `int`\n",
    "5. `threshold` : `FunctionType (function or lambda)`\n",
    "6. `file` : `File object`\n",
    "7. `sample_weight` : python `list` or numpy `array` of any stringable objects\n",
    "8. `transpose` : `bool`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* run `help(ConfusionMatrix)` for more information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TP (True positive / hit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A true positive test result is one that detects the condition when the\n",
    "condition is present. (correctly identified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 3, 1: 1, 2: 3}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TN (True negative/correct rejection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A true negative test result is one that does not detect the condition when\n",
    "the condition is absent. (correctly rejected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 7, 1: 8, 2: 4}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FP (False positive/false alarm/Type I error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A false positive test result is one that detects the condition when the\n",
    "condition is absent. (incorrectly identified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0, 1: 2, 2: 3}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FN (False negative/miss/Type II error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A false negative test result is one that does not detect the condition when\n",
    "the condition is present. (incorrectly rejected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 2, 1: 1, 2: 2}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### P (Condition positive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(number of) positive samples.\n",
    "Also known as support (the number of occurrences of each class in y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 5, 1: 2, 2: 5}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N (Condition negative)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(number of) negative samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 7, 1: 10, 2: 7}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.N"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TOP (Test outcome positive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(number of) positive outcomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 3, 1: 3, 2: 6}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TOP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TON (Test outcome negative)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(number of) negative outcomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 9, 1: 9, 2: 6}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POP (Population)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 12, 1: 12, 2: 12}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.POP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For more information visit <a href=\"https://en.wikipedia.org/wiki/Confusion_matrix\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR  (sensitivity, recall, hit rate, or true positive rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sensitivity (also called the true positive rate, the recall, or probability of detection in some fields) measures the proportion of positives that are correctly identified as such (e.g. the percentage of sick people who are correctly identified as having the condition).\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Sensitivity_and_specificity\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR=\\frac{TP}{P}=\\frac{TP}{TP+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6, 1: 0.5, 2: 0.6}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TNR (specificity or true negative rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specificity (also called the true negative rate) measures the proportion of negatives that are correctly identified as such (e.g. the percentage of healthy people who are correctly identified as not having the condition).\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Sensitivity_and_specificity\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TNR=\\frac{TN}{N}=\\frac{TN}{TN+FP}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1.0, 1: 0.8, 2: 0.5714285714285714}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TNR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV (precision or positive predictive value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predictive value positive is the proportion of positives that correspond to\n",
    "the presence of the condition\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV=\\frac{TP}{TP+FP}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1.0, 1: 0.3333333333333333, 2: 0.5}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NPV (negative predictive value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predictive value negative is the proportion of negatives that correspond to\n",
    "the absence of the condition\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$NPV=\\frac{TN}{TN+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.NPV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FNR (miss rate or false negative rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The false negative rate is the proportion of positives which yield negative test outcomes with the test, i.e., the conditional probability of a negative test result given that the condition being looked for is present.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FNR=\\frac{FN}{P}=\\frac{FN}{FN+TP}=1-TPR$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.4, 1: 0.5, 2: 0.4}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FNR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FPR (fall-out or false positive rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The false positive rate is the proportion of all negatives that still yield positive test outcomes, i.e., the conditional probability of a positive test result given an event that was not present.\n",
    "\n",
    "The false positive rate is equal to the significance level. The specificity of the test is equal to 1 minus the false positive rate.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FPR=\\frac{FP}{N}=\\frac{FP}{FP+TN}=1-TNR$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FPR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FDR (false discovery rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The false discovery rate (FDR) is a method of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons. FDR-controlling procedures are designed to control the expected proportion of \"discoveries\" (rejected null hypotheses) that are false (incorrect rejections)\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/False_discovery_rate\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FDR=\\frac{FP}{FP+TP}=1-PPV$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.0, 1: 0.6666666666666667, 2: 0.5}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FDR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FOR (false omission rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "False omission rate (FOR) is a statistical method used in multiple hypothesis testing to correct for multiple comparisons and it is the complement of the negative predictive value. It measures the proportion of false negatives which are incorrectly rejected.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FOR=\\frac{FN}{FN+TN}=1-NPV$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.2222222222222222, 1: 0.11111111111111116, 2: 0.33333333333333337}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ACC (accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy is the number of correct predictions from all predictions made\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Accuracy_and_precision\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ACC=\\frac{TP+TN}{P+N}=\\frac{TP+TN}{TP+TN+FP+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ERR(Error rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy is the number of incorrect predictions from all predictions made"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ERR=\\frac{FP+FN}{P+N}=\\frac{FP+FN}{TP+TN+FP+FN}=1-ACC$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ERR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FBeta-Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistical analysis of classification, the F1 score (also F-score or F-measure) is a measure of a test's accuracy. It considers both the precision p and the recall r of the test to compute the score.\n",
    "The F1 score is the harmonic average of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/F1_score\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$F_{\\beta}=(1+\\beta^2).\\frac{PPV.TPR}{(\\beta^2.PPV)+TPR}=\\frac{(1+\\beta^2).TP}{(1+\\beta^2).TP+FP+\\beta^2.FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.75, 1: 0.4, 2: 0.5454545454545454}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6521739130434783, 1: 0.45454545454545453, 2: 0.5769230769230769}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6144578313253012, 1: 0.4857142857142857, 2: 0.5930232558139535}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F_beta(Beta=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MCC (Matthews correlation coefficient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary (two-class) classifications, introduced by biochemist Brian W. Matthews in 1975. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes.The MCC is in essence a correlation coefficient between the observed and predicted binary classifications; it returns a value between −1 and +1. A coefficient of +1 represents a perfect prediction, 0 no better than random prediction and −1 indicates total disagreement between prediction and observation.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Matthews_correlation_coefficient\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MCC=\\frac{TP \\times TN-FP \\times FN}{\\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.MCC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BM (Informedness or Bookmaker Informedness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The informedness of a prediction method as captured by a contingency matrix is defined as the probability that the prediction method will make a correct decision as opposed to guessing and is calculated using the bookmaker algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$BM=TPR+TNR-1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.BM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MK (Markedness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistics and psychology, the social science concept of markedness is quantified as a measure of how much one variable is marked as a predictor or possible cause of another, and is also known as Δp (deltaP) in simple two-choice cases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MK=PPV+NPV-1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.MK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PLR (Positive likelihood ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likelihood ratios are used for assessing the value of performing a diagnostic test. They use the sensitivity and specificity of the test to determine whether a test result usefully changes the probability that a condition (such as a disease state) exists. The first description of the use of likelihood ratios for decision rules was made at a symposium on information theory in 1954.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$(LR+)=\\frac{TPR}{FPR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'None', 1: 2.5000000000000004, 2: 1.4}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLR (Negative likelihood ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likelihood ratios are used for assessing the value of performing a diagnostic test. They use the sensitivity and specificity of the test to determine whether a test result usefully changes the probability that a condition (such as a disease state) exists. The first description of the use of likelihood ratios for decision rules was made at a symposium on information theory in 1954.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$(LR-)=\\frac{FNR}{TNR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.4, 1: 0.625, 2: 0.7000000000000001}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.NLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DOR (Diagnostic odds ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The diagnostic odds ratio is a measure of the effectiveness of a diagnostic test. It is defined as the ratio of the odds of the test being positive if the subject has a disease relative to the odds of the test being positive if the subject does not have the disease.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Diagnostic_odds_ratio\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$DOR=\\frac{LR+}{LR-}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'None', 1: 4.000000000000001, 2: 1.9999999999999998}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.DOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PRE (Prevalence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prevalence is a statistical concept referring to the number of cases of a disease that are present in a particular population at a given time (Reference Likelihood)\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Prevalence\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Prevalence=\\frac{P}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.4166666666666667, 1: 0.16666666666666666, 2: 0.4166666666666667}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PRE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### G (G-measure geometric mean of precision and sensitivity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Geometric mean of precision and sensitivity\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/F1_score#G-measure\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$G=\\sqrt{PPV.TPR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.G"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RACC(Random accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expected accuracy from a strategy of randomly guessing categories according to reference and response distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACC=\\frac{TOP\\times P}{Population^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.10416666666666667, 1: 0.041666666666666664, 2: 0.20833333333333334}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RACCU(Random accuracy unbiased)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expected accuracy from a strategy of randomly guessing categories according to the average of the reference and response distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACCU=(\\frac{TOP+P}{2\\times Population})^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.1111111111111111, 1: 0.04340277777777778, 2: 0.21006944444444442}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.RACCU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### J (Jaccard index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Jaccard index, also known as Intersection over Union and the Jaccard similarity coefficient (originally coined coefficient de communauté by Paul Jaccard), is a statistic used for comparing the similarity and diversity of sample sets.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Jaccard_index\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J(A,B)=\\frac{|A\\cap B|}{|A\\cup B|}=\\frac{|A\\cap B|}{|A|+|B|-|A\\cap B|}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6, 1: 0.25, 2: 0.375}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.J"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overall Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kappa (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kappa is a statistic which measures inter-rater agreement for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as kappa takes into account the possibility of the agreement occurring by chance.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Cohen%27s_kappa\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Kappa=\\frac{ACC_{Overall}-RACC_{Overall}}{1-RACC_{Overall}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.35483870967741943"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Kappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kappa Unbiased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unbiased kappa value is defined in terms of total accuracy and a slightly different computation of expected likelihood that averages the reference and response probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Kappa_{Unbiased}=\\frac{ACC_{Overall}-RACCU_{Overall}}{1-RACCU_{Overall}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.34426229508196726"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.KappaUnbiased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kappa No Prevalence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The kappa statistic adjusted for prevalence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Kappa_{NoPrevalence}=2 \\times ACC_{Overall}-1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.16666666666666674"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.KappaNoPrevalence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kappa 95% CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kappa 95% Confidence Interval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$SE_{Kappa}=\\sqrt{\\frac{ACC_{Overall}(1-RACC_{Overall})}{(1-RACC_{Overall})^2}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Kappa \\pm 1.96\\times SE_{Kappa}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2203645326012817"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Kappa_SE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.07707577422109269, 0.7867531935759315)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Kappa_CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chi-Squared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pearson's chi-squared test is a statistical test applied to sets of categorical data to evaluate how likely it is that any observed difference between the sets arose by chance. It is suitable for unpaired data from large samples.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Chi-squared_test\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\chi^2=\\sum_{i=1}^n\\sum_{j=1}^n\\frac{(Matrix(i,j)-E(i,j))^2}{E(i,j)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$E(i,j)=\\frac{TOP_j\\times P_i}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.6000000000000005"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Chi_Squared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chi-Squared DF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of degrees of freedom of this confusion matrix for the chi-squared statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$DF=(|Classes|-1)^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.DF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phi-Squared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistics, the phi coefficient (or mean square contingency coefficient) is a measure of association for two binary variables. Introduced by Karl Pearson, this measure is similar to the Pearson correlation coefficient in its interpretation. In fact, a Pearson correlation coefficient estimated for two binary variables will return the phi coefficient\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Phi_coefficient\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\phi^2=\\frac{\\chi^2}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.55"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Phi_Squared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cramer's V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistics, Cramér's V (sometimes referred to as Cramér's phi) is a measure of association between two nominal variables, giving a value between 0 and +1 (inclusive). It is based on Pearson's chi-squared statistic and was published by Harald Cramér in 1946.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$V=\\sqrt{\\frac{\\phi^2}{|Classes|-1}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5244044240850758"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 95% CI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistics, a confidence interval (CI) is a type of interval estimate (of a population parameter) that is computed from the observed data. The confidence level is the frequency (i.e., the proportion) of possible confidence intervals that contain the true value of their corresponding parameter. In other words, if confidence intervals are constructed using a given confidence level in an infinite number of independent experiments, the proportion of those intervals that contain the true value of the parameter will match the confidence level.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Confidence_interval\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$SE_{ACC}=\\sqrt{\\frac{ACC\\times (1-ACC)}{Population}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ACC \\pm 1.96\\times SE_{ACC}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.30438856248221097, 0.8622781041844558)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.CI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14231876063832777"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bennett et al.'s S score (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bennett, Alpert & Goldstein’s S is a statistical measure of inter-rater agreement. It was created by Bennett et al. in 1954\n",
    "Bennett et al. suggested adjusting inter-rater reliability to accommodate the percentage of rater agreement that might be expected by chance was a better measure than simple agreement between raters.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Bennett,_Alpert,_and_Goldstein%E2%80%99s_S\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\frac{1}{|C|}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$S=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.37500000000000006"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.S"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scott's pi (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scott's pi (named after William A. Scott) is a statistic for measuring inter-rater reliability for nominal data in communication studies. Textual entities are annotated with categories by different annotators, and various measures are used to assess the extent of agreement between the annotators, one of which is Scott's pi. Since automatically annotating text is a popular problem in natural language processing, and goal is to get the computer program that is being developed to agree with the humans in the annotations it creates, assessing the extent to which humans agree with each other is important for establishing a reasonable upper limit on computer performance.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Scott%27s_Pi\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\sum_{i=1}^{|C|}(\\frac{TOP_i + P_i}{2\\times Population})^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\pi=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.34426229508196726"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gwet's AC1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AC1 was originally introduced by Gwet in 2001 (Gwet, 2001). The interpretation of AC1 is similar to generalized kappa (Fleiss, 1971), which is used to assess interrater reliability of when there are multiple raters. Gwet (2002) demonstrated that AC1 can overcome the limitations that kappa is sensitive to trait prevalence and rater's classification probabilities (i.e., marginal probabilities), whereas AC1 provides more robust measure of interrater reliability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\pi=\\frac{TOP_i + P_i}{2\\times Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\frac{1}{|C|-1}\\sum_{i=1}^{|C|}(\\pi_i\\times (1-\\pi_i))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$AC1=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3893129770992367"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.AC1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reference Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entropy of the decision problem itself as defined by the counts for the reference. The entropy of a distribution is the average negative log probability of outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Reference}=\\frac{P_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Entropy_{Reference}=-\\sum_{i=1}^{|C|}Likelihood_{Reference}(i)\\times\\log_{2}{Likelihood_{Reference}(i)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4833557549816874"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ReferenceEntropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Response Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entropy of the response distribution. The entropy of a distribution is the average negative log probability of outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Response}=\\frac{TOP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Entropy_{Response}=-\\sum_{i=1}^{|C|}Likelihood_{Response}(i)\\times\\log_{2}{Likelihood_{Response}(i)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.5"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ResponseEntropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross-entropy of the response distribution against the reference distribution. The cross-entropy is defined by the negative log probabilities of the response distribution weighted by the reference distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Reference}=\\frac{P_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Response}=\\frac{TOP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Entropy_{Cross}=-\\sum_{i=1}^{|C|}Likelihood_{Reference}(i)\\times\\log_{2}{Likelihood_{Response}(i)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.5833333333333335"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.CrossEntropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Joint Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entropy of the joint reference and response distribution as defined by the underlying matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P^{'}(i,j)=\\frac{Matrix(i,j)}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Entropy_{Joint}=-\\sum_{i=1}^{|C|}\\sum_{j=1}^{|C|}P^{'}(i,j)\\times\\log_{2}{P^{'}(i,j)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$0\\times\\log_{2}{0}\\equiv0$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.4591479170272446"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.JointEntropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entropy of the distribution of categories in the response given that the reference category was as specified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P^{'}(j|i)=\\frac{Matrix(j,i)}{P_i}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Entropy_{Conditional}=-\\sum_{j=1}^{|C|}P^{'}(j|i)\\times\\log_{2}{P^{'}(j|i)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9757921620455572"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ConditionalEntropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kullback-Liebler (KL) divergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second, expected probability distribution\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Response}=\\frac{TOP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Reference}=\\frac{P_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Divergence=-\\sum_{i=1}^{|C|}Likelihood_{Reference}\\times\\log_{2}{\\frac{Likelihood_{Reference}}{Likelihood_{Response}}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09997757835164581"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.KL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mutual Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mutual information is defined Kullback-Lieblier divergence, between the product of the individual distributions and the joint distribution.\n",
    "Mutual information is symmetric. We could also subtract the conditional entropy of the reference given the response from the reference entropy to get the same result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P^{'}(i,j)=\\frac{Matrix(i,j)}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Reference}=\\frac{P_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Likelihood_{Response}=\\frac{TOP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MI=-\\sum_{i=1}^{|C|}\\sum_{j=1}^{|C|}P^{'}(i,j)\\times\\log_{2}{\\frac{P^{'}(i,j)}{Likelihood_{Reference}(i)\\times Likelihood_{Response}(i) }}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MI=Entropy_{Response}-Entropy_{Conditional}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5242078379544428"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.MutualInformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goodman and Kruskal's lambda A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In probability theory and statistics, Goodman & Kruskal's lambda  is a measure of proportional reduction in error in cross tabulation analysis.\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Goodman_and_Kruskal%27s_lambda\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\lambda_A=\\frac{(\\sum_{j=1}^{|C|}Max(Matrix(-,j))-Max(P)}{Population-Max(P)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42857142857142855"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.LambdaA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goodman and Kruskal's lambda B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In probability theory and statistics, Goodman & Kruskal's lambda  is a measure of proportional reduction in error in cross tabulation analysis\n",
    "\n",
    "For more information visit <a href=\"https://en.wikipedia.org/wiki/Goodman_and_Kruskal%27s_lambda\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\lambda_B=\\frac{(\\sum_{i=1}^{|C|}Max(Matrix(i,-))-Max(TOP)}{Population-Max(TOP)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.16666666666666666"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.LambdaB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA1 (Strength of Agreement, Landis and Koch benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0 - 0.20</td>\n",
    "           <td style=\"text-align:center\">Slight</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.21 – 0.40</td>\n",
    "           <td style=\"text-align:center\">Fair</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.41 – 0.60</td>\n",
    "           <td style=\"text-align:center\">Moderate</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.61 – 0.80</td>\n",
    "           <td style=\"text-align:center\">Substantial</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.81 – 1.00</td>\n",
    "           <td style=\"text-align:center\">Almost perfect</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Fair'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA2 (Strength of Agreement, : Fleiss’ benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0.40 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.4 - 0.75</td>\n",
    "           <td style=\"text-align:center\">Intermediate to Good</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">More than 0.75</td>\n",
    "           <td style=\"text-align:center\">Excellent</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Poor'"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA3 (Strength of Agreement, Altman’s benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0.2 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.21 – 0.40</td>\n",
    "           <td style=\"text-align:center\">Fair</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.41 – 0.60</td>\n",
    "           <td style=\"text-align:center\">Moderate</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.61 – 0.80</td>\n",
    "           <td style=\"text-align:center\">Good</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.81 – 1.00</td>\n",
    "           <td style=\"text-align:center\">Very Good</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Fair'"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA4 (Strength of Agreement, Cicchetti’s benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0.4 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.4 – 0.59</td>\n",
    "           <td style=\"text-align:center\">Fair</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.6 – 0.74</td>\n",
    "           <td style=\"text-align:center\">Good</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.74 – 1.00</td>\n",
    "           <td style=\"text-align:center\">Excellent</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Poor'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ACC_{Overall}=\\frac{\\sum_{i=1}^{|C|}TP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACC_{Overall}=\\sum_{i=1}^{|C|}RACC_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3541666666666667"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_RACCU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACCU_{Overall}=\\sum_{i=1}^{|C|}RACCU_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3645833333333333"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_RACCU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.8.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FP_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FN_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FP_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.611111111111111"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FN_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5666666666666668"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_J"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J_{Mean}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}J_i$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J_{Sum}=\\sum_{i=1}^{|C|}J_i$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J_{Overall}=(J_{Sum},J_{Mean})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.225, 0.4083333333333334)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_J"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hamming Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hamming_loss computes the average Hamming loss or Hamming distance between two sets of samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$L_{Hamming}=\\frac{1}{Population}\\sum_{i=1}^{|P|}1(y_i \\neq \\widehat{y}_i)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.41666666666666663"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.HammingLoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zero-one Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$L_{0-1}=\\sum_{i=1}^{|P|}1(y_i \\neq \\widehat{y}_i)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ZeroOneLoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NIR (No Information Rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The no information error rate is the error rate when the input and output are independent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$NIR=\\frac{1}{Population}Max(P)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4166666666666667"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.NIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### P-Value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x=\\sum_{i=1}^{|C|}TP_{i}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p=NIR$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$n=Population$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$P-Value_{(ACC > NIR)}=1-\\sum_{i=1}^{x}\\left(\\begin{array}{c}n\\\\ i\\end{array}\\right)p^{i}(1-p)^{n-i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18926430237560654"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PValue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0    1    2    \n",
      "Actual\n",
      "0                3    0    2    \n",
      "1                0    1    1    \n",
      "2                0    2    3    \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Overall Statistics : \n",
      "\n",
      "95% CI                                                           (0.30439,0.86228)\n",
      "Bennett_S                                                        0.375\n",
      "Chi-Squared                                                      6.6\n",
      "Chi-Squared DF                                                   4\n",
      "Conditional Entropy                                              0.97579\n",
      "Cramer_V                                                         0.5244\n",
      "Cross Entropy                                                    1.58333\n",
      "Gwet_AC1                                                         0.38931\n",
      "Hamming Loss                                                     0.41667\n",
      "Joint Entropy                                                    2.45915\n",
      "KL Divergence                                                    0.09998\n",
      "Kappa                                                            0.35484\n",
      "Kappa 95% CI                                                     (-0.07708,0.78675)\n",
      "Kappa No Prevalence                                              0.16667\n",
      "Kappa Standard Error                                             0.22036\n",
      "Kappa Unbiased                                                   0.34426\n",
      "Lambda A                                                         0.42857\n",
      "Lambda B                                                         0.16667\n",
      "Mutual Information                                               0.52421\n",
      "NIR                                                              0.41667\n",
      "Overall_ACC                                                      0.58333\n",
      "Overall_J                                                        (1.225,0.40833)\n",
      "Overall_RACC                                                     0.35417\n",
      "Overall_RACCU                                                    0.36458\n",
      "P-Value                                                          0.18926\n",
      "PPV_Macro                                                        0.61111\n",
      "PPV_Micro                                                        0.58333\n",
      "Phi-Squared                                                      0.55\n",
      "Reference Entropy                                                1.48336\n",
      "Response Entropy                                                 1.5\n",
      "Scott_PI                                                         0.34426\n",
      "Standard Error                                                   0.14232\n",
      "Strength_Of_Agreement(Altman)                                    Fair\n",
      "Strength_Of_Agreement(Cicchetti)                                 Poor\n",
      "Strength_Of_Agreement(Fleiss)                                    Poor\n",
      "Strength_Of_Agreement(Landis and Koch)                           Fair\n",
      "TPR_Macro                                                        0.56667\n",
      "TPR_Micro                                                        0.58333\n",
      "Zero-one Loss                                                    5\n",
      "\n",
      "Class Statistics :\n",
      "\n",
      "Classes                                                          0                       1                       2                       \n",
      "ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 \n",
      "BM(Informedness or bookmaker informedness)                       0.6                     0.3                     0.17143                 \n",
      "DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     \n",
      "ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 \n",
      "F0.5(F0.5 score)                                                 0.88235                 0.35714                 0.51724                 \n",
      "F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 \n",
      "F2(F2 score)                                                     0.65217                 0.45455                 0.57692                 \n",
      "FDR(False discovery rate)                                        0.0                     0.66667                 0.5                     \n",
      "FN(False negative/miss/type 2 error)                             2                       1                       2                       \n",
      "FNR(Miss rate or false negative rate)                            0.4                     0.5                     0.4                     \n",
      "FOR(False omission rate)                                         0.22222                 0.11111                 0.33333                 \n",
      "FP(False positive/type 1 error/false alarm)                      0                       2                       3                       \n",
      "FPR(Fall-out or false positive rate)                             0.0                     0.2                     0.42857                 \n",
      "G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 \n",
      "J(Jaccard index)                                                 0.6                     0.25                    0.375                   \n",
      "LR+(Positive likelihood ratio)                                   None                    2.5                     1.4                     \n",
      "LR-(Negative likelihood ratio)                                   0.4                     0.625                   0.7                     \n",
      "MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 \n",
      "MK(Markedness)                                                   0.77778                 0.22222                 0.16667                 \n",
      "N(Condition negative)                                            7                       10                      7                       \n",
      "NPV(Negative predictive value)                                   0.77778                 0.88889                 0.66667                 \n",
      "P(Condition positive or support)                                 5                       2                       5                       \n",
      "POP(Population)                                                  12                      12                      12                      \n",
      "PPV(Precision or positive predictive value)                      1.0                     0.33333                 0.5                     \n",
      "PRE(Prevalence)                                                  0.41667                 0.16667                 0.41667                 \n",
      "RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 \n",
      "RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007                 \n",
      "TN(True negative/correct rejection)                              7                       8                       4                       \n",
      "TNR(Specificity or true negative rate)                           1.0                     0.8                     0.57143                 \n",
      "TON(Test outcome negative)                                       9                       9                       6                       \n",
      "TOP(Test outcome positive)                                       3                       3                       6                       \n",
      "TP(True positive/hit)                                            3                       1                       3                       \n",
      "TPR(Sensitivity, recall, hit rate, or true positive rate)        0.6                     0.5                     0.6                     \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0    1    2    \n",
      "Actual\n",
      "0                3    0    2    \n",
      "1                0    1    1    \n",
      "2                0    2    3    \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalized Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0               1               2               \n",
      "Actual\n",
      "0                0.6             0.0             0.4             \n",
      "1                0.0             0.5             0.5             \n",
      "2                0.0             0.4             0.6             \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.normalized_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall Statistics : \n",
      "\n",
      "95% CI                                                           (0.30439,0.86228)\n",
      "Bennett_S                                                        0.375\n",
      "Chi-Squared                                                      6.6\n",
      "Chi-Squared DF                                                   4\n",
      "Conditional Entropy                                              0.97579\n",
      "Cramer_V                                                         0.5244\n",
      "Cross Entropy                                                    1.58333\n",
      "Gwet_AC1                                                         0.38931\n",
      "Hamming Loss                                                     0.41667\n",
      "Joint Entropy                                                    2.45915\n",
      "KL Divergence                                                    0.09998\n",
      "Kappa                                                            0.35484\n",
      "Kappa 95% CI                                                     (-0.07708,0.78675)\n",
      "Kappa No Prevalence                                              0.16667\n",
      "Kappa Standard Error                                             0.22036\n",
      "Kappa Unbiased                                                   0.34426\n",
      "Lambda A                                                         0.42857\n",
      "Lambda B                                                         0.16667\n",
      "Mutual Information                                               0.52421\n",
      "NIR                                                              0.41667\n",
      "Overall_ACC                                                      0.58333\n",
      "Overall_J                                                        (1.225,0.40833)\n",
      "Overall_RACC                                                     0.35417\n",
      "Overall_RACCU                                                    0.36458\n",
      "P-Value                                                          0.18926\n",
      "PPV_Macro                                                        0.61111\n",
      "PPV_Micro                                                        0.58333\n",
      "Phi-Squared                                                      0.55\n",
      "Reference Entropy                                                1.48336\n",
      "Response Entropy                                                 1.5\n",
      "Scott_PI                                                         0.34426\n",
      "Standard Error                                                   0.14232\n",
      "Strength_Of_Agreement(Altman)                                    Fair\n",
      "Strength_Of_Agreement(Cicchetti)                                 Poor\n",
      "Strength_Of_Agreement(Fleiss)                                    Poor\n",
      "Strength_Of_Agreement(Landis and Koch)                           Fair\n",
      "TPR_Macro                                                        0.56667\n",
      "TPR_Micro                                                        0.58333\n",
      "Zero-one Loss                                                    5\n",
      "\n",
      "Class Statistics :\n",
      "\n",
      "Classes                                                          0                       1                       2                       \n",
      "ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 \n",
      "BM(Informedness or bookmaker informedness)                       0.6                     0.3                     0.17143                 \n",
      "DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     \n",
      "ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 \n",
      "F0.5(F0.5 score)                                                 0.88235                 0.35714                 0.51724                 \n",
      "F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 \n",
      "F2(F2 score)                                                     0.65217                 0.45455                 0.57692                 \n",
      "FDR(False discovery rate)                                        0.0                     0.66667                 0.5                     \n",
      "FN(False negative/miss/type 2 error)                             2                       1                       2                       \n",
      "FNR(Miss rate or false negative rate)                            0.4                     0.5                     0.4                     \n",
      "FOR(False omission rate)                                         0.22222                 0.11111                 0.33333                 \n",
      "FP(False positive/type 1 error/false alarm)                      0                       2                       3                       \n",
      "FPR(Fall-out or false positive rate)                             0.0                     0.2                     0.42857                 \n",
      "G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 \n",
      "J(Jaccard index)                                                 0.6                     0.25                    0.375                   \n",
      "LR+(Positive likelihood ratio)                                   None                    2.5                     1.4                     \n",
      "LR-(Negative likelihood ratio)                                   0.4                     0.625                   0.7                     \n",
      "MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 \n",
      "MK(Markedness)                                                   0.77778                 0.22222                 0.16667                 \n",
      "N(Condition negative)                                            7                       10                      7                       \n",
      "NPV(Negative predictive value)                                   0.77778                 0.88889                 0.66667                 \n",
      "P(Condition positive or support)                                 5                       2                       5                       \n",
      "POP(Population)                                                  12                      12                      12                      \n",
      "PPV(Precision or positive predictive value)                      1.0                     0.33333                 0.5                     \n",
      "PRE(Prevalence)                                                  0.41667                 0.16667                 0.41667                 \n",
      "RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 \n",
      "RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007                 \n",
      "TN(True negative/correct rejection)                              7                       8                       4                       \n",
      "TNR(Specificity or true negative rate)                           1.0                     0.8                     0.57143                 \n",
      "TON(Test outcome negative)                                       9                       9                       6                       \n",
      "TOP(Test outcome positive)                                       3                       3                       6                       \n",
      "TP(True positive/hit)                                            3                       1                       3                       \n",
      "TPR(Sensitivity, recall, hit rate, or true positive rate)        0.6                     0.5                     0.6                     \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.stat()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  cm.params() in prev versions (<0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .pycm file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'D:\\\\For Asus Laptop\\\\projects\\\\pycm\\\\Document\\\\cm1.pycm',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_stat(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.pycm'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_stat(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'D:\\\\For Asus Laptop\\\\projects\\\\pycm\\\\Document\\\\cm1.html',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_html(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.html'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_html(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'D:\\\\For Asus Laptop\\\\projects\\\\pycm\\\\Document\\\\cm1.csv',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_csv(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.csv'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_csv(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OBJ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'D:\\\\For Asus Laptop\\\\projects\\\\pycm\\\\Document\\\\cm1.obj',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_obj(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.obj'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_obj(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.9.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Vectors Must Be List\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm2=ConfusionMatrix(y_actu, 2)\n",
    "except pycmVectorError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Vectors Must Be The Same Length\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm3=ConfusionMatrix(y_actu, [1,2,3])\n",
    "except pycmVectorError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Vectors Are Empty\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm_4 = ConfusionMatrix([], [])\n",
    "except pycmVectorError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Vectors Must Be The Same Length\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm_5 = ConfusionMatrix([1,1,1,], [1,1,1,1])\n",
    "except pycmVectorError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Confusion Matrix Format Error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm3=ConfusionMatrix(matrix={})\n",
    "except pycmMatrixError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Matrix Classes Must Be Same Type\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm_4=ConfusionMatrix(matrix={1:{1:2,\"1\":2},\"1\":{1:2,\"1\":3}})\n",
    "except pycmMatrixError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number Of Classes < 2\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    cm_5=ConfusionMatrix(matrix={1:{1:2}})\n",
    "except pycmVectorError as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  updated in version 0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Example-1 (Comparison of three different classifiers)\t\n",
    "\n",
    "- [Jupyter Notebook](https://nbviewer.jupyter.org/github/sepandhaghighi/pycm/blob/master/Document/Example1.ipynb)\n",
    "- [HTML](http://www.shaghighi.ir/pycm/doc/Example1.html)\n",
    "\n",
    "### Example-2 (How to plot via matplotlib)\n",
    "\n",
    "- [Jupyter Notebook](https://nbviewer.jupyter.org/github/sepandhaghighi/pycm/blob/master/Document/Example2.ipynb)\n",
    "- [HTML](http://www.shaghighi.ir/pycm/doc/Example2.html)\n",
    "\n",
    "### Example-3 (Activation Threshold)\n",
    "\n",
    "- [Jupyter Notebook](https://nbviewer.jupyter.org/github/sepandhaghighi/pycm/blob/master/Document/Example3.ipynb)\n",
    "- [HTML](http://www.shaghighi.ir/pycm/doc/Example3.html)\n",
    "\n",
    "### Example-4 (Activation Threshold)\n",
    "\n",
    "- [Jupyter Notebook](https://nbviewer.jupyter.org/github/sepandhaghighi/pycm/blob/master/Document/Example4.ipynb)\n",
    "- [HTML](http://www.shaghighi.ir/pycm/doc/Example4.html)\n",
    "\n",
    "### Example-5 (Sample Weights)\n",
    "\n",
    "- [Jupyter Notebook](https://nbviewer.jupyter.org/github/sepandhaghighi/pycm/blob/master/Document/Example5.ipynb)\n",
    "- [HTML](http://www.shaghighi.ir/pycm/doc/Example5.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>1- J. R. Landis, G. G. Koch, “The measurement of observer agreement for categorical data. Biometrics,” in International Biometric Society,  pp. 159–174, 1977. </blockquote>\n",
    "\n",
    "<blockquote>2- D. M. W. Powers, “Evaluation: from precision, recall and f-measure to roc, informedness, markedness & correlation,” in Journal of Machine Learning Technologies, pp.37-63, 2011.</blockquote>\n",
    "\n",
    "\n",
    "<blockquote>3-  C. Sammut, G. Webb, “Encyclopedia of Machine Learning” in Springer, 2011.</blockquote>\n",
    "\n",
    "<blockquote>4- J. L. Fleiss, “Measuring nominal scale agreement among many raters,” in Psychological Bulletin, pp. 378-382. </blockquote>\n",
    "\n",
    "<blockquote>5- D.G. Altman, “Practical Statistics for Medical Research,” in Chapman and Hall, 1990.</blockquote>\n",
    "\n",
    "<blockquote>6- K. L. Gwet, “Computing inter-rater reliability and its variance in the presence of high agreement,” in The British Journal of Mathematical and Statistical Psychology, pp. 29–48, 2008.”</blockquote>\n",
    "\n",
    "<blockquote>7- W. A. Scott, “Reliability of content analysis: The case of nominal scaling,” in Public Opinion Quarterly, pp. 321–325, 1955.</blockquote>\n",
    "\n",
    "<blockquote>8- E. M. Bennett, R. Alpert, and A. C. Goldstein, “Communication through limited response questioning,” in The Public Opinion Quarterly, pp. 303–308, 1954.</blockquote>\n",
    "\n",
    "<blockquote>9- D. V. Cicchetti, \"Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology,\" in Psychological Assessment, pp. 284–290, 1994.</blockquote>\n",
    "\n",
    "<blockquote>10- R.B. Davies, \"Algorithm AS155: The Distributions of a Linear Combination of χ2 Random Variables,\" in Journal of the Royal Statistical Society, pp. 323–333, 1980.</blockquote>\n",
    "\n",
    "<blockquote>11- S. Kullback, R. A. Leibler \"On information and sufficiency,\" in Annals of Mathematical Statistics, pp. 79–86, 1951.</blockquote>\n",
    "\n",
    "<blockquote>12- L. A. Goodman, W. H. Kruskal, \"Measures of Association for Cross Classifications, IV: Simplification of Asymptotic Variances,\" in Journal of the American Statistical Association, pp. 415–421, 1972.</blockquote>\n",
    "\n",
    "<blockquote>13- L. A. Goodman, W. H. Kruskal, \"Measures of Association for Cross Classifications III: Approximate Sampling Theory,\" in Journal of the American Statistical Association, pp.  310–364, 1963. </blockquote>\n",
    "\n",
    "<blockquote>14- T. Byrt, J. Bishop and J. B. Carlin, “Bias, prevalence, and kappa,” in Journal of Clinical Epidemiology pp. 423-429, 1993.</blockquote>\n",
    "\n",
    "<blockquote>15- M. Shepperd, D. Bowes, and T. Hall, “Researcher Bias: The Use of Machine Learning in Software Defect Prediction,” in IEEE Transactions on Software Engineering, pp. 603-616, 2014.</blockquote>\n",
    "\n",
    "<blockquote>16- X. Deng, Q. Liu, Y. Deng, and S. Mahadevan, “An improved method to construct basic probability assignment based on the confusion matrix for classification problem, ” in Information Sciences, pp.250-261, 2016.</blockquote>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}