JNotebooks/tutorial07_softmax_one_hot_encoding_loss_functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Softmax activation, one-hot encoding, categorical cross-entropy and accuracy\n",
    "\n",
    "In this tutorial we will review concepts seen in previous lectures and tutorials. The learning goals are:\n",
    "- Review the softmax activation for converting a vector of scores into a vector of probabilities;\n",
    "- Review the one-hot encoding representation;\n",
    "- Contrast categorical cross-entropy and accuracy as potential loss functions. Categorical corss-entropy is better!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax\n",
    "\n",
    "The softmax function is commonly used as an activation for multi-class supervised classification in deep learning.\n",
    "The softmax function converts an input vector of real values to an output vector that can be interpreted as categorical probabilities. The elements of the output vector are in the range between 0 and 1 and they sum to 1, which are elements necessary to interpret the output of the softmax function as probabilities.\n",
    "\n",
    "The softmax is often used as the activation for the last layer of a classification model.\n",
    "\n",
    "The softmax equation for an input vector $\\overrightarrow{Z}$ is given by:\n",
    "\n",
    "\n",
    "$$ Softmax(\\overrightarrow{Z}) = \\frac{e^{\\overrightarrow{Z}}}{\\sum_{j=0}^{k-1}e^{z_j}} $$\n",
    "\n",
    "Rewriting the vector notation to make the equation simpler:\n",
    "\n",
    "$$ Softmax(\\left [  z_0 z_1 \\ldots z_{k-1} \\right ]) = \\left [ \\frac{e^{z_0}}{\\sum_{i=0}^{k-1} e^{z_i}} \\frac{e^{z_1}}{\\sum_{i=0}^{k-1} e^{z_i}} \\ldots \\frac{e^{z_{k-1}}}{\\sum_{i=0}^{k-1} e^{z_i}} \\right ] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax implementation using NumPy\n",
    "\n",
    "In this section, we define the softmax function where the input is a two-dimensional matrix, where each row represents a vector of real values (*i.e.*, $\\overrightarrow{Z}$). We will represent the two-dimensional matrix by $\\boldsymbol{Z}$. This matrix has $n$ rows and $k$ columns, which represent the number of real-valued vectors and the number of probabilities to output for each vector.\n",
    "\n",
    "$$ \\boldsymbol{Z} = z_{i,j}; i \\in (0,1,\\ldots n-1); j \\in (0,1, \\ldots k-1) $$\n",
    "\n",
    "\n",
    "$$\n",
    "Softmax(\\boldsymbol{Z}) = Softmax(\\begin{bmatrix}  \n",
    "z_{0,0} & z_{0,1} & \\ldots & z_{0,k-1} \\\\\n",
    "z_{1,0} & z_{1,1} & \\ldots & z_{1,k-1} \\\\\n",
    "\\vdots & \\vdots & & \\vdots \\\\\n",
    "z_{n-1,0} & z_{n-1,1} & \\ldots & z_{n-1,k-1} \n",
    " \\end{bmatrix}) = \n",
    "\\begin{bmatrix}\n",
    "\\frac{e^{z_{0,0}}}{\\sum_{j=0}^{k-1} e^{z_{0,j}}}& \\frac{e^{z_{0,1}}}{\\sum_{j=0}^{k-1} e^{z_{0,j}}} & \\ldots & \\frac{e^{z_{0,k-1}}}{\\sum_{j=0}^{k-1} e^{z_{0,j}}} \\\\\n",
    "\\frac{e^{z_{1,0}}}{\\sum_{j=0}^{k-1} e^{z_{1,j}}}& \\frac{e^{z_{1,1}}}{\\sum_{j=0}^{k-1} e^{z_{1,j}}} & \\ldots & \\frac{e^{z_{1,k-1}}}{\\sum_{j=0}^{k-1} e^{z_{1,j}}} \\\\\n",
    "\\vdots & \\vdots & & \\vdots \\\\\n",
    "\\frac{e^{z_{n-1,0}}}{\\sum_{j=0}^{k-1} e^{z_{n-1,j}}}& \\frac{e^{z_{n-1,1}}}{\\sum_{j=0}^{k-1} e^{z_{n-1,j}}} & \\ldots & \\frac{e^{z_{n-1,k-1}}}{\\sum_{j=0}^{k-1} e^{z_{n-1,j}}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import matplotlib.pylab as plt\n",
    "np.set_printoptions(suppress=True, precision=3) # Limits the number of decimal houses when printing values to 3\n",
    "import ipywidgets as widgets # for cells with interactivity\n",
    "from tensorflow.keras.utils import to_categorical # Function to convert labels to one-hot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(Z):\n",
    "    EZ = np.exp(Z)\n",
    "    S = EZ / EZ.sum(axis=1,keepdims = True)\n",
    "    return S"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test the code on a case where n = 10 and k = 3! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z=\n",
      " [[-2.     1.     0.2  ]\n",
      " [-1.111  1.     0.2  ]\n",
      " [-0.222  1.     0.2  ]\n",
      " [ 0.667  1.     0.2  ]\n",
      " [ 1.556  1.     0.2  ]\n",
      " [ 2.444  1.     0.2  ]\n",
      " [ 3.333  1.     0.2  ]\n",
      " [ 4.222  1.     0.2  ]\n",
      " [ 5.111  1.     0.2  ]\n",
      " [ 6.     1.     0.2  ]]\n",
      "S=\n",
      " [[0.033 0.667 0.3  ]\n",
      " [0.077 0.637 0.286]\n",
      " [0.169 0.573 0.258]\n",
      " [0.331 0.462 0.207]\n",
      " [0.546 0.313 0.141]\n",
      " [0.745 0.176 0.079]\n",
      " [0.877 0.085 0.038]\n",
      " [0.945 0.038 0.017]\n",
      " [0.977 0.016 0.007]\n",
      " [0.99  0.007 0.003]]\n"
     ]
    }
   ],
   "source": [
    "aux = np.linspace(-2, 6.0,10).reshape(-1,1)\n",
    "Z = np.hstack([aux, np.ones_like(aux), 0.2 * np.ones_like(aux)]) # we are keeping two values constant and \n",
    "                                                                 # changing the thirs one\n",
    "S = softmax(Z)\n",
    "print('Z=\\n',Z)\n",
    "print('S=\\n',S)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example below, we will change the values of $\\overrightarrow{Z}$ interactively. The interactivety is generated using the ipywidgets Python module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24cd9928e26949d69cd63a6de21833d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(FloatSlider(value=5.0, description='s1', max=10.0, min=1.0), FloatSlider(value=5.0, desc…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.plotmodel(s1, s2, s3)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def plotmodel(s1,s2,s3):\n",
    "    scores = np.array([[s1, s2, s3]]) # shape: (1,3) \n",
    "    S = softmax(scores)[0] # (1,3) array as (3,) array\n",
    "    plt.rcdefaults()\n",
    "    fig, ax = plt.subplots(figsize=(3, 2))\n",
    "    classes = ('0', '1', '2')\n",
    "    x_pos = [2,4,6]\n",
    "    ax.bar(x_pos, S, align='center',color='green', ecolor='black')\n",
    "    ax.set_xticks(x_pos)\n",
    "    ax.set_xticklabels(classes)\n",
    "    ax.set_ylim([0,1])\n",
    "    ax.set_xlabel('$\\overrightarrow{Z}$')\n",
    "    ax.set_ylabel('Softmax')\n",
    "    plt.show()\n",
    "                       \n",
    "widgets.interact(plotmodel,s1 = (1,10,.1),s2 = (1,10,.1),s3 = (1,10,.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One hot encoding\n",
    "\n",
    "One hot encoding represents categorical data as a list of binary values with one element in the list for each possible category. The name \"one hot\" comes from the fact that only one binary element is set to 1 (hot) at a time and all other elements are set to 0 (cold).\n",
    "\n",
    "Most deep learning algorithms cannot work with categorical data directly. The categories need to be converted into numerical representations. This is true for both the input ($X$) and output ($\\widehat{Y}$) of our models.\n",
    "\n",
    "Let's think about this class garbage classification assignment. There are three classes: \"green\", \"blue\" and \"black\" garbage bins. We often encode these classes by assigning an integer label without even giving too much thought about it:\n",
    "\n",
    "- \"green\" - class 0\n",
    "- \"bllue\" - class 1\n",
    "- \"black\" - class 2\n",
    "\n",
    "This label assignment is called label encoding. Label encoding can be a proper representation if there is a natural ordering relationship between the categories. In the example of garbage classiifcation, where there is no clear ordering, label encoding is not a good strategy. One example of categorical data that has an ordering relationship is the [Likert scale](https://en.wikipedia.org/wiki/Likert_scale), which is split into five categories that clearly have an ordering relationship among them: \"Like\", \"Like Somewhat\",\t\"Neutral\", \t\"Dislike Somewhat\", \t\"Dislike\".\n",
    "\n",
    "When our data does not have an ordering relationship, we employ one hot encoding. \n",
    "\n",
    "The code snippet below show how to get the one hot encoding representation from a list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>blue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>blue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>green</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>black</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>green</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>black</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>black</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>green</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     bin\n",
       "0   blue\n",
       "1   blue\n",
       "2  green\n",
       "3  black\n",
       "4  green\n",
       "5  black\n",
       "6  black\n",
       "7  green"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.DataFrame({\"bin\": [\"blue\",\"blue\",\"green\",\"black\",\"green\",\"black\",\"black\",\"green\"]})\n",
    "s.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bin</th>\n",
       "      <th>bin_black</th>\n",
       "      <th>bin_blue</th>\n",
       "      <th>bin_green</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>blue</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>blue</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>green</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>black</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>green</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>black</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>black</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>green</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     bin  bin_black  bin_blue  bin_green\n",
       "0   blue          0         1          0\n",
       "1   blue          0         1          0\n",
       "2  green          0         0          1\n",
       "3  black          1         0          0\n",
       "4  green          0         0          1\n",
       "5  black          1         0          0\n",
       "6  black          1         0          0\n",
       "7  green          0         0          1"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot = pd.get_dummies(s) # Get one-hot encoding of variable\n",
    "# Join the one hot encoding to the data frame\n",
    "s2 = s.join(one_hot)\n",
    "s2.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bin_black</th>\n",
       "      <th>bin_blue</th>\n",
       "      <th>bin_gree</th>\n",
       "      <th>bin_green</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   bin_black  bin_blue  bin_gree  bin_green\n",
       "0          0         1         0          0\n",
       "1          0         1         0          0\n",
       "2          0         0         1          0\n",
       "3          1         0         0          0\n",
       "4          0         0         0          1\n",
       "5          1         0         0          0\n",
       "6          1         0         0          0\n",
       "7          0         0         0          1"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3 = s2.drop('bin',axis = 1) # Drop the bin column \n",
    "s3.head(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code snippet below show how to get the one hot encoding representation from an array of integers using the keras [to_categorical](https://keras.io/api/utils/python_utils/#to_categorical-function)\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Y=\n",
      "[0 0 1 0 2 1 1]\n",
      "\n",
      "Yoh=\n",
      "[[1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "Y = np.array([0, 0, 1, 0, 2 ,1 ,1])\n",
    "Yoh = to_categorical(Y)\n",
    "\n",
    "print('Y=')\n",
    "print(Y)\n",
    "print('\\nYoh=')\n",
    "print(Yoh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To go back from one hot encoding to label encoding, you just need to use the [numpy argmax](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html) funcation across the columns of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Y2=\n",
      "[0 0 1 0 2 1 1]\n",
      "\n",
      "Y = Y2?\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "Y2 = np.argmax(Yoh, axis = 1)\n",
    "print('\\nY2=')\n",
    "print(Y2)\n",
    "print(\"\\nY = Y2?\")\n",
    "print(np.all(Y == Y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical cross-entropy and Accuracy\n",
    "\n",
    "Cross-entropy builds upon the idea of entropy from information theory. Cross-entropy essentially calculates the number of bits required to represent an average event from one distribution compared to another distribution. In our case, one distribution is the ground-truth distribution represented by the labels of our data and the other distribution is represented by the probabilities that our model outputs (*i.e.*, softmax output of the final layer). The categorical cross-entropy (CCE) is computed by the following equation:\n",
    "\n",
    "$$CCE[Y_{oh},\\widehat{Y_{oh}}] = -\\frac{1}{N}\\sum_{i=0}^{N-1}\\sum_{j=0}^{k-1}Y_{oh}[i,j]log(\\widehat{Y_{oh}[i,j]})$$\n",
    "\n",
    "Another imortant metric is accuracy, which is a metric that ranges between 0 and 1. Zero meaning that the model classified all samples incorrectly and one meaning the model classified all sample perfectly. The accuracy is computed by the following euqation:\n",
    "\n",
    "$$accuracy = \\frac{samples \\quad classified \\quad correctly}{total \\quad number \\quad  of \\quad samples}$$\n",
    "\n",
    "Accuracy values are a lot easier to understand and interpret than CCE values. So, why we use CCE most of the time for training our models?\n",
    "\n",
    "**Important comment**: If accuracy was used as the loss function of our model, our goal would be to maximize it. In the case of CCE, we want to minimize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cce(Yoh,Yoh_pred):\n",
    "    cce = (-Yoh*np.log(Yoh_pred)).mean()\n",
    "    return cce\n",
    "\n",
    "def compute_accuracy(Yoh,Yoh_pred):\n",
    "    Y = np.argmax(Yoh, axis = 1)\n",
    "    Ypred = np.argmax(Yoh_pred, axis = 1)\n",
    "    accuracy = (Y == Ypred).sum()/Y.size\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labels\n",
    "Yoh = np.array([[0, 0, 1],\\\n",
    "                [1, 0, 0],\\\n",
    "                [0, 1, 0]])\n",
    "\n",
    "# Confident predictions\n",
    "Yoh_pred = np.array([[0.01, 0.02, 0.97],\\\n",
    "                     [0.94, 0.03, 0.03],\\\n",
    "                     [0.02, 0.95, 0.03]])\n",
    "\n",
    "# Low confidence predictions\n",
    "Yoh_pred2 = np.array([[0.33, 0.33, 0.34],\\\n",
    "                     [0.40, 0.30, 0.30],\\\n",
    "                     [0.31, 0.36, 0.33]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confident predictions case\n",
      "CCE:\n",
      "0.015958656176705183\n",
      "Accuracy:\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Confident predictions case\")\n",
    "print(\"CCE:\")\n",
    "print(compute_cce(Yoh,Yoh_pred))\n",
    "print(\"Accuracy:\")\n",
    "print(compute_accuracy(Yoh,Yoh_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Low confidence predictions case\n",
      "CCE:\n",
      "0.3351946267531185\n",
      "Accuracy:\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Low confidence predictions case\")\n",
    "print(\"CCE:\")\n",
    "print(compute_cce(Yoh,Yoh_pred2))\n",
    "print(\"Accuracy:\")\n",
    "print(compute_accuracy(Yoh,Yoh_pred2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both the high and low confidence prediction cases, the accuracy was 1. On the other hand, the CCE achieved a considrable smaller value for predictions with high confidence compared to prediction with low confidence. We want to have confident preictions and that is one of the many reasons why we prefer using CCE than accuracy as a loss function for training our models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Suggested References\n",
    "\n",
    "- https://gombru.github.io/2018/05/23/cross_entropy_loss/\n",
    "- http://www.jussihuotari.com/2018/01/17/why-loss-and-accuracy-metrics-conflict/"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "widgets": {
   "state": {
    "9756cb94952a4ebab87dfd2cc403cb05": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "b531692a126f4adba72e663a0a1ae8fc": {
     "views": [
      {
       "cell_index": 8
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}