semantic_segmentation/Dice_Coefficient_Metric.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dice Coefficient Metric\n",
    "\n",
    "In this notebook, we will discuss the **Intersection over Union-like metric implementation for Semantic\n",
    "Segmentation** task, which is commonly known as the **Dice coefficient**, also called the **Sørensen-Dice coefficient**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Dice Coefficient\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "The **`Sørensen-Dice coefficient`** is applied for the pixel-wise comparison between a model prediction (segmented input) and the ground truth. There are several formulas for coefficient computation: the original was based on the\n",
    "cardinality of two sets, for example, $A$ and $B$: $$\\frac{2|A\\cap B|}{|A|+|B|},$$ where $|A|$, $|B|$ are the cardinalities of the $A$, $B$ sets accordingly.\n",
    "\n",
    "---\n",
    "\n",
    "<img src='https://www.learnopencv.com/wp-content/uploads/2020/04/c3-w11-dice-coeff.png'>\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "In the current notebook we will use another coefficient interpretation for boolean data based of `true positive`\n",
    "($TP$), `false positive` ($FP$) and `false negative` ($FN$) values: $$\\frac{2\\cdot TP}{2\\cdot TP+FP+FN},$$ where:\n",
    "\n",
    "- `true positives` ($TP$) - the cases when the model properly predicted positive class.\n",
    "- `false positives` ($FP$)- the opposite to $TP$ outcomes, when the positive class was incorrectly predicted.\n",
    "- `false negative` ($FN$) - the cases of improperly predicted negative classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Implementation\n",
    "\n",
    "We can see that we have to find $TP$, $FP$, and $FN$. Recall the [confusion matrix for classification](https://courses.opencv.org/api/jupyter/render_notebook/?url=https%3A%2F%2Fwww.dropbox.com%2Fs%2Fwa865zr73ddpd7j%2FClassification_Evaluation_Metrics.ipynb%3Fdl%3D1&images_url=#4.-Confusion-Matrix), where we have implemented a confusion matrix for two classes. We know that the confusion matrix has all these values. So to implement the dice coefficient, first, we implement the confusion matrix for `n-classes`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "from dataclasses import dataclass\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Configuration for reproducible results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class SystemConfig:\n",
    "    seed: int = 42  # seed number to set the state of all random number generators\n",
    "    cudnn_benchmark_enabled: bool = False  # enable CuDNN benchmark for the sake of performance\n",
    "    cudnn_deterministic: bool = True  # make cudnn deterministic (reproducible training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def setup_system(system_config: SystemConfig) -> None:\n",
    "    torch.manual_seed(system_config.seed)\n",
    "    np.random.seed(system_config.seed)\n",
    "    random.seed(system_config.seed)\n",
    "    torch.set_printoptions(precision=10)\n",
    "    if torch.cuda.is_available():\n",
    "        torch.cuda.manual_seed_all(system_config.seed)\n",
    "        torch.backends.cudnn_benchmark_enabled = system_config.cudnn_benchmark_enabled\n",
    "        torch.backends.cudnn.deterministic = system_config.cudnn_deterministic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply system settings\n",
    "setup_system(SystemConfig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Confusion Matrix\n",
    "\n",
    "**It calculates the confusion matrix for n-classes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConfusionMatrix:\n",
    "    \"\"\" Implementation of Confusion Matrix.g\n",
    "\n",
    "    Arguments:\n",
    "        num_classes (int): number of evaluated classes.\n",
    "        normalized (bool): if normalized is True then confusion matrix will be normalized.\n",
    "    \"\"\"\n",
    "    def __init__(self, num_classes, normalized=False):\n",
    "        self.num_classes = num_classes\n",
    "        self.normalized = normalized\n",
    "        self.conf = np.ndarray((num_classes, num_classes), np.int32)\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\" Reset of the Confusion Matrix.\n",
    "        \"\"\"\n",
    "        self.conf.fill(0)\n",
    "\n",
    "    def add(self, pred, target):\n",
    "        \"\"\" Add sample to the Confusion Matrix.\n",
    "\n",
    "        Arguments:\n",
    "            pred (torch.Tensor() or numpy.ndarray): predicted mask.\n",
    "            target (torch.Tensor() or numpy.ndarray): ground-truth mask.\n",
    "        \"\"\"\n",
    "        if torch.is_tensor(pred):\n",
    "            pred = pred.detach().cpu().numpy()\n",
    "        if torch.is_tensor(target):\n",
    "            target = target.detach().cpu().numpy()\n",
    "\n",
    "        valid_indices = np.where((target >= 0) & (target < self.num_classes))\n",
    "        pred = pred[valid_indices]\n",
    "        target = target[valid_indices]\n",
    "\n",
    "        replace_indices = np.vstack((target.flatten(), pred.flatten())).T\n",
    "\n",
    "        conf, _ = np.histogramdd(\n",
    "            replace_indices,\n",
    "            bins=(self.num_classes, self.num_classes),\n",
    "            range=[(0, self.num_classes), (0, self.num_classes)]\n",
    "        )\n",
    "\n",
    "        self.conf += conf.astype(np.int32)\n",
    "\n",
    "    def value(self):\n",
    "        \"\"\" Return of the Confusion Matrix.\n",
    "\n",
    "        Returns:\n",
    "            numpy.ndarray(num_classes, num_classes): confusion matrix.\n",
    "        \"\"\"\n",
    "        if self.normalized:\n",
    "            conf = self.conf.astype(np.float32)\n",
    "            return conf / conf.sum(1).clip(min=1e-12)[:, None]\n",
    "        return self.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Input\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a toy data of batch_size = 1\n",
    "# let's our model predicting segments of the three classes\n",
    "\n",
    "# input data for batch size 1\n",
    "\n",
    "# create ground truth data\n",
    "ground_truth = torch.zeros(1, 224, 224)\n",
    "\n",
    "# class 1 label\n",
    "ground_truth[:, 50:100, 50:100] = 1\n",
    "\n",
    "# class 2 label\n",
    "ground_truth[:, 50:150, 150:200] = 2\n",
    "\n",
    "# generate torch tensor to check the solution\n",
    "prediction_prob = torch.zeros(1, 3, 224, 224).uniform_().softmax(dim=1)\n",
    "\n",
    "\n",
    "class_prediction = prediction_prob.argmax(dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate confusion matrix without normalize (`normalized=False`)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[14090, 14265, 14321],\n",
       "       [  820,   863,   817],\n",
       "       [ 1667,  1711,  1622]], dtype=int32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate confusion matrix without normalize\n",
    "\n",
    "conf_mat = ConfusionMatrix(num_classes=3, normalized=False)\n",
    "\n",
    "conf_mat.add(pred=class_prediction, target=ground_truth)\n",
    "\n",
    "conf_mat.value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate confusion matrix with normalize (`normalized=True`)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.33016214, 0.33426282, 0.33557504],\n",
       "       [0.328     , 0.3452    , 0.3268    ],\n",
       "       [0.3334    , 0.3422    , 0.3244    ]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate confusion matrix with normalize\n",
    "\n",
    "conf_mat = ConfusionMatrix(num_classes=3, normalized=True)\n",
    "\n",
    "conf_mat.add(pred=class_prediction, target=ground_truth)\n",
    "\n",
    "conf_mat.value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Dice Coefficient\n",
    "\n",
    "Before implement Class for the dice coefficient (`DiceSolution`), let's implement a helper class (`ConfusionMatrixBasedMetric`).\n",
    "\n",
    "\n",
    "The helper class will use the confusion matrix class. Additionally, it will do the following:\n",
    "\n",
    "- It will be to take the probability prediction as well as class prediction.\n",
    "\n",
    "\n",
    "- Sometimes we want to ignore a few classes at the time of calculating the dice coefficient. So it will prepare a list of those classes that will be ignored at the time of calculating the dice coefficient. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConfusionMatrixBasedMetric:\n",
    "    \"\"\" Implementation of base class for Confusion Matrix based metrics.\n",
    "\n",
    "    Arguments:\n",
    "        num_classes (int): number of evaluated classes.\n",
    "        reduced_probs (bool): if True then argmax was applied to input predicts.\n",
    "        normalized (bool): if normalized is True then confusion matrix will be normalized.\n",
    "        ignore_indices (int or iterable): list of ignored classes index.\n",
    "    \"\"\"\n",
    "    def __init__(self, num_classes, reduced_probs=False, normalized=False, ignore_indices=None):\n",
    "        self.conf_matrix = ConfusionMatrix(num_classes=num_classes, normalized=normalized)\n",
    "        self.reduced_probs = reduced_probs\n",
    "\n",
    "        if ignore_indices is None:\n",
    "            self.ignore_indices = None\n",
    "        elif isinstance(ignore_indices, int):\n",
    "            self.ignore_indices = (ignore_indices, )\n",
    "        else:\n",
    "            try:\n",
    "                self.ignore_indices = tuple(ignore_indices)\n",
    "            except TypeError:\n",
    "                raise ValueError(\"'ignore_indices' must be an int or iterable\")\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\" Reset of the Confusion Matrix\n",
    "        \"\"\"\n",
    "        self.conf_matrix.reset()\n",
    "\n",
    "    def add(self, pred, target):\n",
    "        \"\"\" Add sample to the Confusion Matrix.\n",
    "\n",
    "        Arguments:\n",
    "            pred (torch.Tensor() or numpy.ndarray): predicted mask.\n",
    "            target (torch.Tensor() or numpy.ndarray): ground-truth mask.\n",
    "        \"\"\"\n",
    "        if not self.reduced_probs:\n",
    "            pred = pred.argmax(dim=1)\n",
    "        self.conf_matrix.add(pred, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's implement a class for the Dice coefficient.**\n",
    "\n",
    "**The class will implement a mean dice coefficient as well as the dice coefficient for each class.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DiceCoefficient class inherited from ConfusionMatrixBasedMetric\n",
    "class DiceCoefficient(ConfusionMatrixBasedMetric):\n",
    "    \"\"\" Correct implementation of the Dice metric.\n",
    "\n",
    "    Arguments:\n",
    "        num_classes (int): number of evaluated classes.\n",
    "        reduced_probs (bool): if True then argmax was applied to input predicts.\n",
    "        normalized (bool): if normalized is True then confusion matrix will be normalized.\n",
    "        ignore_indices (int or iterable): list of ignored classes indices.\n",
    "    \"\"\"\n",
    "\n",
    "    # the core coefficient computation method\n",
    "    def value(self):\n",
    "        \"\"\" Return of the mean Dice and Dice per class.\n",
    "\n",
    "        Returns:\n",
    "            mdice (float32): mean dice.\n",
    "            dice (list): list of dice coefficients per class.\n",
    "        \"\"\"\n",
    "        # get confusion matrix value\n",
    "        conf_matrix = self.conf_matrix.value()\n",
    "\n",
    "        # check whether the list of indices to ignore is empty\n",
    "        if self.ignore_indices is not None:\n",
    "            # set column values of ignore classes to 0\n",
    "            conf_matrix[:, self.ignore_indices] = 0\n",
    "            # set row values of ignore classes to 0\n",
    "            conf_matrix[self.ignore_indices, :] = 0\n",
    "\n",
    "        # get TP, FP and FN values for Dice calculation using confusion matrix\n",
    "        true_positive = np.diag(conf_matrix)\n",
    "        false_positive = np.sum(conf_matrix, 0) - true_positive\n",
    "        false_negative = np.sum(conf_matrix, 1) - true_positive\n",
    "\n",
    "        # use errstate to handle the case of zero denominator value\n",
    "        with np.errstate(divide='ignore', invalid='ignore'):\n",
    "            # calculate dice by its formula\n",
    "            dice = 2 * true_positive / (2 * true_positive + false_positive + false_negative)\n",
    "\n",
    "        # check whether the list of indices to ignore is empty\n",
    "        if self.ignore_indices is not None:\n",
    "            # exclude ignore indices\n",
    "            dice_valid_cls = np.delete(dice, self.ignore_indices)\n",
    "            # get mean class dice coefficient ignoring NaN values\n",
    "            mdice = np.nanmean(dice_valid_cls)\n",
    "        else:\n",
    "            # get mean class dice coefficient ignoring NaN values\n",
    "            mdice = np.nanmean(dice)\n",
    "\n",
    "        return mdice, dice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate the dice coefficient with class label prediction (`reduced_probs=True`)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mdice: 0.2379727729935648\n",
      "dice: [0.47558773 0.0892497  0.14908088]\n"
     ]
    }
   ],
   "source": [
    "dice_coeff = DiceCoefficient(num_classes=3, \n",
    "                             reduced_probs=True, \n",
    "                             normalized=False, \n",
    "                             ignore_indices=None)\n",
    "\n",
    "\n",
    "# reduced_probs=True, means we have to give class prediction as target\n",
    "dice_coeff.add(pred=class_prediction, target=ground_truth)\n",
    "\n",
    "mdice, dice = dice_coeff.value()\n",
    "\n",
    "print('mdice: {}'.format(mdice))\n",
    "print('dice: {}'.format(dice))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate the dice coefficient with class probabilities prediction (`reduced_probs=False`)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mdice: 0.2379727729935648\n",
      "dice: [0.47558773 0.0892497  0.14908088]\n"
     ]
    }
   ],
   "source": [
    "dice_coeff = DiceCoefficient(num_classes=3, \n",
    "                             reduced_probs=False, \n",
    "                             normalized=False, \n",
    "                             ignore_indices=None)\n",
    "\n",
    "\n",
    "# reduced_probs=False, means we have to give prediction probability as target\n",
    "dice_coeff.add(pred=prediction_prob, target=ground_truth)\n",
    "\n",
    "mdice, dice = dice_coeff.value()\n",
    "\n",
    "print('mdice: {}'.format(mdice))\n",
    "print('dice: {}'.format(dice))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate the dice coefficient with ignored class (`ignore_indices=(0,)`)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mdice: 0.4838796700573852\n",
      "dice: [       nan 0.40573578 0.56202356]\n"
     ]
    }
   ],
   "source": [
    "dice_coeff = DiceCoefficient(num_classes=3, \n",
    "                             reduced_probs=True, \n",
    "                             normalized=False, \n",
    "                             ignore_indices=(0,))\n",
    "\n",
    "\n",
    "# reduced_probs=True, means we have to give class prediction as target\n",
    "# ignore_indices=(0,) \n",
    "dice_coeff.add(pred=class_prediction, target=ground_truth)\n",
    "\n",
    "mdice, dice = dice_coeff.value()\n",
    "\n",
    "print('mdice: {}'.format(mdice))\n",
    "print('dice: {}'.format(dice))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}