reader/2.5_CAD_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 2: Computer-aided diagnosis\n",
    "\n",
    "**Contents:** <br>\n",
    "\n",
    "- [Goal](#goal)<br>\n",
    "- [Deliverables](#deliverables)<br>\n",
    "- [Assessment](#assessment)<br>\n",
    "\n",
    "- [Guided project work](#guided_work)<br>\n",
    "\n",
    "    A. [Linear regression for nuclei area measurement](#section_1)<br>\n",
    "    \n",
    "    B. [Logistic regression for nuclei classification](#section_2)<br>\n",
    "\n",
    "    C. [Neural network training for nuclei classification](#section_3)<br>\n",
    "\n",
    "    D. [Using k-NN for nuclei classification](#section_4)<br>\n",
    "        \n",
    "\n",
    "<div id='references'></div>\n",
    "\n",
    "**References:**<br>\n",
    "\n",
    "[1] Veta M., van Diest P.J., Pluim J.P.W. 2016. Cutting Out the Middleman: Measuring Nuclear Area in Histopathology Slides Without Segmentation. Medical Image Computing and Computer-Assisted Intervention. [LINK](https://www.doi.org/10.1007/978-3-319-46723-8_73)\n",
    "\n",
    "[2] Graham, S., Vu, Q. D., Raza, S. E. A., Azam, A., Tsang, Y. W., Kwak, J. T., & Rajpoot, N. 2019. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58, 101563. [LINK](https://doi.org/10.1016/j.media.2019.101563)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id='goal'></div>\n",
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/read_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "**Clarification:** Following the guided project work below, completing the programming tasks and answering the theory questions, is the minimal solution to this project. If accompanied by a suitable report, this will be graded with a ‘sufficient’ grade.\n",
    "\n",
    "To achieve higher grades, you are expected to go beyond the minimal solution. You should use what you have implemented and the available data to come up with and answer a suitable research question. Write about this in your report.\n",
    "\n",
    "## Goal\n",
    "Implement and apply linear regression, logistic regression, a neural network and $k$-NN for classifying nuclei size in histopathology images, and evaluate and analyze the results.\n",
    "\n",
    "The size of the cell nuclei of the tumor in breast cancer patients can be indicative of the outcome. Large nuclei size indicates more aggressive tumor and in turn worse prognosis for the patient. As part of their routine work, pathologists make qualitative evaluation of the size of the nuclei by examining the tissue under a microscope. Quantitative measurement (e.g. by manual segmentation) is a much better solution, however, it is unfeasible as it takes additional time away from the busy pathologists. A solution to this problem is to develop an automatic method for measurement of nuclei area.\n",
    "\n",
    "All data required for this mini-project is provided with the code handout. In the exercises, you applied regression and classification methods on toy datasets, and in the project work you will apply the same methods to a dataset of RGB images of nuclei with size $24\\times24$ pixels. The images originate from the dataset that was previously described in [Veta et al. (2015)](#references). \n",
    "\n",
    "<div id='deliverables'></div>\n",
    "\n",
    "## Deliverables\n",
    "There is no hard limit for the length of the report, however, concise and short reports are **strongly** encouraged. Aim to present your most important findings in the main body of the report and (if needed) any additional information in an appendix. The following report structure is suggested for the main body of the report:\n",
    "\n",
    "1. Introduction\n",
    "2. Methods\n",
    "3. Results\n",
    "4. Discussion\n",
    "\n",
    "The introduction and result sections can be very brief in this case (e.g. half a page each). The discussion section should contain the analysis of the results.\n",
    "\n",
    "The report must be submitted as a single PDF file. The code must be submitted as a single archive file (e.g. zip) that is self-contained and can be used to reproduce the results in the report. \n",
    "\n",
    "Note that there is not a single correct solution for the project. You have to demonstrate to the reader that you understand the methods that you have studied and can critically analyze the results of applying the methods. Below, you can find a set of assignments (guided project work) that will help you get started with the project work and when correctly completed will present you with a **minimal solution**. Solutions which go beyond these assignments are of course encouraged. \n",
    "\n",
    "Code and a report describing your implementation, results and analysis. \n",
    "\n",
    "Your report must contain a paragraph describing the contributions of each group member which can be used to to adjust individual grades to reflect a student’s (lack of) contribution to the group\n",
    "\n",
    "\n",
    "## Assessment\n",
    "The rubric that will be used for assessment of the project work is given in [this table](https://github.com/tueimage/8dc00-mia/blob/master/rubric.md)\n",
    "\n",
    "<div id='assessment'></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id='guided_work'></div>\n",
    "\n",
    "## Guided project work\n",
    "\n",
    "<div id='section_1'></div>\n",
    "\n",
    "### A. Linear regression for nuclei area measurement\n",
    "\n",
    "The Python function `nuclei_measurement()` implements training of a linear regression model for measuring the area of nuclei in microscopy images. The dataset for this problem consists of small RGB images of size $24 \\times 24$ pixels with a nucleus in the center. Such images can be obtained, for example, by cropping from larger images after performing a nuclei detection step. The targets are the areas of the  nucleus in the center of the image obtained by manual measurement. The linear regression model that we are going to train will enable us to automatically measure the size of new, previously unseen samples (without resorting to manual measurement).\n",
    "\n",
    "The first section of code loads and prepares the dataset. The data is already split into a training and testing set, each containing more than $20,000$ samples (a validation dataset is not needed as we are not going to perform model selection, i.e. we are going to stick to linear regression). The last few lines of the first section of code visualize the $300$ smallest and $300$ largest nuclei in the training dataset.\n",
    "\n",
    "In this example, we are not going to perform feature extraction but use the raw pixel values as features. Since each sample is an RGB image with size $24 \\times 24$ pixels, we end up with $24 \\times 24 \\times 3=1728$ features. Locate the code that reshapes each image into a feature vector and make sure you understand how it works. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import sys\n",
    "sys.path.append(\"../code\")\n",
    "from cad_project import nuclei_measurement\n",
    "\n",
    "nuclei_measurement()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/todo_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "### *Task 1*: \n",
    "Implement the missing functionality for training a linear regression model for automatic measurement of the nuclei area. Evaluate the performance on the independent test dataset. The next lines of code plot the predicted vs. the actual area. What is your analysis of the results shown in the plot?\n",
    "\n",
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/todo_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "### *Task 2*: \n",
    "Train a new linear regression model with a reduced number of training samples. Which model results in larger error on the testing set and why? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id='section_2'></div>\n",
    "\n",
    "### B. Logistic regression for nuclei classification\n",
    "\n",
    "The Python function `nuclei_classification()` implements the training of a logistic regression model that classifies nuclei into the classes \"large\" (class label $y = 1$) and \"small\" (class label $y = 0$). Examine the code and comments and make sure that you understand what it does. One notable difference from before is that this code uses the analytical expression for the gradient of the loss function, instead of computing it numerically with `ngradient` as before. Using `ngradient` will also work, but is much slower. The script is mostly complete. The only missing component is the values for the parameters of the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import sys\n",
    "sys.path.append(\"../code\")\n",
    "from cad_project import nuclei_classification\n",
    "from IPython.display import display, clear_output\n",
    "\n",
    "nuclei_classification()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/todo_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "### *Task 3*: \n",
    "\n",
    "1. Select values for the learning rate, batch size and number of iterations (these are sometimes called hyper-parameters of the model), as well as initial values for the model parameters that will result in fast training of an accurate model for this classification problem. Note that if you don't choose the hyper-parameters and initial parameters well, the resulting loss might be out of range of the plot.\n",
    "    \n",
    "    Experiment with a few variations of the parameters and analyze and compare the resulting loss curves. Describe how the different hyper-parameters influence the training process.\n",
    "    \n",
    "2. Instead of running gradient descent for a fixed number of iterations, can you propose a stopping criterion for the training?\n",
    "\n",
    "3. Report the classification accuracy for your best trained model.\n",
    "\n",
    "4. Reduce the size of the training set by a very large factor (e.g. 0.5% of the original number of samples). Train the model with this reduced number of samples. Does the model overfit the training dataset? How did you come to this conclusion?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id='section_3'></div>\n",
    "\n",
    "### C. Neural network training for nuclei classification\n",
    "\n",
    "The code in SECTION 3 of the `cad_test.py` file, enabled you to train a neural network that classifies whether the input image contains a large or small nuclei. During the training of this network the hyper parameters (i.e. batch size, learning rate, number of iterations, ...) have been kept fixed.\n",
    "However, these parameters can be a crucial factor in optimizing your model results. Therefore we are going to investigate the effect of these parameters on the loss curve behaviour and accuracy of the model. To put the improvements in perspective, we are also going to compare these results to the Logistic Regression model (see Task 4.1).\n",
    "\n",
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/todo_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "### *Task 4*:\n",
    "\n",
    "1. Select values for the learning rate, batch size and number of iterations similar to those used in Task 4. Analyze and compare the resulting loss curves with the loss curves obtained from the logistic regression. What do you observe? Also, report the accuracy of the methods you compare. Is there a best one?\n",
    "2. Now, with a fixed set of hyper parameters, try to change the size of the hidden layer and report the obtained loss curves. How do the number of parameters of the model influence the loss curve? Can you think of an appropriate number of parameters for this dataset?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id='section_4'></div>\n",
    "\n",
    "### D. Using $k$-NN for nuclei classification\n",
    "In Notebook 2.1 we have introduced the $k$-NN algorithm, in Notebook 2.4 the PCA method was introduced. In this task we would like to combine these two to create a new classifier.\n",
    "How does a clustering algorithm hold up against the algorithm you have implemented thus far?\n",
    "\n",
    "<div style=\"float:right;margin:-5px 5px\"><img src=\"../reader/assets/todo_ico.png\" width=\"42\" height=\"42\"></div> \n",
    "\n",
    "### *Task 5*:\n",
    "1. Use the training set of the cell nuceli data for the $k$-NN algorithm. Next, use the test set to classify the images with this algorithm. How did you choose the parameter $k$? What happens to the accuracy when you first use PCA to reduce the dimensions on the data, how many components do you keep?\n",
    "2. Report the accuracy, and compare it to the best performing neural network, logistic regression and linear regression models. What do you see?"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "interpreter": {
   "hash": "7cf3cfb4d2a53586223bf4603cd7f9e645cf44a77dbcec96182c9a81e54296ad"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}