monailabel/monailabel_pathology_nuclei_segmentation_QuPath.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License. \n",
    "\n",
    "# MONAI Label Pathology App - QuPath Nuclei Segmentation Example\n",
    "\n",
    "***The pathology bioimage analysis with MONAI Label***\n",
    "\n",
    "In this notebook, we show the example of MONAI Label Pathology usecase. The nuclei segmentation is demonstrated. This tutorial uses Qupath as the client viewer, and we demonstrate the end-to-end workflow with MONAI Label + QuPath integration.\n",
    "\n",
    "![qupath1](./figures/monailabel_pathology_qupath/qupath1.png)\n",
    "\n",
    "## 1. Prepare MONAI Label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "\n",
    "### Prerequisites\n",
    "- **Install MONAI Label** weekly preview release: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install monailabel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download pathology app\n",
    "Choose \"pathology\" as the app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel apps --download --name pathology --output apps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download sample data\n",
    "Download the sample data from digital slide archive with wget and save to the **datasets** folder for further use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir datasets\n",
    "!wget \"https://demo.kitware.com/histomicstk/api/v1/item/5d5c07509114c049342b66f8/download\" -O \"datasets/JP2K-33003-1.svs\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Starting MONAI Label Server\n",
    "\n",
    "The command will start MONAI Label server with all available models. If user wants to start with a specific model, set the model name in the **--conf models** argument.\n",
    "\n",
    "Such as\n",
    "\n",
    "```bash\n",
    "monailabel start_server --app <path to pathology app> --studies <path to downloaded datafolder> --conf models segmentation_nuclei\n",
    "```\n",
    "In the command, ```--app``` specifies the pathology app path, ```--studies``` specifies the dataset folder path.\n",
    "\n",
    "**Note:** If you are running a MONAI Label server in this notebook and plan to close the notebook, ensure that you terminate the kernel first. Failing to do so might result in an issue where the port remains occupied from the previous session when you try to restart the server later. To prevent this, always terminate the kernel before closing the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel start_server --app apps/pathology --studies datasets/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Install QuPath and MONAI Label Plugin\n",
    "\n",
    "***MONAI Plugin supports QuPath version <=0.5.0***\n",
    "\n",
    "\n",
    "[QuPath](https://qupath.github.io/) is an open software for bioimage analysis. it can be used as an efficient tool for annotation and visualization of whole slide images. \n",
    "\n",
    "Note: Nvidia does not maintain the thrid-party visualization tools. Please refer to [installation guide](https://qupath.readthedocs.io/en/stable/docs/intro/installation.html) for details or trouble shooting. \n",
    "\n",
    "In this tutorial, we provide minimum steps for QuPath setup. \n",
    "\n",
    "**Step 1**: Get QuPath from website.  \n",
    "\n",
    "Go to the [QuPath website](https://qupath.github.io/), click \"Download for Linux\" to pull the *tar.xz* file\n",
    "\n",
    "Extract the QuPath software by:\n",
    "\n",
    "```bash\n",
    "# E.g., QuPath 0.5.0\n",
    "tar -xf <QuPath-0.5.0-Linux.tar.xz>\n",
    "```\n",
    "\n",
    "**Step 2**: Start QuPath with excutable launcher file.\n",
    "\n",
    "Start QuPath by opening a **Terminal**, Use ```chmod u+x /path/to/QuPath/bin/QuPath``` to make the extracted QuPath launcher executable.\n",
    "If you suffer from problem, please refer to QuPath [QA](https://github.com/qupath/qupath/releases/tag/v0.5.0).\n",
    "\n",
    "Then type or drag the launcher file: ```/path/to/QuPath/bin/QuPath``` to the terminal, and enter to open QuPath UI.\n",
    "\n",
    "**Step 3** Install MONAI Label Plugin for QuPath\n",
    "\n",
    "Download [qupath-extension-monailabel-0.3.1.jar](https://github.com/Project-MONAI/MONAILabel/releases/download/data/qupath-extension-monailabel-0.3.1.jar)\n",
    "and **drag the jar** on top of the running QuPath application window (black screen area) to install the extension.\n",
    "If you have previously installed then make sure to **_remove/uninstall_** the extension before updating.\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath_plugin.png\" alt=\"drawing\" width=\"700\"/></p>\n",
    "<p align = \"center\">Drag the downloaded MONAI Label QuPath plugin JAR file into QuPath viewer UI.</p>\n",
    "\n",
    "Note: Make sure MONAILabel Server URL (Host+Port) is correct through `Preferences`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Nuclei Auto Segmentation with QuPath\n",
    "\n",
    "Below is a step-by-step tutorial for nuclei segmentation using QuPath. Pre-trained model is provided and loaded by MONAI Label server.\n",
    "\n",
    "\n",
    "### 3.1 Start QuPath, MONAI Label Plugin, and Fetch Next Image\n",
    "\n",
    "- On the menu bar, navigate and click MONAI Label. Click *Next Sample/Patch* to load data specified in the start_server command. If the QuPath starts from beginning, the button will automatically load next whole slide image.\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath2.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    " - Users can also use the **Next Sample/patch** button to select a patch ROI automatically, by selecting **Next Patch**. （Uncheck the **Next Patch** box will load next whole slide image.）\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath3.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "### 3.2 Select ROI by Rectangle Tool\n",
    "\n",
    "First, let's show patch inference with the rectangle ROI tool.\n",
    "\n",
    "- Users can automatically select the rectangle ROI by fetching \"Next Patch\", or use the QuPath rectangle tool to pick an ROI.\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath4.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "### 3.3 Run Nuclei Segmentation\n",
    "\n",
    "- After selecting the rectangle ROI, users can navigate to the MONAI Label plugin and click **Annotations**.\n",
    "Then select models in the pop-up interface, and click **OK** to run inference. \n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath5.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "MONAI Label server will run auto segmentation with the selected patch data and return the segmentation with polygons.\n",
    "\n",
    "### 3.4 Submit ROI Label for Training\n",
    "\n",
    "- In the next step, users can edit the predicted nuclei segmentation or use Nuclick single click segmentation to add/modify the annotation. Once the annotation is done, users can navigate to MONAI Label plugin and click **Submit Label** to save the ground truth.\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath11.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "- Then click **Yes** to save ROI label:\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath12.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "### 3.5 Run Interactive ROI-based Segmentation\n",
    "\n",
    "- Users can also use the interactive ROI tool to get auto-segmentation with one ROI draw:\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath7.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "- Draw any rectangle ROI, and MONAI Label server will return segmentation immediately:\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath8.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "\n",
    "### 3.6 Run Nuclic Model by Single Click (Inference)\n",
    "\n",
    "- MONAI Label server provide Nuclic models, users can do instance segmentation by single click:\n",
    "\n",
    "- Use **Interactor** tool in the toolbar\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath9.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "- By click-and-inference, users can get nuclei segmentation without an ROI:\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath10.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Training Nuclei Segmentation\n",
    "\n",
    "Once users finish annotations on several patch images, MONAI Label provide training option to fine-tune the model.\n",
    "\n",
    "- Navigate to MONAI Label plugin and click **Training**\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath13.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n",
    "- In the training option pop-up interface, select **segmentation_nuclei**, select training customized training options. Uncheck **pretrained** to start training model from scratch. Finally, click **OK** to start training in MONAI Label server.\n",
    "\n",
    "<p align = \"center\"><img src=\"./figures/monailabel_pathology_qupath/qupath14.png\" alt=\"drawing\" width=\"800\"/></p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Repeat the Interactive Labeling Process \n",
    "\n",
    "Repeated fetch data and interactive learning process to section 3.1, until a satisfied model is achieved or entire datasets/WSIs are annotated. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Conclusion\n",
    "\n",
    "This tutorial demonstrates the basic usage of MONAI Label with the pathology app, the nuclei segmentation task is introduced. We introduced several inference strategies including ROI, Nuclick ROI, and single click. Submit label and training options are demonstrated to fine-tune the model or train from scratch. \n",
    "\n",
    "In the pathology usecase, we highlight the interactive learning process when labeling a whole slide image or a dataset. MONAI Label achieves annotation actively and gradually improves the model and segmentation performance during labeling. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}