S04-Extracting-Patch-Features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eab698c9",
   "metadata": {},
   "source": [
    "# S04: Extracting Patch Features\n",
    "\n",
    "Here we utilize a image feature extractor to extract deep features from all the patches that we have obtained from the step `S03`. The image feature extractor used in this step is [`CONCH`](https://github.com/mahmoodlab/CONCH), a vision-language model pretrained on pathology images. \n",
    "\n",
    "We adopt [our enhanced CLAM](./tools/CLAM) to introduce how to extract patch features.\n",
    "\n",
    "## 1. Some Notes\n",
    "\n",
    "### 1.1 Overall Procedure\n",
    "\n",
    "In this step, for each slide, its patch coordinates, which are stored in a `h5` file (see `ROOT_DIR_FOR_DATA_SAVING/tiles-20x-s256/patches/` in your server), would be loaded and then used to locate certain patch regions in this slide image (at the magnification you specified in the step `S03`). \n",
    "\n",
    "Meanwhile, the source file of the slide will also be loaded for reading patch regions. At the end, all patch features of the slide will be saved in a `pt` file. \n",
    "\n",
    "### 1.2 Extracting Patch Features at a Unified Magnification\n",
    "\n",
    "Recalling the previous step (S03), we patch all WSIs at a unified magnification by specifying `--patch_magnification 20` and `--patch_size 256`. \n",
    "\n",
    "So, when the WSIs have **different highest magnifications** (`20x` or `40x`, often seen in `TCGA`), different patch sizes (e.g. `256 x 256` or `512 x 512`) could be produced. Therefore, we need to specifiy `target_patch_size` as `256` to **resize patches** to the same size for feature extraction.\n",
    "\n",
    "By doing the above, we actually read `256 x 256` patches at `20x` and extract features from the patches with the same size. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f49349",
   "metadata": {},
   "source": [
    "## 2. Running feature extraction\n",
    "\n",
    "In this step, we have improved CLAM specifically in terms of \n",
    "- more alternative architectures (including 6+ pretrained SOTA model) for patch feature extracting, where \n",
    "  - `CONCH` is recommended \n",
    "  - `CTransPath` and `PLIP` are good alternatives (both are free for use), when you cannot use CONCH due to limited access rights\n",
    "  - `CTransPath` and `PLIP` scripts are provided in [tools/scripts](https://github.com/liupei101/Pipeline-Processing-TCGA-Slides-for-MIL/tree/main/tools/scripts)\n",
    "  - The `truncated ResNet50` and `ResNet18 w/ SimCL` are **NOT** recommended\n",
    "\n",
    "### 2.1 Running Scripts\n",
    "\n",
    "A detailed bash script (placed at `./tools/scripts/S04-Extracting-Feats.sh`), with `CONCH` as the patch feature extractor, is as follows:\n",
    "\n",
    "```bash\n",
    "#!/bin/bash\n",
    "set -e\n",
    "\n",
    "# Sample patches of SIZE x SIZE at MAG (as used in S03)\n",
    "MAG=20\n",
    "SIZE=256\n",
    "\n",
    "# Path where CLAM is installed\n",
    "DIR_REPO=../CLAM\n",
    "\n",
    "# Root path to pathology images \n",
    "DIR_RAW_DATA=/NAS02/RawData/tcga_rcc\n",
    "DIR_EXP_DATA=/NAS02/ExpData/tcga_rcc\n",
    "\n",
    "# Sub-directory to the patch coordinates generated from S03\n",
    "SUBDIR_READ=tiles-${MAG}x-s${SIZE}\n",
    "\n",
    "# Arch to be used for patch feature extraction (CONCH is strongly recommended)\n",
    "ARCH=CONCH\n",
    "\n",
    "# Model path\n",
    "# You need to first apply for its access rights via https://huggingface.co/MahmoodLab/CONCH\n",
    "# and then download a model file named `pytorch_model.bin`.\n",
    "MODEL_CKPT=/path/to/conch/pytorch_model.bin\n",
    "\n",
    "# Sub-directory to the patch features \n",
    "SUBDIR_SAVE=${SUBDIR_READ}/feats-${ARCH}\n",
    "\n",
    "cd ${DIR_REPO}\n",
    "\n",
    "echo \"running for extracting features from all tiles\"\n",
    "CUDA_VISIBLE_DEVICES=0 python3 extract_features_fp.py \\\n",
    "    --arch ${ARCH} \\\n",
    "    --ckpt_path ${MODEL_CKPT} \\\n",
    "    --data_h5_dir ${DIR_EXP_DATA}/${SUBDIR_READ} \\\n",
    "    --data_slide_dir ${DIR_RAW_DATA} \\\n",
    "    --csv_path ${DIR_EXP_DATA}/${SUBDIR_READ}/process_list_autogen.csv \\\n",
    "    --feat_dir ${DIR_EXP_DATA}/${SUBDIR_SAVE} \\\n",
    "    --target_patch_size ${SIZE} \\\n",
    "    --batch_size 128 \\\n",
    "    --slide_ext .svs \\\n",
    "    --slide_in_child_dir \\\n",
    "    --proj_to_contrast N\n",
    "\n",
    "```\n",
    "\n",
    "You could run this script using the following command:\n",
    "```bash\n",
    "nohup ./S04-Extracting-Feats.sh > S04-Extract-Feats.log 2>&1 &\n",
    "```\n",
    "\n",
    "Full running logs could be found in `./tools/scripts/S04-Extract-Feats.log`. \n",
    "\n",
    "Next, we check if the number of generated files is consistent with that of patch files from the step `S03`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bdf2d4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import os.path as osp\n",
    "\n",
    "DIR_FEAT = \"/NAS02/ExpData/tcga_rcc/tiles-20x-s256/feats-CONCH/pt_files\"\n",
    "feat_files = [f for f in os.listdir(DIR_FEAT) if f.endswith(\".pt\")]\n",
    "print(\"This step generated {} feature files in {}.\".format(len(feat_files), DIR_FEAT))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6f48269",
   "metadata": {},
   "outputs": [],
   "source": [
    "DIR_PATCH = \"/NAS02/ExpData/tcga_rcc/tiles-20x-s256/patches\"\n",
    "patch_files = [f for f in os.listdir(DIR_PATCH) if f.endswith(\".h5\")]\n",
    "print(\"The step S03 generated {} patch files in {}.\".format(len(patch_files), DIR_PATCH))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d64f8553",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_filenames = [osp.splitext(f)[0] for f in feat_files]\n",
    "patch_filenames = [osp.splitext(f)[0] for f in patch_files]\n",
    "flag = False\n",
    "for f in patch_filenames:\n",
    "    if f not in feat_filenames:\n",
    "        flag = True\n",
    "        print(\"Expected {}, but it was not found in features files.\".format(f))\n",
    "if flag:\n",
    "    print(\"Some slides were not processed.\")\n",
    "else:\n",
    "    print(\"All slides in patch directory have been processed in this step.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203ebd48",
   "metadata": {},
   "source": [
    "\n",
    "### Example of Running Logs\n",
    "\n",
    "The running log of the first WSI is presented as follows:\n",
    "\n",
    "```txt\n",
    "progress: 0/940\n",
    "TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676\n",
    "downsample [4.00005125 4.00008641]\n",
    "downsampled_level_dim [39021 23146]\n",
    "level_dim [39021 23146]\n",
    "name TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676\n",
    "patch_level 0\n",
    "patch_size 512\n",
    "save_path /NAS02/ExpData/tcga_rcc/tiles-20x-s256/patches\n",
    "\n",
    "feature extraction settings:\n",
    "-- target patch size:  None\n",
    "-- imagenet_pretrained:  False\n",
    "-- patches sampler: None\n",
    "-- color normalization: None\n",
    "-- color argmentation: None\n",
    "-- add_patch_noise: None\n",
    "-- vertical_flip: False\n",
    "-- transformations:  Compose(\n",
    "    Resize(size=256, interpolation=bicubic, max_size=None, antialias=None)\n",
    "    CenterCrop(size=(256, 256))\n",
    "    <function _convert_to_rgb at 0x7f63c5177160>\n",
    "    ToTensor()\n",
    "    Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))\n",
    ")\n",
    "-- enable direct transform:  True\n",
    "processing /NAS02/ExpData/tcga_rcc/tiles-20x-s256/patches/TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676.h5: total of 57 batches\n",
    "batch 0/57, 0 files processed\n",
    "batch 20/57, 2560 files processed\n",
    "batch 40/57, 5120 files processed\n",
    "features size:  torch.Size([7274, 1024])\n",
    "saved pt file:  /NAS02/ExpData/tcga_rcc/tiles-20x-s256/feats-CONCH/pt_files/TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676.pt\n",
    "\n",
    "computing features for /NAS02/ExpData/tcga_rcc/tiles-20x-s256/feats-CONCH/pt_files/TCGA-2K-A9WE-01Z-00-DX1.ED8ADE3B-D49B-403B-B4EB-BD11D91DD676.pt took 75.39572024345398 s\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}