path-foundation/linear-classifier-demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nCy7vXMN73tI"
      },
      "source": [
        "\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/Google-Health/imaging-research/blob/master/path-foundation/linear-classifier-demo.ipynb\"\u003e\n",
        "  \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\n",
        "\u003c/a\u003e\n",
        "\n",
        "# Path Foundation Linear Probe Demo\n",
        "This notebook is a demonstration of generating and using embeddings from the Path Foundation API to train a linear classifier. This API enables users to compute embeddings for histopathology images. The contents include how to: build an API request to generate embeddings from stored patches and train a linear model using the embeddings. Note: This notebook is for API demonstration purposes only. As with all machine-learning use-cases it is critical to consider training and evaluation datasets that reflect the expected distribution of the intended use case.\n",
        "\n",
        "**Additional details**: For this demo, whole slide images (WSIs) available from the dataset below were split into train and evaluation sets. A subset of patches were sampled randomly from across all available slides and embeddings were generated via the Path Foundation model.\n",
        "\n",
        "**Dataset**: This notebook uses the [CAMELYON16](https://camelyon16.grand-challenge.org/) dataset, which contains WSIs from lymph node specimens with and without metastatic breast cancer. Any work that uses this dataset should consider additional details along with usage and citation requirements listed on [their website](https://camelyon17.grand-challenge.org/Data/).\n",
        "\n",
        "**Dataset citation**: Babak Ehteshami Bejnordi; Mitko Veta; Paul Johannes van Diest; Bram van Ginneken; Nico Karssemeijer; Geert Litjens; Jeroen A. W. M. van der Laak; and the CAMELYON16 Consortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA. 2017;318(22):2199–2210. DOI: 10.1001/jama.2017.14585\n",
        "# Prerequisites\n",
        "You must have access to the Pathology Foundation Tool. See the project's [README](https://github.com/Google-Health/imaging-research/blob/master/path-foundation/README.md) for details.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lv80cFw676RO"
      },
      "source": [
        "\n",
        "# Train a Linear Classifier Using Stored Patches\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zCCYlzPChsFO"
      },
      "outputs": [],
      "source": [
        "!pip install ez-wsi-dicomweb\n",
        "!pip install hcls-imaging-ml-toolkit-ez-wsi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H_wez6RRhvSK"
      },
      "outputs": [],
      "source": [
        "# Imports\n",
        "\n",
        "from collections import defaultdict\n",
        "from dataclasses import dataclass\n",
        "from enum import Enum\n",
        "import json\n",
        "import random\n",
        "from typing import Any, List\n",
        "import warnings\n",
        "import ez_wsi_dicomweb\n",
        "from ez_wsi_dicomweb import dicom_slide\n",
        "from ez_wsi_dicomweb import dicom_web_interface\n",
        "import ez_wsi_dicomweb.dicomweb_credential_factory as dicomweb_credential_factory\n",
        "import ez_wsi_dicomweb.pixel_spacing as pixel_spacing\n",
        "from google.cloud import storage\n",
        "import google.cloud.storage\n",
        "import hcls_imaging_ml_toolkit.dicom_path as dicom_path\n",
        "import hcls_imaging_ml_toolkit.tags as tags\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import requests\n",
        "import sklearn.linear_model\n",
        "import sklearn.metrics\n",
        "import sklearn.model_selection\n",
        "import sklearn.pipeline\n",
        "import sklearn.preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0XPfdfgS8T3Q"
      },
      "outputs": [],
      "source": [
        "# Constants\n",
        "\n",
        "\n",
        "PROJECT_ID = (  # Project that contains the stored patches\n",
        "    'hai-cd3-foundations'\n",
        ")\n",
        "BUCKET_NAME = (  # Bucket that contains the patches\n",
        "    'hai-cd3-foundations-pathology-vault-entry'\n",
        ")\n",
        "DATASET_PROJECT_ID = 'hai-cd3-foundations'  # @param {type: 'string'}\n",
        "DATASET_LOCATION = 'us-west1'  # @param {type: 'string'}\n",
        "DATASET_ID = 'pathology'  # @param {type: 'string'}\n",
        "STORE_ID = 'camelyon'  # @param {type: 'string'}\n",
        "PATCHES_DIR_NAME = 'patches/'  # @param {type: 'string'}\n",
        "EMBEDDINGs_DIR_NAME = 'embeddings/'  # @param {type: 'string'}\n",
        "CANCER_FILE = 'all_cancer_patches.json'  # @param {type: 'string'}\n",
        "NON_CANCER_FILE = 'all_non_cancer_patches.json'  # @param {type: 'string'}\n",
        "TRAINING_CANCER_PATCH_COUNT = 250  # @param {type: 'integer'}\n",
        "TRAINING_NON_CANCER_PATCH_COUNT = 250  # @param {type: 'integer'}\n",
        "EVAL_SET_PATCH_COUNT = 100  # @param {type: 'integer'}\n",
        "\n",
        "# Generated using above values\n",
        "DICOM_WEB_STORE_URL = f'projects/{DATASET_PROJECT_ID}/locations/{DATASET_LOCATION}/datasets/{DATASET_ID}/dicomStores/{STORE_ID}'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GNDYqgXHHqhf"
      },
      "outputs": [],
      "source": [
        "# Constants that should not be modified\n",
        "\n",
        "PATCH_SIZE = 224\n",
        "TARGET_PIXEL_SPACING = pixel_spacing.PixelSpacing.FromMagnificationString('20X')\n",
        "EVAL_RESERVED_SLIDES = 50\n",
        "\n",
        "# API config\n",
        "PROD_API_PROJECT_ID = 'hai-cd3-foundations'\n",
        "PROD_API_LOCATION = 'us-central1'\n",
        "ENDPOINT_ID = '160'\n",
        "ENCODER_ENDPOINT_URL = f'https://{PROD_API_LOCATION}-aiplatform.googleapis.com/v1/projects/{PROD_API_PROJECT_ID}/locations/{PROD_API_LOCATION}/endpoints/{ENDPOINT_ID}:predict'\n",
        "MODEL_SIZE = 'MEDIUM'\n",
        "MODEL_KIND = 'LOW_PIXEL_SPACING'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Cx6AHODBpk1"
      },
      "outputs": [],
      "source": [
        "from google.colab import auth\n",
        "# Authenticate user for access. There will be a popup asking you to sign in with your user account and approve access.\n",
        "auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CdGQNGw6AbjE"
      },
      "outputs": [],
      "source": [
        "# Defines a helper Dataclass for converting Embeddings from JSON\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class Embedding:\n",
        "  # Assumes patch size is always 224 (so width and height of a patch are 224 pixels)\n",
        "  dicom_study_uid: str\n",
        "  dicom_series_uid: str\n",
        "  x_origin: int  # Top left\n",
        "  y_origin: int  # Top left\n",
        "  instance_uids: list[str]\n",
        "  embedding: list[float]\n",
        "\n",
        "  def to_json(self) -\u003e str:\n",
        "    return json.dumps({\n",
        "        'dicom_study_uid': self.dicom_study_uid.strip(),\n",
        "        'dicom_series_uid': self.dicom_series_uid.strip(),\n",
        "        'x_origin': self.x_origin,\n",
        "        'y_origin': self.y_origin,\n",
        "        'instance_uids': self.instance_uids,\n",
        "        'embedding': json.dumps(self.embedding),\n",
        "    })\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class Embeddings:\n",
        "  embeddings: list[Embedding]\n",
        "\n",
        "  def to_json(self) -\u003e str:\n",
        "    return json.dumps([em.construct_json_object() for em in self.embeddings])\n",
        "\n",
        "  def concat(self, other):\n",
        "    return Embeddings(embeddings=self.embeddings + other.embeddings)\n",
        "\n",
        "\n",
        "def embedding_from_json(json_str: str) -\u003e Embedding:\n",
        "  json_object = json.loads(json_str)\n",
        "  return Embedding(\n",
        "      dicom_study_uid=json_object['dicom_study_uid'],\n",
        "      dicom_series_uid=json_object['dicom_series_uid'],\n",
        "      x_origin=json_object['x_origin'],\n",
        "      y_origin=json_object['y_origin'],\n",
        "      instance_uids=json_object['instance_uids'],\n",
        "      embedding=json.loads(json_object['embedding']),  # Deserialize embedding\n",
        "  )\n",
        "\n",
        "\n",
        "def embeddings_from_json(json_str: str) -\u003e Embeddings:\n",
        "  json_objects = json.loads(json_str)\n",
        "  return Embeddings(\n",
        "      embeddings=[embedding_from_json(obj) for obj in json_objects]\n",
        "  )\n",
        "\n",
        "\n",
        "def embeddings_dataclass_from_response(response: str) -\u003e Embeddings:\n",
        "  response = json.loads(response)\n",
        "  embeddings = []\n",
        "  for prediction in response['predictions']:\n",
        "    if type(prediction) == list:\n",
        "      for result in prediction:\n",
        "        for inner_result in result['patch_embeddings']:\n",
        "          embeddings.append(\n",
        "              Embedding(\n",
        "                  dicom_study_uid=result['dicom_study_uid'],\n",
        "                  dicom_series_uid=result['dicom_series_uid'],\n",
        "                  x_origin=inner_result['patch_coordinate']['x_origin'],\n",
        "                  y_origin=inner_result['patch_coordinate']['y_origin'],\n",
        "                  instance_uids=result['instance_uids'],\n",
        "                  embedding=inner_result['embeddings'],\n",
        "              )\n",
        "          )\n",
        "\n",
        "  return Embeddings(embeddings=embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nl6XfMT19h21"
      },
      "outputs": [],
      "source": [
        "# Defines a helper Dataclass for converting Patches from JSON\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class Patch:\n",
        "  slide_id: str\n",
        "  study_instance_uid: str\n",
        "  series_instance_uid: str\n",
        "  x_origin: int\n",
        "  y_origin: int\n",
        "\n",
        "  def to_json(self) -\u003e str:\n",
        "    return json.dumps(asdict(self))\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class PatchCollection:\n",
        "  patches: list[Patch]\n",
        "\n",
        "  def to_json(self) -\u003e str:\n",
        "    return json.dumps(asdict(self))\n",
        "\n",
        "\n",
        "def patch_collection_from_json(json_str: str) -\u003e PatchCollection:\n",
        "  data = json.loads(json_str)\n",
        "  patches = [Patch(**patch_data) for patch_data in data['patches']]\n",
        "  return PatchCollection(patches=patches)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GHJ5DvKj8ZiT"
      },
      "outputs": [],
      "source": [
        "# Downloads the pre-generated patches for the Training and Eval sets\n",
        "\n",
        "\n",
        "client = storage.Client(project=PROJECT_ID)\n",
        "bucket = client.bucket(BUCKET_NAME)\n",
        "\n",
        "\n",
        "def download_and_convert_patches(blob_path: str) -\u003e Patch:\n",
        "  \"\"\"Downloads a blob and converts JSON to dataclass\"\"\"\n",
        "  json_data = (\n",
        "      client.bucket(BUCKET_NAME).get_blob(blob_path).download_as_string()\n",
        "  )\n",
        "  return patch_collection_from_json(json_data)\n",
        "\n",
        "\n",
        "# Downloads patch collections\n",
        "cancer_patch_collection = download_and_convert_patches(\n",
        "    PATCHES_DIR_NAME + CANCER_FILE\n",
        ")\n",
        "non_cancer_patch_collection = download_and_convert_patches(\n",
        "    PATCHES_DIR_NAME + NON_CANCER_FILE\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "luYgDjz-9ZHy"
      },
      "outputs": [],
      "source": [
        "# Helper function to render patches\n",
        "\n",
        "\n",
        "dcf = dicomweb_credential_factory.CredentialFactory()\n",
        "dwi = dicom_web_interface.DicomWebInterface(dcf)\n",
        "\n",
        "\n",
        "def render_patch_from_embedding(embedding: Embedding, plot_name: str = ''):\n",
        "\n",
        "  series_path = dicom_path.Path(\n",
        "      project_id=DATASET_PROJECT_ID,\n",
        "      location=DATASET_LOCATION,\n",
        "      dataset_id=DATASET_ID,\n",
        "      store_id=STORE_ID,\n",
        "      study_uid=embedding.dicom_study_uid,\n",
        "      series_uid=embedding.dicom_series_uid,\n",
        "  )\n",
        "  ds = dicom_slide.DicomSlide(\n",
        "      dwi=dwi, path=series_path, enable_client_slide_frame_decompression=True\n",
        "  )\n",
        "  patch_bytes = ds.get_patch(\n",
        "      pixel_spacing=TARGET_PIXEL_SPACING,\n",
        "      x=embedding.x_origin,\n",
        "      y=embedding.y_origin,\n",
        "      width=PATCH_SIZE,\n",
        "      height=PATCH_SIZE,\n",
        "  ).image_bytes()\n",
        "  plt.figure(figsize=(2, 2))\n",
        "  plt.imshow(patch_bytes)\n",
        "  plt.title(plot_name)\n",
        "  plt.axis('off')\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U6GzRCJiBGVL"
      },
      "outputs": [],
      "source": [
        "# Reserve a portion of slides for eval and separate patches into different lists\n",
        "\n",
        "\n",
        "# Bucket patches by slide_id\n",
        "def build_patches_by_slide_id(\n",
        "    patch_collection: PatchCollection,\n",
        ") -\u003e dict[str, list[Patch]]:\n",
        "  patches_by_slide = defaultdict(list)  # Create a defaultdict of lists\n",
        "  for patch in patch_collection.patches:\n",
        "    patches_by_slide[patch.slide_id].append(patch)  # Directly append\n",
        "  return patches_by_slide\n",
        "\n",
        "\n",
        "def select_random_slide_ids(\n",
        "    patches_by_slide: dict[str, list[Patch]], num_slides: int\n",
        ") -\u003e list[str]:\n",
        "  slide_ids = list(patches_by_slide.keys())  # Get all slide IDs\n",
        "  random.shuffle(slide_ids)  # Shuffle for randomness\n",
        "  return slide_ids[:num_slides]  # Select the first num_slides elements\n",
        "\n",
        "\n",
        "def get_patches_from_slide_ids(\n",
        "    patches_by_slide: dict[str, list[Patch]],\n",
        "    selected_slide_ids: list[str],\n",
        "    include_selected: bool = True,\n",
        ") -\u003e list[Patch]:\n",
        "  def filter_patches(slide_id: str) -\u003e bool:\n",
        "    return (\n",
        "        slide_id in selected_slide_ids\n",
        "        if include_selected\n",
        "        else slide_id not in selected_slide_ids\n",
        "    )\n",
        "\n",
        "  return [\n",
        "      patch\n",
        "      for slide_id in patches_by_slide\n",
        "      if filter_patches(slide_id)\n",
        "      for patch in patches_by_slide[slide_id]\n",
        "  ]\n",
        "\n",
        "\n",
        "cancer_slide_to_patches = build_patches_by_slide_id(cancer_patch_collection)\n",
        "non_cancer_slide_to_patches = build_patches_by_slide_id(\n",
        "    non_cancer_patch_collection\n",
        ")\n",
        "\n",
        "eval_reserved_slides = select_random_slide_ids(\n",
        "    cancer_slide_to_patches, EVAL_RESERVED_SLIDES\n",
        ")\n",
        "\n",
        "training_cancer_patches = get_patches_from_slide_ids(\n",
        "    cancer_slide_to_patches, eval_reserved_slides, include_selected=False\n",
        ")\n",
        "training_non_cancer_patches = get_patches_from_slide_ids(\n",
        "    non_cancer_slide_to_patches, eval_reserved_slides, include_selected=False\n",
        ")\n",
        "\n",
        "eval_cancer_patches = get_patches_from_slide_ids(\n",
        "    cancer_slide_to_patches, eval_reserved_slides, include_selected=True\n",
        ")\n",
        "eval_non_cancer_patches = get_patches_from_slide_ids(\n",
        "    non_cancer_slide_to_patches, eval_reserved_slides, include_selected=True\n",
        ")\n",
        "\n",
        "print(f'Total Training Non cancer patches: {len(training_non_cancer_patches)}')\n",
        "print(f'Total Training Cancer patches: {len(training_cancer_patches)}')\n",
        "print(f'Total Eval Non cancer patches: {len(eval_non_cancer_patches)}')\n",
        "print(f'Total Eval Cancer patches: {len(eval_cancer_patches)}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JzlgrUh-JsMc"
      },
      "outputs": [],
      "source": [
        "# Generate Auth Token\n",
        "\n",
        "auth_token = !gcloud auth print-access-token\n",
        "AUTH_TOKEN = auth_token[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "823zWUcsIEwD"
      },
      "outputs": [],
      "source": [
        "# Generates an API call for embedding generation\n",
        "\n",
        "\n",
        "def generate_embeddings_payload(\n",
        "    patch_count: int, input_patches: List[Patch]\n",
        ") -\u003e dict[str, Any]:\n",
        "  selected_patches = random.sample(input_patches, patch_count)\n",
        "\n",
        "  # Group patches by series for efficient processing\n",
        "  patches_by_series = _group_patches_by_series(selected_patches)\n",
        "\n",
        "  instances = []\n",
        "  for series_uid, patches in patches_by_series.items():\n",
        "    instances.append(_create_instance_data(patches))\n",
        "\n",
        "  return {\n",
        "      'parameters': {'model_size': MODEL_SIZE, 'model_kind': MODEL_KIND},\n",
        "      'instances': instances,\n",
        "  }\n",
        "\n",
        "\n",
        "def _group_patches_by_series(patches: List[Patch]) -\u003e dict[str, List[Patch]]:\n",
        "  patches_by_series = defaultdict(list)\n",
        "  for patch in patches:\n",
        "    patches_by_series[patch.series_instance_uid].append(patch)\n",
        "  return patches_by_series\n",
        "\n",
        "\n",
        "def _create_instance_data(patches: List[Patch]) -\u003e dict[str, Any]:\n",
        "  first_patch = patches[0]\n",
        "  series_path = _create_dicom_slide_path(first_patch)\n",
        "  ds = dicom_slide.DicomSlide(\n",
        "      dwi=dwi, path=series_path, enable_client_slide_frame_decompression=True\n",
        "  )\n",
        "\n",
        "  instance_uids = _get_instance_uids(ds)\n",
        "\n",
        "  return {\n",
        "      'dicom_web_store_url': DICOM_WEB_STORE_URL,\n",
        "      'dicom_study_uid': first_patch.study_instance_uid,\n",
        "      'dicom_series_uid': first_patch.series_instance_uid,\n",
        "      'bearer_token': AUTH_TOKEN,\n",
        "      'ez_wsi_state': {},\n",
        "      'instance_uids': instance_uids,\n",
        "      'patch_coordinates': _format_patch_coordinates(patches),\n",
        "  }\n",
        "\n",
        "\n",
        "def _create_dicom_slide_path(patch: Patch) -\u003e dicom_path.Path:\n",
        "  return dicom_path.Path(\n",
        "      project_id=DATASET_PROJECT_ID,\n",
        "      location=DATASET_LOCATION,\n",
        "      dataset_id=DATASET_ID,\n",
        "      store_id=STORE_ID,\n",
        "      study_uid=patch.study_instance_uid,\n",
        "      series_uid=patch.series_instance_uid,\n",
        "  )\n",
        "\n",
        "\n",
        "def _get_instance_uids(ds: dicom_slide.DicomSlide) -\u003e List[str]:\n",
        "  # Get instance UID at TARGET_PIXEL_SPACING\n",
        "  instance_uids = []\n",
        "  for instance_id, instance in ds.get_level_by_pixel_spacing(\n",
        "      TARGET_PIXEL_SPACING\n",
        "  ).instances.items():\n",
        "    instance_uids.append(instance.dicom_object.get_value(tags.SOP_INSTANCE_UID))\n",
        "  return instance_uids\n",
        "\n",
        "\n",
        "def _format_patch_coordinates(patches: List[Patch]) -\u003e List[dict[str, int]]:\n",
        "  return [\n",
        "      {\n",
        "          'x_origin': patch.x_origin,\n",
        "          'y_origin': patch.y_origin,\n",
        "          'width': PATCH_SIZE,\n",
        "          'height': PATCH_SIZE,\n",
        "      }\n",
        "      for patch in patches\n",
        "  ]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x7HObNnGGvz_"
      },
      "outputs": [],
      "source": [
        "# Select the configured patch counts from the above lists. May take 5 minutes\n",
        "\n",
        "\n",
        "eval_cancer_payload = generate_embeddings_payload(\n",
        "    patch_count=EVAL_SET_PATCH_COUNT, input_patches=eval_cancer_patches\n",
        ")\n",
        "eval_non_cancer_payload = generate_embeddings_payload(\n",
        "    patch_count=EVAL_SET_PATCH_COUNT, input_patches=eval_non_cancer_patches\n",
        ")\n",
        "training_cancer_payload = generate_embeddings_payload(\n",
        "    patch_count=TRAINING_CANCER_PATCH_COUNT,\n",
        "    input_patches=training_cancer_patches,\n",
        ")\n",
        "training_non_cancer_payload = generate_embeddings_payload(\n",
        "    patch_count=TRAINING_NON_CANCER_PATCH_COUNT,\n",
        "    input_patches=training_non_cancer_patches,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kZB-kvlgHZjs"
      },
      "outputs": [],
      "source": [
        "# Generate Embeddings for the Training and Eval sets. Takes 5 Minutes\n",
        "\n",
        "# Because we are fetching patches from random locations across many slides, this\n",
        "# may take several minutes. In scenarios where multiple patches can be retrieved,\n",
        "# sequentially across a whole slide image, performance will be faster for this step.\n",
        "\n",
        "headers = {'Authorization': f'Bearer {AUTH_TOKEN}'}\n",
        "\n",
        "eval_cancer_response = requests.post(\n",
        "    ENCODER_ENDPOINT_URL, headers=headers, json=eval_cancer_payload\n",
        ").text\n",
        "\n",
        "eval_non_cancer_response = requests.post(\n",
        "    ENCODER_ENDPOINT_URL, headers=headers, json=eval_non_cancer_payload\n",
        ").text\n",
        "\n",
        "training_cancer_response = requests.post(\n",
        "    ENCODER_ENDPOINT_URL, headers=headers, json=training_cancer_payload\n",
        ").text\n",
        "\n",
        "training_non_cancer_response = requests.post(\n",
        "    ENCODER_ENDPOINT_URL, headers=headers, json=training_non_cancer_payload\n",
        ").text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OdrBjODEK3HU"
      },
      "outputs": [],
      "source": [
        "### Process Embeddings for the Training and Eval sets\n",
        "\n",
        "\n",
        "eval_cancer_embeddings = embeddings_dataclass_from_response(\n",
        "    eval_cancer_response\n",
        ")\n",
        "eval_non_cancer_embeddings = embeddings_dataclass_from_response(\n",
        "    eval_non_cancer_response\n",
        ")\n",
        "training_cancer_embeddings = embeddings_dataclass_from_response(\n",
        "    training_cancer_response\n",
        ")\n",
        "training_non_cancer_embeddings = embeddings_dataclass_from_response(\n",
        "    training_non_cancer_response\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6LuZpl-p8prK"
      },
      "outputs": [],
      "source": [
        "# Pass the Embeddings for the Training and Eval sets into scikit-learn\n",
        "\n",
        "\n",
        "def concatenate_embeddings(embeddings_obj: Embeddings) -\u003e np.array:\n",
        "  \"\"\"Concatenates embeddings into a NumPy array.\"\"\"\n",
        "  return np.array(\n",
        "      [embedding.embedding for embedding in embeddings_obj.embeddings]\n",
        "  )\n",
        "\n",
        "\n",
        "def concatenate_series_ids(embeddings_obj: Embeddings) -\u003e np.array:\n",
        "  \"\"\"Concatenates instance UIDs into a NumPy array.\"\"\"\n",
        "  # Assume there is one instance uid per series.\n",
        "  return np.array(\n",
        "      [embedding.instance_uids[0] for embedding in embeddings_obj.embeddings]\n",
        "  )\n",
        "\n",
        "\n",
        "# Create NumPy arrays directly (more efficient)\n",
        "training_embeddings = np.concatenate([\n",
        "    concatenate_embeddings(training_cancer_embeddings),\n",
        "    concatenate_embeddings(training_non_cancer_embeddings),\n",
        "])\n",
        "eval_embeddings = np.concatenate([\n",
        "    concatenate_embeddings(eval_cancer_embeddings),\n",
        "    concatenate_embeddings(eval_non_cancer_embeddings),\n",
        "])\n",
        "\n",
        "training_ids = np.concatenate([\n",
        "    concatenate_series_ids(training_cancer_embeddings),\n",
        "    concatenate_series_ids(training_non_cancer_embeddings),\n",
        "])\n",
        "eval_ids = np.concatenate([\n",
        "    concatenate_series_ids(eval_cancer_embeddings),\n",
        "    concatenate_series_ids(eval_non_cancer_embeddings),\n",
        "])\n",
        "\n",
        "# Create labels (already done in the previous question)\n",
        "training_labels = np.concatenate((\n",
        "    np.ones(TRAINING_CANCER_PATCH_COUNT),\n",
        "    np.zeros(TRAINING_NON_CANCER_PATCH_COUNT),\n",
        "))\n",
        "eval_labels = np.concatenate(\n",
        "    (np.ones(EVAL_SET_PATCH_COUNT), np.zeros(EVAL_SET_PATCH_COUNT))\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aYWil5ur8ruZ"
      },
      "outputs": [],
      "source": [
        "# Train a linear classifier\n",
        "\n",
        "\n",
        "with warnings.catch_warnings():\n",
        "  warnings.simplefilter('ignore')\n",
        "  clf_pipeline = sklearn.pipeline.Pipeline([\n",
        "      ('scaler', sklearn.preprocessing.StandardScaler()),\n",
        "      (\n",
        "          'logreg',\n",
        "          sklearn.model_selection.GridSearchCV(\n",
        "              sklearn.linear_model.LogisticRegression(\n",
        "                  random_state=0,\n",
        "                  multi_class='ovr',\n",
        "                  verbose=False,\n",
        "              ),\n",
        "              cv=sklearn.model_selection.StratifiedGroupKFold(n_splits=5).split(\n",
        "                  training_embeddings, y=training_labels, groups=training_ids\n",
        "              ),\n",
        "              param_grid={'C': np.logspace(start=-4, stop=4, num=10, base=10)},\n",
        "              scoring='roc_auc_ovr',\n",
        "              refit=True,\n",
        "          ),\n",
        "      ),\n",
        "  ]).fit(training_embeddings, training_labels)\n",
        "\n",
        "  test_predictions = clf_pipeline.predict_proba(eval_embeddings)[:, 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JykI-foV8vuL"
      },
      "outputs": [],
      "source": [
        "# Evaluate the linear classifiers performance using the eval set\n",
        "\n",
        "sklearn.metrics.roc_auc_score(eval_labels, test_predictions)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iS-6ahz3i2Hn"
      },
      "outputs": [],
      "source": [
        "# @title Plot the ROC Curve\n",
        "\n",
        "display = sklearn.metrics.RocCurveDisplay.from_predictions(\n",
        "    eval_labels, test_predictions, name=\"Tumor Classifier\"\n",
        ")\n",
        "display.ax_.set_title(\"ROC of Tumor Classifier\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ORAsWcLVcERu"
      },
      "outputs": [],
      "source": [
        "# @title Find Youden's index for threshold selection\n",
        "\n",
        "thresholds = np.linspace(0, 1, 100)\n",
        "sensitivities = []\n",
        "specificities = []\n",
        "for threshold in thresholds:\n",
        "  predictions = test_predictions \u003e threshold\n",
        "  sensitivities.append(sklearn.metrics.recall_score(eval_labels, predictions))\n",
        "  specificities.append(\n",
        "      sklearn.metrics.recall_score(eval_labels == 0, predictions == 0)\n",
        "  )\n",
        "index = np.argmax(np.array(sensitivities) + np.array(specificities))\n",
        "best_threshold = thresholds[index]\n",
        "sens = sensitivities[index]\n",
        "spec = specificities[index]\n",
        "print(\n",
        "    f\"Best threshold: {round(best_threshold,2)}. Sensitivity is\"\n",
        "    f\" {round(sens*100,2)}% and Specificity is {round(spec*100,2)}% \"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sIlXrLMQjeeb"
      },
      "outputs": [],
      "source": [
        "# @title Show the results in a table\n",
        "\n",
        "eval_embeddings_obj = eval_cancer_embeddings.concat(eval_non_cancer_embeddings)\n",
        "\n",
        "df = pd.DataFrame(\n",
        "    {'ground_truth': eval_labels, 'model_score': test_predictions}\n",
        ")\n",
        "df['tumor_prediction'] = df['model_score'] \u003e best_threshold\n",
        "df['embeddings'] = [embedding for embedding in eval_embeddings_obj.embeddings]\n",
        "\n",
        "df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kxfGFAyJVf5S"
      },
      "outputs": [],
      "source": [
        "# @title Visualize True Positives\n",
        "\n",
        "df_tp = (\n",
        "    df[(df['tumor_prediction'] == True) \u0026 (df['ground_truth'] == 1)]\n",
        "    .sort_values('model_score', ascending=False)\n",
        "    .head(5)\n",
        ")\n",
        "for _, row in df_tp.iterrows():\n",
        "  print(f'model score is {row.model_score}')\n",
        "  render_patch_from_embedding(row.embeddings, f'True Positive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jvb8-wbZVtkS"
      },
      "outputs": [],
      "source": [
        "# @title Visualize True Negatives\n",
        "\n",
        "df_tn = (\n",
        "    df[(df['tumor_prediction'] == False) \u0026 (df['ground_truth'] == 0)]\n",
        "    .sort_values('model_score', ascending=False)\n",
        "    .head(5)\n",
        ")\n",
        "for _, row in df_tn.iterrows():\n",
        "  print(f'model score is {row.model_score}')\n",
        "  render_patch_from_embedding(row.embeddings, f'True Negative')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mCB6sfxvjnru"
      },
      "outputs": [],
      "source": [
        "# @title Visualize False Positives\n",
        "\n",
        "df_fp = df[\n",
        "    (df['tumor_prediction'] == True) \u0026 (df['ground_truth'] == 0)\n",
        "].sort_values('model_score', ascending=False)\n",
        "for _, row in df_fp.iterrows():\n",
        "  print(f'model score is {row.model_score}')\n",
        "  render_patch_from_embedding(row.embeddings, f'False Positive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "laWdwj_7jp2K"
      },
      "outputs": [],
      "source": [
        "# @title Visualize False Negatives\n",
        "\n",
        "df_fn = df[\n",
        "    (df['tumor_prediction'] == False) \u0026 (df['ground_truth'] == 1)\n",
        "].sort_values('model_score', ascending=True)\n",
        "for _, row in df_fn.iterrows():\n",
        "  print(f'model score is {row.model_score}')\n",
        "  render_patch_from_embedding(row.embeddings, f'False Negative')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dWqQeUFkPATt"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "last_runtime": {
        "build_target": "",
        "kind": "local"
      },
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}