AREN0532/AREN0532.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "SmeqJoR5k9z0",
        "CSTkcuXMgCDd",
        "7YGlC_ajggC8",
        "nj6P7YCmll4S",
        "I9_O-nq1lr7c",
        "JkCZMxa_ua6P",
        "BJEBAyTyqcU5",
        "YMIPh4vzqfN5",
        "9Zp_hWmXsKZ9"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/AREN0532/AREN0532.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/AREN0532/AREN0532.ipynb)"
      ],
      "metadata": {
        "id": "zhJ1PywgTDmh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Accessing DICOM images and annotations from the AREN0532 dataset hosted on TCIA\n",
        "\n",
        "This notebook is focused on accessing the **\"Vincristine, Dactinomycin, and Doxorubicin With or Without Radiation Therapy or Observation Only in Treating Younger Patients Who Are Undergoing Surgery for Newly Diagnosed Stage I, Stage II, or Stage III Wilms' Tumor (AREN0532)\"** Collection hosted on [The Cancer Imaging Archive(TCIA)](https://cancerimagingarchive.net).  This dataset includes DICOM images hosted on TCIA and clinical data hosted by the NCTN Data Archive.  The National Cancer Institute has also funded an activity to generate and publish annotations (3d segmentation labels and seed points) on TCIA to help jumpstart research on tumor detection and auto-segmentation methods.  \n"
      ],
      "metadata": {
        "id": "KmXfYFZtja2F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1 Learn about and request access to the datasets\n",
        "\n",
        "The images, annotations (tumor segmentation and seed point labels), and clinical data associated with this trial are described in detail at the following links.  These pages are publicly visible without logging in, and can be used to obtain an understanding of the dataset before going through the trouble of requesting access:\n",
        "\n",
        "1.  [Collection Summary](https://doi.org/10.7937/6PJ1-M859)\n",
        "2.  [Annotation Summary](https://doi.org/10.7937/kja4-1z76)\n",
        "3.  [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/689)\n",
        "\n",
        "**Note:** You can use the link above to view data dictionaries outlining the specific clinical variables that were collected before requesting access.\n",
        "\n",
        "### Requesting Access to the data\n",
        "In order to download the actual data you must request access through the NCTN Data Archive via the following steps:\n",
        " \n",
        " 1. [Register an account on the NCTN Data Archive](https://nctn-data-archive.nci.nih.gov/).  \n",
        " 2. After logging in, use the \"Request Data\" link in the left side menu.  \n",
        " 3. Follow the on screen instructions, and enter **NCT00352534** when asked which trial you want to request.  \n",
        " 4. In step 2 of the Create Request form, be sure to select “Imaging Data Requested”. \n",
        " \n",
        "Once you are approved for access you'll be able to download the clinical data from the NCTN Archive.  You will then be asked to create an account on TCIA with the same email address to access the imaging data.  Please contact NCINCTNDataArchive@mail.nih.gov for any questions about access requests.  "
      ],
      "metadata": {
        "id": "AruUGe3lmjkh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2 Import tcia_utils\n",
        "\n",
        "The following imports **[tcia_utils](https://pypi.org/project/tcia-utils/)**, which contains a variety of useful functions for accessing TCIA via Python and Juptyter Notebooks."
      ],
      "metadata": {
        "id": "bqulqBEDMSS6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --upgrade -q tcia_utils\n",
        "!pip install --upgrade -q pandas"
      ],
      "metadata": {
        "id": "zP4VRfgg-QXU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "from tcia_utils import nbia"
      ],
      "metadata": {
        "id": "p3zEqnxi9rk2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3 Downloading images and annotations with the NBIA Data Retriever\n",
        "\n",
        "TCIA uses software called NBIA to manage its DICOM data.  One way to download TCIA data is to install the [linux command-line version of the NBIA Data Retriever](https://wiki.cancerimagingarchive.net/x/2QKPBQ) using the following steps.  This tool provides a number of useful features such as auto-retry if there are any problems, saving data in an organized hierarchy on your hard drive (Collection > Patient > Study > Series > Images), and providing a CSV file containing key DICOM metadata about the images you've downloaded.\n",
        "\n",
        "_**Note:**_ It's also possible to download these data via our REST API if you can't or don't want to install the NBIA Data Retriever. This is covered later in the notebook."
      ],
      "metadata": {
        "id": "eRvQa63-kJIY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3.1 Install the NBIA Data Retriever CLI package"
      ],
      "metadata": {
        "id": "RtLE_18NoaJ8"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B714bOkDk8kd"
      },
      "source": [
        "# Install NBIA Data Retriever CLI software for downloading images later in this notebook.\n",
        "\n",
        "!mkdir /usr/share/desktop-directories/\n",
        "!wget -P /content/NBIA-Data-Retriever https://cbiit-download.nci.nih.gov/nbia/releases/ForTCIA/NBIADataRetriever_4.4/nbia-data-retriever-4.4.deb\n",
        "!dpkg -i /content/NBIA-Data-Retriever/nbia-data-retriever-4.4.deb\n",
        "\n",
        "# NOTE: If you're working on a Linux OS that uses RPM packages, you can change the wget line above to point to\n",
        "#       https://cbiit-download.nci.nih.gov/nbia/releases/ForTCIA/NBIADataRetriever_4.4/NBIADataRetriever-4.4-1.x86_64.rpm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3.2 Set up your credential file\n",
        "Since this Collection requires logging in, you must setup a **credentials.txt** file that contains your user name and password. We will leverage the **nbia.makeCredentialFile()** function to do this."
      ],
      "metadata": {
        "id": "ZSUFFDVRkJIZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.makeCredentialFile()"
      ],
      "metadata": {
        "id": "4sNEsvm7kJIZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3.3 Download data\n",
        "The NBIA Data Retriever software works by ingesting a \"manifest\" file that contains the DICOM Series Instance UIDs of the scans you want to download. The manifest files can be downloaded from [this page](https://doi.org/10.7937/D8A8-6252), but you can also obtain these manifests with the commands below.\n",
        "\n",
        "* Annotations - segmentations, seed points, and Negative Findings Assessments \n",
        "* Source images used to create segmentations & seed points\n",
        "* Source images used to create Negative Assessment Reports"
      ],
      "metadata": {
        "id": "Ts_aUgA1kJIZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Annotations - segmentations, seed points, and Negative Findings Assessments\n",
        "manifest = requests.get(\"https://wiki.cancerimagingarchive.net/download/attachments/145752341/AREN0532_Tumor-Annotations-manifest_2-8-2023.tcia?api=v2\")\n",
        "with open('AREN0532_Tumor-Annotations-manifest_2-8-2023.tcia', 'wb') as f:\n",
        "    f.write(manifest.content)"
      ],
      "metadata": {
        "id": "yNJmSdWk4STr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Source images used to create segmentations & seed points\n",
        "manifest = requests.get(\"https://wiki.cancerimagingarchive.net/download/attachments/145752341/AREN0532_SourceImages_SEGSandSeedpoints-manifest_2-8-2023.tcia?api=v2\")\n",
        "with open('AREN0532_SourceImages_SEGSandSeedpoints-manifest_2-8-2023.tcia', 'wb') as f:\n",
        "    f.write(manifest.content)\n"
      ],
      "metadata": {
        "id": "GQ6y0K4Z4SKw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Source images used to create Negative Assessment Reports\n",
        "# (no segmentation or seed points created for the scan)\n",
        "manifest = requests.get(\"https://wiki.cancerimagingarchive.net/download/attachments/145752341/AREN0532_SourceImages_NegativeAssessments-manifest-2-8-2023.tcia?api=v2\")\n",
        "with open('AREN0532_SourceImages_NegativeAssessments-manifest-2-8-2023.tcia', 'wb') as f:\n",
        "    f.write(manifest.content)"
      ],
      "metadata": {
        "id": "00sVKS7H4SBC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "A manifest containing sample images and annotations for a single subject has also been created for use with this notebook to facilitate quick testing and demonstrations."
      ],
      "metadata": {
        "id": "rVspqXGGOlct"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Single subject manifest containing examples of each annotation type\n",
        "# Use this one for a quick demo.\n",
        "manifest = requests.get(\"https://github.com/kirbyju/TCIA_Notebooks/raw/main/AREN0532/AREN0532-PAPVBY.tcia\")\n",
        "with open('AREN0532-PAPVBY.tcia', 'wb') as f:\n",
        "    f.write(manifest.content)"
      ],
      "metadata": {
        "id": "BzGvAkiP4RME"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can open the sample manifest file with the NBIA Data Retriever to download the actual data. You can repeat this step below for each dataset by changing the name of the TCIA manifest you want to download.\n",
        "\n",
        "**<font color='red'>After running the following command, click in the output cell, type \"y,\" and press Enter to agree with the TCIA Data Usage Policy and start the download.</font>**"
      ],
      "metadata": {
        "id": "_wpJTOmOkJIZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!/opt/nbia-data-retriever/nbia-data-retriever --cli '/content/AREN0532-PAPVBY.tcia' -d /content/ -l /content/credentials.txt"
      ],
      "metadata": {
        "id": "XsgWyA2hkJIZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVOOJWDckJIa"
      },
      "source": [
        "# 4 Accessing the REST APIs \n",
        "The [NBIA REST APIs](https://wiki.cancerimagingarchive.net/x/ZoATBg) allow TCIA users to query metadata and download image data.  We'll rely heavily on [tcia_utils](https://pypi.org/project/tcia-utils/) to simplify accessing them.  "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.1 Create an API token\n",
        "\n",
        "We'll use **nbia.getToken()** to generate an access token to query restricted Collections on TCIA.  **<font color='red'>Tokens are valid for 2 hours and must be refreshed after that point.</font>** See https://wiki.cancerimagingarchive.net/x/X4ATBg for more details. "
      ],
      "metadata": {
        "id": "eUww-ijsQNvz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getToken()"
      ],
      "metadata": {
        "id": "URrG3EXRkJIa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.2 Explore the data with REST API queries\n",
        "\n",
        "Let's start by looking at what body parts and modalities are contained in the Collection.  For this dataset, RTSTRUCTs were used to record  the segmentations, seed points, and scans where no tumor was found. By default, most functions from **tcia_utils** return results in JSON."
      ],
      "metadata": {
        "id": "g6D_3Dn7kJIa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# count patients for each modality\n",
        "data = nbia.getModalityCounts(collection = \"AREN0532\")\n",
        "print(data)"
      ],
      "metadata": {
        "id": "oLDN4BAhbhvZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "However, you can also use **format = \"df\"** to return the results as a dataframe."
      ],
      "metadata": {
        "id": "VFrw8FAlkY8x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Count patients for each body part examined, \n",
        "# return results as dataframe\n",
        "df = nbia.getBodyPartCounts(collection = \"AREN0532\", format = \"df\")\n",
        "\n",
        "# rename headers and sort by PatientCount\n",
        "df.rename(columns = {'criteria':'BodyPartExamined', 'count':'PatientCount'}, inplace = True)\n",
        "df.PatientCount = df.PatientCount.astype(int)\n",
        "display(df.sort_values(by='PatientCount', ascending=False, ignore_index = True))"
      ],
      "metadata": {
        "id": "_FZqff2fkJIa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's run **nbia.getPatient()** and **nbia.getStudy()** to see what we can learn about the patient cohort from the DICOM metadata.  This information can include things like age, gender, and ethnicity.  However, in the case of AREN0532, most of this information is also available in the clinical data at https://nctn-data-archive.nci.nih.gov/node/689."
      ],
      "metadata": {
        "id": "OJQQ4-_ZkJIa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtain patient details (e.g. species, gender, ethnicity) for the collection \n",
        "df = nbia.getPatient(collection = \"AREN0532\", api_url = \"restricted\", format = \"df\")\n",
        "display(df)"
      ],
      "metadata": {
        "id": "XUBlXp7P_JI8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's use **format = \"csv\"** this time to save a CSV file in addition to returning a dataframe.  Verify that **getPatientStudy.csv** has been saved to your file system before proceeding."
      ],
      "metadata": {
        "id": "-IkG3w4Xb6W7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtain study/visit details (e.g. anonymized study date, age at the time of visit)\n",
        "df = nbia.getStudy(collection = \"AREN0532\", api_url = \"restricted\", format = \"csv\")\n",
        "display(df)"
      ],
      "metadata": {
        "id": "5Kn8nhMSB5u1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also create a report with **nbia.getSeries()** that gives useful metadata about each scan in the dataset (e.g. series description, modality, scanner manufacturer & software version, number of images).  We'll return the results as JSON this time so we can use them in a subsequent step, but still convert them to a dataframe for readability."
      ],
      "metadata": {
        "id": "BahXeSfFkJIb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtain scan/series metadata for a collection as JSON for use in next example\n",
        "data = nbia.getSeries(collection = \"AREN0532\", api_url = \"restricted\")\n",
        "\n",
        "# format as dataframe for easy viewing\n",
        "df = pd.DataFrame(data)\n",
        "display(df)"
      ],
      "metadata": {
        "id": "VQVbwxCEkJIb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we can use the results from the getSeries() query to generate some summary statistics about the scans in the collection."
      ],
      "metadata": {
        "id": "X8p4rjHCkJIb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate summary statistics for a given collection \n",
        "nbia.makeSeriesReport(data)"
      ],
      "metadata": {
        "id": "BYS-sdd7kJIb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.3 Downloading data with the REST API\n",
        "Next we'll cover using the API to download data.  This can be useful if you'd like to download results from API queries rather than using an existing manifest file.  It's also useful if you can't install the NBIA Data Retriever or want to integrate TCIA downloads into other pipelines/tools.  \n",
        "\n",
        "As a reminder, many of the scans in the AREN0532 Collection were not annotated by the authors of https://doi.org/10.7937/KJA4-1Z76.  The reasons are outlined in the Annotation Protocol on that page.  As a result, you may want to download only a subset of the scans such as:\n",
        "\n",
        "1. Seed point labels \n",
        "2. 3d segmentation labels \n",
        "3. Source images used to create seed points and segmentations\n",
        "4. Source images with negative finding assessments\n",
        "\n",
        "The following examples demonstrate how to tackle each of these use cases. "
      ],
      "metadata": {
        "id": "aWUOGHyLkJIb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3.1 tcia_utils download functions\n",
        "**tcia_utils** contains a **downloadSeries()** function that has multiple options for specifying the seriesUids you'd like to download.  By default, the function expects JSON data containing \"SeriesInstanceUID\" elements, which can be generated using **getSeries()** or **getCart()**.  However, if you have a series UID list from some other source, you can set **input_type = \"list\"** to pass a python list of one or more series UIDs instead of JSON. \n",
        "\n",
        "Data are saved to a **tciaDownload** folder in your current working directory by default, but you can use the **path** parameter to change this to a different directory.\n",
        "\n",
        "There is an optional **format** parameter that can be used to return metadata about what was downloaded.  It can be set to **df** to return dataframe or **csv** to save a spreadsheet. There's also a **csv_filename** parameter if you want to set a specific file name.\n",
        "\n",
        "You can specify **number = n** to tell the function to only download the first **n** scans of your seriesUids.  Remove this parameter in the examples below to download the full dataset. \n",
        "\n",
        "The **api_url** parameter can be omitted in most cases.  However, it must be set to **api_url = \"nlst\"** to access the [National Lung Screening Trial (NLST)](https://doi.org/10.7937/TCIA.HMQ8-J677) collection and you must use **api_url = \"restricted\"** for datasets that require logging in.  \n",
        "\n",
        "Last but not least, there is some logic built in to detect whether you've already downloaded a series.  If a directory named after the seriesUid already exists the function will assume it's already been downloaded and skip that series."
      ],
      "metadata": {
        "id": "woFF4omPkJIb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3.2 Download a sample subject\n",
        "First, let's re-use the **AREN0532-PAPVBY.tcia** file we downloaded earlier to demonstrate how we can work with manifest files to download images via the API.\n",
        "\n",
        "If you open this manifest file in a text editor you'll notice that it contains several lines of download parameters that precede a list of Series Instance UIDs to download.  The **manifestToList()** function will put the Series UIDs into a Python list and ignore the parameters in the first six lines of the manifest so that we can feed it to **nbia.downloadSeries()**."
      ],
      "metadata": {
        "id": "LwdU_1TjkJIb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# enter manifest path/filename\n",
        "manifest = \"AREN0532-PAPVBY.tcia\"\n",
        "\n",
        "# converts manifest to list of UIDs\n",
        "uids = nbia.manifestToList(manifest)"
      ],
      "metadata": {
        "id": "kjXZG-YodVuv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can download the series in the list and return the metadata to a dataframe. "
      ],
      "metadata": {
        "id": "UG--djUgH2UA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = nbia.downloadSeries(uids, input_type = \"list\", api_url = \"restricted\", format = \"df\")\n"
      ],
      "metadata": {
        "id": "9s1l6qFpeSvd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here's what the metadata looks like in the dataframe."
      ],
      "metadata": {
        "id": "2GMyfBFIakHz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(df)"
      ],
      "metadata": {
        "id": "0GU-MlmQaZ9R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Visualize the data\n",
        "You can preview a series that you've downloaded directly in your notebook using the **viewSeries()** function.  This function requires EITHER a seriesUid or path parameter.  Leave the seriesUid empty if you want to provide a custom path.  The function assumes \"tciaDownload/**seriesUid**/\" as the path if a seriesUid is provided since this is where downloadSeries() saves data.\n",
        "\n",
        "**Note:** This function only works with regular scans and cannot be used to visualize the annotation data."
      ],
      "metadata": {
        "id": "L0dxkzIep2Fe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# view a sample scan we've downloaded using a Series UID from the previous dataframe\n",
        "nbia.viewSeries(\"1.3.6.1.4.1.14519.5.2.1.1610.1216.996552775034540775693160619298\")\n",
        "\n",
        "# example showing how to use the path parameter instead of a UID\n",
        "# nbia.viewSeries(path = \"tciaDownload/1.3.6.1.4.1.14519.5.2.1.1610.1216.996552775034540775693160619298\")"
      ],
      "metadata": {
        "id": "iWjXulUFqiWt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3.3 Download subsets of the data\n",
        "To identify the subsets for the other use cases, we'll leverage the **annotation metadata** spreadsheet the authors provided, which you can download from https://doi.org/10.7937/kja4-1z76 or retrieve directly into a dataframe with the code below."
      ],
      "metadata": {
        "id": "VZuiv-1ykJIc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load annotation metadata spreadsheet to dataframe\n",
        "\n",
        "annotation_Metadata = pd.read_csv('https://wiki.cancerimagingarchive.net/download/attachments/145752341/Metadata_Report_AREN0532_01122023.csv?api=v2')\n",
        "\n",
        "display(annotation_Metadata)"
      ],
      "metadata": {
        "id": "LmO_29HhkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download seed points\n",
        "Since we're working with Series UIDs from a dataframe instead of JSON output from the API, we'll use the  **input_type = \"list\"** parameter in the remaining download steps.  Options to download a sample (three scans) or the entire dataset are provided.  We'll also specify a **csv_filename** to save the related metadata to a file."
      ],
      "metadata": {
        "id": "kgkWxt-QUM3i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include seed point rows\n",
        "seedPoints = annotation_Metadata[annotation_Metadata['Annotation Type'].str.contains('Seed point')]\n",
        "#display(seedPoints)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = seedPoints[\"SeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans \n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata \n",
        "nbia.downloadSeries(series_data, number = 3, api_url = \"restricted\", input_type = \"list\", csv_filename = \"seedPoints\")"
      ],
      "metadata": {
        "id": "gIXTLO5dkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download 3D segmentations"
      ],
      "metadata": {
        "id": "1irWBSZvU05T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include segmentations\n",
        "segs = annotation_Metadata[annotation_Metadata['Annotation Type'].str.contains('Segmentation')]\n",
        "#display(segs)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = segs[\"SeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans \n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata \n",
        "nbia.downloadSeries(series_data, number = 3, api_url = \"restricted\", input_type = \"list\", csv_filename = \"acns0332_SEGs\")"
      ],
      "metadata": {
        "id": "_lG5TO7wkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download source images for seed points and segmentations"
      ],
      "metadata": {
        "id": "9XsnZm2hkJIc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include seg and seed point rows (remove \"no findings\")\n",
        "ref_Series = annotation_Metadata[(annotation_Metadata['Annotation Type'] == 'Seed point') |\n",
        "                                 (annotation_Metadata['Annotation Type'] == 'Segmentation')]\n",
        "\n",
        "# remove duplicate ReferencedSeriesUIDs\n",
        "clean_refSeries = ref_Series.drop_duplicates(subset='ReferencedSeriesInstanceUID')\n",
        "#display(clean_refSeries)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = clean_refSeries[\"ReferencedSeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans \n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata \n",
        "nbia.downloadSeries(series_data, number = 3, api_url = \"restricted\", input_type = \"list\", csv_filename = \"seg_seed_source_images\")"
      ],
      "metadata": {
        "id": "O4ivqjOHkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following code will download the scans with negative finding assessments.  These are cases where the authors of the dataset did not find anything that could be annotated.  Downloading these scans could be useful if you are training a tumor/metastases detection model."
      ],
      "metadata": {
        "id": "f6exl7ORkJIc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include scans with \"no findings\"\n",
        "ref_Series = annotation_Metadata[annotation_Metadata['Annotation Type'] == 'No findings']\n",
        "\n",
        "# remove duplicate ReferencedSeriesUIDs\n",
        "clean_refSeries = ref_Series.drop_duplicates(subset='ReferencedSeriesInstanceUID')\n",
        "#display(clean_refSeries)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = clean_refSeries[\"ReferencedSeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans\n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata \n",
        "nbia.downloadSeries(series_data, number = 3, api_url = \"restricted\", input_type = \"list\", csv_filename = \"noFinding_source_images\")"
      ],
      "metadata": {
        "id": "enSo8BLikJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Acknowledgements\n",
        "[The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) is a service which de-identifies and hosts a large publicly available archive of medical images of cancer.  TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/), and is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/).\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/).  If you leverage TCIA datasets in your work please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). Upon receiving access, you must also abide by the terms of your NCTN/NCORP Data Archive’s Data Use Agreement (DUA). You are not allowed to redistribute the data or use it for other purposes. Attribution should include references to the following citations:\n",
        "\n",
        "## Data Citations\n",
        "\n",
        "1. Fernandez, C. V., Mullen, E. A., Chi, Y.-Y., Ehrlich, P. F., Perlman, E. J., Kalapurakal, J. A., Khanna, G., Paulino, A. C., Hamilton, T. E., Gow, K. W., Tochner, Z., Hoffer, F. A., Withycombe, J. S., Shamberger, R. C., Kim, Y., Geller, J. I., Anderson, J. R., Grundy, P. E., & Dome, J. S. (2022). Vincristine, Dactinomycin, and Doxorubicin With or Without Radiation Therapy or Observation Only in Treating Younger Patients Who Are Undergoing Surgery for Newly Diagnosed Stage I, Stage II, or Stage III Wilms’ Tumor (AREN0532) (Version 1) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/6PJ1-M859\n",
        "2. Rozenfeld, M., & Jordan, P. (2023). Annotations for Vincristine, Dactinomycin, and Doxorubicin With or Without Radiation Therapy or Observation Only in Treating Younger Patients Who Are Undergoing Surgery for Newly Diagnosed Stage I, II, or III Wilms' Tumor (AREN0532-Tumor-Annotations) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/KJA4-1Z76\n",
        "\n",
        "## Publication Citation\n",
        "\n",
        "Fernandez, C. V., Mullen, E. A., Chi, Y.-Y., Ehrlich, P. F., Perlman, E. J., Kalapurakal, J. A., Khanna, G., Paulino, A. C., Hamilton, T. E., Gow, K. W., Tochner, Z., Hoffer, F. A., Withycombe, J. S., Shamberger, R. C., Kim, Y., Geller, J. I., Anderson, J. R., Grundy, P. E., & Dome, J. S. (2018). Outcome and Prognostic Factors in Stage III Favorable-Histology Wilms Tumor: A Report From the Children’s Oncology Group Study AREN0532. In Journal of Clinical Oncology (Vol. 36, Issue 3, pp. 254–261). American Society of Clinical Oncology (ASCO). https://doi.org/10.1200/jco.2017.73.7999\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ],
      "metadata": {
        "id": "DCOM4lO4_tET"
      }
    }
  ]
}