Notebooks/Quick_Start_Guide_to_ISB_CGC.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "cells": [
{
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/isb-cgc/Community-Notebooks/blob/master/Notebooks/Quick_Start_Guide_to_ISB_CGC.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },    
{
      "cell_type": "markdown",
      "metadata": {
        "id": "V_oh9DspLamP"
      },
      "source": [
        "# ISB-CGC Community Notebooks\n",
        "\n",
        "Check out more notebooks at our [Community Notebooks Repository](https://github.com/isb-cgc/Community-Notebooks)!\n",
        "\n",
        "```\n",
        "Title:   Quick Start Guide to ISB-CGC\n",
        "Author:  Lauren Hagen\n",
        "Created: 2019-06-20\n",
        "Updated: 2023-08\n",
        "Purpose: Painless intro to working with ISB-CGC in the cloud\n",
        "URL:     https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/Quick_Start_Guide_to_ISB_CGC.ipynb\n",
        "Notes:   This Quick Start Guide gives an overview of the data available in ISB-CGC and getting started with a basic example in python.\n",
        "```\n",
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35Ew4aYCspKn"
      },
      "source": [
        "# Quick Start Guide to [ISB-CGC](https://isb-cgc.appspot.com/) in BigQuery\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xlx2svEHfLfN"
      },
      "source": [
        "## Account Set-up\n",
        "To run this notebook, you will need to have your Google Cloud Account set up. If you need to set up a Google Cloud Account, follow the \"Obtain a Google identity\" and \"Set up a Google Cloud Project\" steps on our [Quick-Start Guide documentation](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html) page.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJs8FZBJ8x1f"
      },
      "source": [
        "## Libraries needed for the Notebook\n",
        "This notebook requires the BigQuery API to be loaded [(click here for more information)](https://googleapis.github.io/google-cloud-python/latest/bigquery/usage/client.html) allowing access to BigQuery programmatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "usHb94n-L0a5"
      },
      "source": [
        "# GCP libraries\n",
        "from google.cloud import bigquery\n",
        "from google.colab import auth"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OAgOtUGpvR5h"
      },
      "source": [
        "## Overview of ISB-CGC\n",
        "The ISB-CGC provides interactive and programmatic access to data hosted by institutes such as the [Genomic Data Commons (GDC)](https://gdc.cancer.gov/) and [Proteomic Data Commons (PDC)](https://proteomic.datacommons.cancer.gov/pdc/) from the [National Cancer Institute (NCI)](https://www.cancer.gov/) while leveraging many aspects of the Google Cloud Platform. You can also import your data, analyze it side by side with the datasets, and share your data when you see fit. The ISB-CGC hosts carefully curated high-level clinical, biospecimen, and molecular datasets and tables in Google BigQuery, including data from programs such as [The Cancer Genome Atlas (TCGA)](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga), [Therapeutically Applicable Research to Generate Effective Treatments (TARGET)](https://ocg.cancer.gov/programs/target), and [Clinical Proteomic Tumor Analysis Consortium (CPTAC)](https://proteomics.cancer.gov/programs/cptac). For more information can be found at our [Programs and Data Sets page](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/Hosted-Data.html). This data can be explored via python, [Google Cloud Console](https://console.cloud.google.com/) and/or our [BigQuery Table Search tool](https://isb-cgc.appspot.com/bq_meta_search/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLkFocDF265b"
      },
      "source": [
        "## Example of Accessing BigQuery Data with Python\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwUePpm5uLNS"
      },
      "source": [
        "### Log into Google Cloud Storage and Authenticate ourselves\n",
        "\n",
        "Steps to authenticate yourself:\n",
        "1. Run the code block to authenticate yourself with your Google Cloud Login\n",
        "2. A second tab will open or follow the link provided\n",
        "3. Follow prompts to Authorize your account to use Google Cloud SDK\n",
        "4. Copy code provided and paste into the box under the Command\n",
        "5. Press Enter\n",
        "\n",
        "[Alternative authentication methods](https://googleapis.github.io/google-cloud-python/latest/core/auth.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CzjAwn9w3Ghc"
      },
      "source": [
        "# if you're using Google Colab, authenticate to gcloud with the following\n",
        "auth.authenticate_user()\n",
        "\n",
        "# alternatively, use the gcloud SDK\n",
        "#!gcloud auth application-default login"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating a client and using a billing project\n",
        "\n",
        "To access BigQuery, you will need a Google Cloud Project for queries to be billed to. If you need to create a Project, instructions on how to create one can be found on our [Quick-Start Guide page](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html).\n",
        "\n",
        "A BigQuery Client object with the billing Project needs to be created to interface with BigQuery.\n",
        "\n",
        "> Note: Any costs that you incur are charged under your current project, so you will want to make sure you are on the correct one if you are part of multiple projects.\n"
      ],
      "metadata": {
        "id": "7yZFt49ggjb5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a variable for which client to use with BigQuery\n",
        "project_id = 'YOUR_PROJECT_ID_CHANGE_ME' # Update with your Google Project Id"
      ],
      "metadata": {
        "id": "FCH4HzmTWC15"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a BigQuery Client\n",
        "if project_id == 'YOUR_PROJECT_ID_CHANGE_ME': # checking that project id was changed\n",
        "  print('Please update the project number with your Google Cloud Project')\n",
        "else: client = bigquery.Client(project_id)"
      ],
      "metadata": {
        "id": "t8J9mSHGSPJq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lqtslyr2zNTA"
      },
      "source": [
        "### View ISB-CGC Datasets and Tables in BigQuery\n",
        "Let us look at the datasets available through ISB-CGC that are in BigQuery."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "usDiQye0PWPF"
      },
      "source": [
        "# Which project to view datasets\n",
        "project_with_data = 'isb-cgc-bq'\n",
        "\n",
        "# Create a variable of datasets\n",
        "datasets = list(client.list_datasets(project_with_data))\n",
        "\n",
        "# If there are datasets available then print their names,\n",
        "# else print that there are no datasets available\n",
        "if datasets:\n",
        "    print(f\"Datasets in project {project_with_data}:\")\n",
        "    for dataset in datasets:  # API request(s)\n",
        "        print(\"\\t{}\".format(dataset.dataset_id))\n",
        "else:\n",
        "    print(f\"{project_with_data} project does not contain any datasets.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcKkXP0H8W6I"
      },
      "source": [
        "The ISB-CGC has two datasets for each Program or source. One dataset contains the most current data, and the other contains versioned tables, which serve as an archive for reproducibility. The current tables are labeled with \"_current\" and are updated when new data is released. For more information, visit our [ISB-CGC BigQuery Projects](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/BigQuery/ISBCGC-BQ-Projects.html) page. Let's see which tables are under the TCGA dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IZ_Odw0z1fOn"
      },
      "source": [
        "dataset_with_data = 'TCGA_versioned'\n",
        "\n",
        "print(\"Tables:\")\n",
        "# Create a variable with the list of tables in the dataset\n",
        "tables = list(client.list_tables(f'{project_with_data}.{dataset_with_data}'))\n",
        "\n",
        "# If there are tables then print their names,\n",
        "# else print that there are no tables\n",
        "if tables:\n",
        "    for table in tables:\n",
        "        print(\"\\t{}\".format(table.table_id))\n",
        "else:\n",
        "    print(\"\\tThis dataset does not contain any tables.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "c8Hp8lUMYjsm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjXF-Vg4z5mv"
      },
      "source": [
        "### Query ISB-CGC BigQuery Tables\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this section, we will create a string variable with our SQL then call to BigQuery and save the result to a dataframe."
      ],
      "metadata": {
        "id": "3aa4opf4XvJe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Syntax for the query\n",
        "```\n",
        "SELECT # Select a few columns to view\n",
        "  proj__project_id, # GDC project\n",
        "  submitter_id, # case barcode\n",
        "  proj__name # GDC project name\n",
        "FROM # Which table in BigQuery in the format of `project.dataset.table`\n",
        "  `project_name.dataset_name.table_name` # From the GDC TCGA Clinical Dataset\n",
        "LIMIT\n",
        "  5 # Limit to 5 rows as the dataset is very large and we only want to see a few results\n",
        "```\n",
        "\n",
        "> Note: `LIMIT` only limits the number of rows returned and not the number of rows that the query looks at\n"
      ],
      "metadata": {
        "id": "Uzn8oXZfkBLQ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TYf4eBOKnKEz"
      },
      "source": [
        "query = (\"\"\"\n",
        "  SELECT\n",
        "    proj__project_id,\n",
        "    submitter_id,\n",
        "    proj__name\n",
        "  FROM\n",
        "    `isb-cgc-bq.TCGA_versioned.clinical_gdc_r37`\n",
        "  LIMIT\n",
        "    5\"\"\")\n",
        "result = client.query(query).to_dataframe()  # API request\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4rbvEDj6xfMT"
      },
      "source": [
        "## Resources\n",
        "There are several ways to access and explore the data hosted by ISB-CGC.\n",
        "\n",
        "* ISB-CGC\n",
        "  * [ISB-CGC WebApp](https://isb-cgc.appspot.com/)\n",
        "    * Provides a graphical interface to file and case data\n",
        "    * Cohort creation\n",
        "    * File exploration\n",
        "  * [ISB-CGC BigQuery Table Search](https://isb-cgc.appspot.com/bq_meta_search/)\n",
        "    * Provides a table search for available ISB-CGC BigQuery Tables\n",
        "  * [ISB-CGC APIs](https://api-dot-isb-cgc.appspot.com/v4/swagger/)\n",
        "    * Provides programmatic access to metadata\n",
        "\n",
        "* Google Cloud\n",
        "  * [Google Cloud Platform](https://cloud.google.com/)\n",
        "    * Access and store data in [Google Cloud Storage](https://cloud.google.com/storage) and [BigQuery](https://cloud.google.com/bigquery) via User Interfaces or programmatically\n",
        "    \n",
        "* Suggested Programming Languages and Programs to use\n",
        " * SQL\n",
        "    * Can be used directly in [BigQuery Console](https://console.cloud.google.com/bigquery)\n",
        "    * Or via API in Python or R\n",
        " * [Python](https://www.python.org/)\n",
        "    * [gsutil tool](https://cloud.google.com/storage/docs/gsutil)\n",
        "    * [Jupyter Notebooks](https://jupyter.org/)\n",
        "    * [Google Colabratory](https://colab.research.google.com/)\n",
        "    * [Cloud Datalab](https://cloud.google.com/datalab/)\n",
        " * [R](https://www.r-project.org/)\n",
        "    * [RStudio](https://rstudio.com/)\n",
        "    * [RStudio.Cloud](https://rstudio.cloud/)\n",
        "* Command Line Interfaces\n",
        "    * Cloud Shell via Project Console\n",
        "    * [CLOUD SDK](https://cloud.google.com/sdk/)\n",
        "* Getting Started for Free:\n",
        "    * [Free Cloud Credits from ISB-CGC for Cancer Research](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowtoRequestCloudCredits.html)\n",
        "    * [Google Free Tier with up to 1TB of free queries a month](https://cloud.google.com/free)\n",
        "\n",
        "Useful ISB-CGC Links:\n",
        "\n",
        "* [ISB-CGC Landing Page](https://isb-cgc.appspot.com/)\n",
        "* [ISB-CGC Documentation](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/)\n",
        "* [How to Get Started on ISB-CGC](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html)\n",
        "* [How to access Google BigQuery](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/progapi/bigqueryGUI/HowToAccessBigQueryFromTheGoogleCloudPlatform.html)\n",
        "* [Community Notebook Repository](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowTos.html)\n",
        "\n",
        "Useful Google Tutorials:\n",
        "\n",
        "* [Google's What is BigQuery?](https://cloud.google.com/bigquery/docs/introduction)\n",
        "* [Google Cloud Client Library for Python](https://googleapis.github.io/google-cloud-python/latest/index.html)"
      ]
    }
  ]
}