Example Notebooks/K-means Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-means Clustering in GenePattern Notebook\n",
    "\n",
    "Cluster genes and/or samples into a specified number of clusters. The result is k clusters, each centered around a randomly selected data point. \n",
    "\n",
    "## Before you begin\n",
    "\n",
    "* Sign in to GenePattern by entering your username and password into the form below. If you are seeing a block of code instead of the login form, go to the menu above and select Cell > Run All.\n",
    "* Gene expression data must be in a [GCT or RES file](https://genepattern.broadinstitute.org/gp/pages/protocols/GctResFiles.html).\n",
    "    * Example file: [all_aml_test.gct](https://datasets.genepattern.org/data/all_aml/all_aml_test.gct).\n",
    "* Learn more by reading about [file formats](http://www.broadinstitute.org/cancer/software/genepattern/file-formats-guide#GCT).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "genepattern": {
     "server": "https://genepattern.broadinstitute.org/gp",
     "type": "auth"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "341d51b6dc354f62a804a375c1c33823"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Requires GenePattern Notebook: pip install genepattern-notebook\n",
    "import gp\n",
    "import genepattern\n",
    "\n",
    "# Username and password removed for security reasons.\n",
    "genepattern.GPAuthWidget(genepattern.register_session(\"https://genepattern.broadinstitute.org/gp\", \"\", \"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: PreprocessDataset\n",
    "\n",
    "Preprocess gene expression data to remove platform noise and genes that have little variation. Although researchers generally preprocess data before clustering if doing so removes relevant biological information, skip this step. \n",
    "\n",
    "### Considerations\n",
    "\n",
    "* PreprocessDataset can preprocess the data in one or more ways (in this order):\n",
    "    1. Set threshold and ceiling values. Any value lower/higher than the threshold/ceiling value is reset to the threshold/ceiling value.\n",
    "    2. Convert each expression value to the log base 2 of the value.\n",
    "    3. Remove genes (rows) if a given number of its sample values are less than a given threshold.\n",
    "    4. Remove genes (rows) that do not have a minimum fold change or expression variation.\n",
    "    5. Discretize or normalize the data.\n",
    "* When using ratios to compare gene expression between samples, convert values to log base 2 of the value to bring up- and down-regulated genes to the same scale. For example, ratios of 2 and .5 indicating two-fold changes for up- and down-regulated expression, respectively, are converted to +1 and -1. \n",
    "* If you did not generate the expression data, check whether preprocessing steps have already been taken before running the PreprocessDataset module. \n",
    "* Learn more by reading about the [PreprocessDataset](https://genepattern.broadinstitute.org/gp/getTaskDoc.jsp?name=PreprocessDataset) module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "genepattern": {
     "type": "task"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca98f93c93184567a78af6fd986a156c"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preprocessdataset_task = gp.GPTask(genepattern.get_session(0), 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00020')\n",
    "preprocessdataset_job_spec = preprocessdataset_task.make_job_spec()\n",
    "preprocessdataset_job_spec.set_parameter(\"input.filename\", \"https://datasets.genepattern.org/data/all_aml/all_aml_test.gct\")\n",
    "preprocessdataset_job_spec.set_parameter(\"threshold.and.filter\", \"1\")\n",
    "preprocessdataset_job_spec.set_parameter(\"floor\", \"20\")\n",
    "preprocessdataset_job_spec.set_parameter(\"ceiling\", \"20000\")\n",
    "preprocessdataset_job_spec.set_parameter(\"min.fold.change\", \"3\")\n",
    "preprocessdataset_job_spec.set_parameter(\"min.delta\", \"100\")\n",
    "preprocessdataset_job_spec.set_parameter(\"num.outliers.to.exclude\", \"0\")\n",
    "preprocessdataset_job_spec.set_parameter(\"row.normalization\", \"0\")\n",
    "preprocessdataset_job_spec.set_parameter(\"row.sampling.rate\", \"1\")\n",
    "preprocessdataset_job_spec.set_parameter(\"threshold.for.removing.rows\", \"\")\n",
    "preprocessdataset_job_spec.set_parameter(\"number.of.columns.above.threshold\", \"\")\n",
    "preprocessdataset_job_spec.set_parameter(\"log2.transform\", \"0\")\n",
    "preprocessdataset_job_spec.set_parameter(\"output.file.format\", \"3\")\n",
    "preprocessdataset_job_spec.set_parameter(\"output.file\", \"<input.filename_basename>.preprocessed\")\n",
    "genepattern.GPTaskWidget(preprocessdataset_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: KMeansClustering\n",
    "\n",
    "Run k-means clustering on genes (rows) or samples (columns). The module creates a GCT file for each cluster and a GCT file that organizes all of the expression data by cluster. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "genepattern": {
     "type": "task"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6af8268cfd9a4e0bb4a7b9ad6976fd3c"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "kmeansclustering_task = gp.GPTask(genepattern.get_session(0), 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00081')\n",
    "kmeansclustering_job_spec = kmeansclustering_task.make_job_spec()\n",
    "kmeansclustering_job_spec.set_parameter(\"input.filename\", \"https://datasets.genepattern.org/data/all_aml/all_aml_test.preprocessed.gct\")\n",
    "kmeansclustering_job_spec.set_parameter(\"output.base.name\", \"<input.filename_basename>_KMcluster_output\")\n",
    "kmeansclustering_job_spec.set_parameter(\"number.of.clusters\", \"2\")\n",
    "kmeansclustering_job_spec.set_parameter(\"seed.value\", \"12345\")\n",
    "kmeansclustering_job_spec.set_parameter(\"cluster.by\", \"0\")\n",
    "kmeansclustering_job_spec.set_parameter(\"distance.metric\", \"0\")\n",
    "genepattern.GPTaskWidget(kmeansclustering_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: HeatMapViewer\n",
    "\n",
    "For an overview of the results, use a heatmap to display the expression data organized by cluster. \n",
    "\n",
    "### Considerations\n",
    "\n",
    "* The HeatMapViewer displays gene expression data as a heat map, which makes it easier to see patterns in the numeric data. Gene names are row labels and sample names are column labels. \n",
    "* Learn more by reading about the [HeatMapViewer](https://genepattern.broadinstitute.org/gp/getTaskDoc.jsp?name=HeatMapViewer) module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "genepattern": {
     "type": "task"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd00ab3f107045af87e6b277adb1572a"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmapviewer_task = gp.GPTask(genepattern.get_session(0), 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.visualizer:00010')\n",
    "heatmapviewer_job_spec = heatmapviewer_task.make_job_spec()\n",
    "heatmapviewer_job_spec.set_parameter(\"dataset\", \"\")\n",
    "genepattern.GPTaskWidget(heatmapviewer_task)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "widgets": {
   "state": {
    "4140a768171c481da4045f8704d89199": {
     "views": [
      {
       "cell_index": 5
      }
     ]
    },
    "ac9fcc74bc5d439192bec1e190a2e924": {
     "views": [
      {
       "cell_index": 1
      }
     ]
    },
    "e7f6bb70f8a1414fab89d73abb6d6d41": {
     "views": [
      {
       "cell_index": 5
      }
     ]
    },
    "f727586e26094082909432db37c5bfd4": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}