Guide_to_Running.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Guide to Running"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main Data Analysis\n",
    "\n",
    "Here is where I do the main data analysis for the manuscript.   \n",
    "* [__Unsupervised Age HIV Analysis__](./Unsupervised_Age_HIV_Analysis.ipynb)   \n",
    "Unsupervised analysis of age associated probes with the aim of showing a shared influence of age and HIV on the methylome. This contains the code for the generation of Figure 1. \n",
    "* [__HIV Age Advancment__](./HIV_Age_Advancement.ipynb)  \n",
    "Here I read in the data and run the methylation age models.  \n",
    "* [__HIV Age Advancement: Confounders__](./HIV_Age_Advancement_Confounders.ipynb)  \n",
    "Here I am looking at confounding from patients' blood composition as well as association of age advancment with other clinical variables that we have available. \n",
    "* [__Figure 2__](./Figure2.ipynb)  \n",
    "Generation of Figure 2 for the manuscript. \n",
    "* [__Validation_figure__](./Validation/Validation_Figure.ipynb)   \n",
    "Generation of Figure 3 of the manuscript. \n",
    "* [__Figure 3__](./Figure3.ipynb)   \n",
    "Generation of Figure 4 for the manuscript. Figure 3 got added in revisions, so I'm keeping the name consistent to not lose the version control. \n",
    "* [__Figure 5_top__](./new_fig_5_top.ipynb)  \n",
    "Generation of Figure 5a and 5b for the manuscript. Also includes a look at general disorder in response to age and HIV as well as some post-hoc analysis on the HLA and sourounding regions. \n",
    "* [__Figure_5_bottom__](./Validation/new_fig_5_bottom.ipynb)   \n",
    "Generation of Figure 5c-f for the manuscript."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks, QC, ect. \n",
    "\n",
    "Quality control and benchmarking important to the manuscript, but not necissarily directly refered to in the paper.\n",
    "\n",
    "* [__Model Comparison__](./Benchmarks/Model_Comparison_MF.ipynb)  \n",
    "Comparison of the Hannum and Horvath epigenetic aging models and compilation statistics on each across multiple datasets. I then construct a consensus model and use the agreement of the two models as to filter out subjects that may not have good methylation data. \n",
    "* [__Sorted Cell Benchmark__](./Benchmarks/Sorted_Cell_Benchmark.ipynb)  \n",
    "Analysis of methylation age models in populations of sorted cells. \n",
    "* [__Validation Cohort Power Calculation__](./Other/power_calculation.ipynb)  \n",
    "Power calculation done prior to vaidation of the age advancment effect in a smaller cohort of patients with DNA extracted from purified cells. \n",
    "* [__CD14 aging__](./Other/CD14_aging.ipynb)  \n",
    "Analysis of epigenetic aging in 1202 patients with DNA extracted from a purified population of monocytes from [this paper](http://www.nature.com/ncomms/2014/141118/ncomms6366/full/ncomms6366.html).   \n",
    "* [__Lupus cohort, multiple sorted cell types__](./Other/Lupus_Cohort.ipynb)  \n",
    "Here I look at epigenetic aging in a case/control Lupus cohort. I pulled this dataset because it contains a group of patients with sorted (purified cell populations) blood samples profiled across multiple different cell types. It shows that the epigenetic aging models are fairly robust across cell types. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis Setup Notebooks\n",
    "\n",
    "These are notebooks which I generally run as upstream dependencies of other notebooks, think of them as python modules. I include these as notebook rather than modules because there are some global variables ad-hoc decisions being invoked and I do not think it is appropriate to abstract away into modules. Is this the best software development process? Likely not, but this is data-analysis and this is also grad-ware, both of which have a certain level of sloppiness which is more or less unavoidable. \n",
    "* [__Imports__](./Setup/Imports.ipynb)  \n",
    "Imports packages into global namespace, sets up a few helper functions, and global path variables.  __If you are running this yourself you must [change the global parameters here](Setup/Imports.ipynb#Global-Parameters)__. \n",
    "* [__DX Imports__](./Setup/DX_Imports.ipynb)  \n",
    "Sets up some helper functions for doing analysis of differential methylation and reads in probe annotations for methylation arrays. \n",
    "* [__Methylation Age Models__](./Setup/MethylationAgeModels.ipynb)  \n",
    "Reads in the linear data for the two epigenetic aging models and creates helper functions to apply each model across a set of patients. \n",
    "* [__Read HIV Data__](./Setup/Read_HIV_Data.ipynb)  \n",
    "This reads in the methylation data for our primary case-control cohort as well a lot of associated clinical data.  I also do a little pre-filtering based on basic selection criteria (age range, HAART status, gender)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Parallel Jobs for Linear Models\n",
    "\n",
    "I have created various scripts for running linear models on the methylation data.  With about a half of a million probes on the chip, this can take a while to do in series so I use a relatively primative map-reduce paradigm. \n",
    "* A dataset is saved in 100 chunks in HDF5 format\n",
    "* Covariates are saved in the same data-store  \n",
    "* A python script to run the linear models is hard coded with the table chunk number as an input parameter\n",
    "* An SGE driver script is created to run a parallel job \n",
    "* The parallel job is run on the cluster \n",
    "* The results are merged back together on the local machine  \n",
    "\n",
    "In the [Parallel](./Parallel) folder, I have a number of notebooks which contain the hard coded python script and the code for generating the SGE driver. Our cluster shares its filesystem with our network storage, so for applying this to different systems some modifications to the protocol are likely needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre Processing\n",
    "\n",
    "Here I am doing some pre-processing on the raw datasets to make them more usable for downstream analysis.  \n",
    "\n",
    "\n",
    "This stuff is pretty computationally intensive.  If you want to run this part do the following things:  \n",
    "* Ask yourself if you really need to do this or if you can start with the data already processed (I can provide this in a nice small binary file which is much easier to handle). \n",
    "* Get yourself a pretty high memory machine. You are going to need to do quantile-normalization on about 2000 450k arrays.  This took up about 100GB in memory.  You could probably figure out how to do this out of memory but its only really a one time thing so probably not necessary. \n",
    "* Download the raw EPIC dataset from GEO [link](http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE51032).  \n",
    "* Contact me for the raw dataset from Hannum et al. (This data is on GEO but not in the most raw form... I will try and fix this soon)\n",
    "\n",
    "[__Metylation Normalization Protocol__](./PreProcessing/Methylation_Normalization_MINFI.ipynb)  \n",
    "This mainly uses functions provided in the [R bioconductor minfi package](http://bioconductor.org/packages/release/bioc/html/minfi.html). \n",
    "* Read in all of the raw .IDAT methylation files for our datasets as well as the two controls \n",
    "* Extract detection p-values for all probes across all samples\n",
    "* Run quantile normalization on the concatinated dataset\n",
    "* Run estimateCellCounts on the concatinated dataset\n",
    "* Save the results of this analysis \n",
    "\n",
    "#### Saving things in HDF5 Format\n",
    "\n",
    "These 450k arrays can get pretty unweildy due to their size when trying to do in-memory data analysis.  To help out with this and generally speed up I/O I like to store these data as [Pandas HDF5](http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables) data objects.  This data format has the advantage of being much faster than handling everything in .csvs like you would generally do with expression datasets.\n",
    "\n",
    "* [__save detection p values__](./PreProcessing/save_detection_p_values.ipynb) \n",
    "* [__compile probe annotations__](./PreProcessing/Compile_Probe_Annotations.ipynb) \n",
    "* [__Pan Study Data__](./PreProcessing/Pan_Study_Data.ipynb)\n",
    "\n",
    "#### BMIQ Processing\n",
    "\n",
    "We run quantile normalization in the minfi processing pipeline but it has been recomended to also run BMIQ on this normalized data.  In addition the processing protocol for use of the Horvath aging model requires a modified verision of this.  This is done for various datasets in the following notebooks. \n",
    "\n",
    "* [__BMIQ Horvath__](./PreProcessing/BMIQ_Horvath.ipynb) \n",
    "* [__BMIQ Normalization__](./PreProcessing/BMIQ_Normalization.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}