12S/How_to_download_12S_Rawdata_from_SRA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "\n",
    "Illumina data can be downloaded from SRA using the following workflow.\n",
    "\n",
    "It will requre the [SRA-toolkit](http://www.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc) program `fastq-dump` to be installed on your machine and in your path.\n",
    "\n",
    "If the necessary program is not yet present in your system you can get it as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/working\r\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Download the latest version of sratoolkit__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "%%bash\n",
    "\n",
    "mkdir SRA-toolkit\n",
    "cd SRA-toolkit\n",
    "\n",
    "wget ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-centos_linux64.tar.gz\n",
    "\n",
    "#decompress the archive\n",
    "tar -xzf sratoolkit.current-centos_linux64.tar.gz\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sratoolkit.2.9.0-centos_linux64  sratoolkit.current-centos_linux64.tar.gz\r\n"
     ]
    }
   ],
   "source": [
    "!ls SRA-toolkit/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/working\r\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "\n",
    "#add the fastq-dump executable to your PATH (may require sudo)\n",
    "cp SRA-toolkit/sratoolkit.2.9.0-centos_linux64/bin/fastq-dump /usr/local/bin\n",
    "\n",
    "#remove SRA toolkit files that are not required\n",
    "\n",
    "rm -rf SRA-toolkit/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The following commmand will read through the file Sample_accessions.tsv (see [here](https://github.com/HullUni-bioinformatics/Li_et_al_2018_eDNA_fish_monitoring/blob/master/12S/supplementary_data/Sample_accessions.tsv)) and download the relevant raw read files from SRA basd on the accessions provided in column 3 of the file. It will then rename the files according to the sampleID (column 1) and the marker (column 2).Then save the raw data to 'raw_data' folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!mkdir raw_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/working/raw_data\n"
     ]
    }
   ],
   "source": [
    "cd raw_data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PEN\tSRR7106552\t12S\n",
      "Read 407502 spots for SRR7106552\n",
      "Written 407502 spots for SRR7106552\n",
      "LLG\tSRR7106553\t12S\n",
      "Read 836076 spots for SRR7106553\n",
      "Written 836076 spots for SRR7106553\n",
      "MAP\tSRR7106554\t12S\n",
      "Read 751669 spots for SRR7106554\n",
      "Written 751669 spots for SRR7106554\n",
      "CAM\tSRR7106555\t12S\n",
      "Read 591654 spots for SRR7106555\n",
      "Written 591654 spots for SRR7106555\n",
      "PAD\tSRR7106556\t12S\n",
      "Read 668293 spots for SRR7106556\n",
      "Written 668293 spots for SRR7106556\n",
      "CWE\tSRR7106557\t12S\n",
      "Read 706834 spots for SRR7106557\n",
      "Written 706834 spots for SRR7106557\n",
      "BET\tSRR7106558\t12S\n",
      "Read 572517 spots for SRR7106558\n",
      "Written 572517 spots for SRR7106558\n",
      "OSS\tSRR7106559\t12S\n",
      "Read 634771 spots for SRR7106559\n",
      "Written 634771 spots for SRR7106559\n",
      "FEN\tSRR7106560\t12S\n",
      "Read 543376 spots for SRR7106560\n",
      "Written 543376 spots for SRR7106560\n",
      "WLF\tSRR7106561\t12S\n",
      "Read 693112 spots for SRR7106561\n",
      "Written 693112 spots for SRR7106561\n",
      "TRA\tSRR7106562\t12S\n",
      "Read 806784 spots for SRR7106562\n",
      "Written 806784 spots for SRR7106562\n",
      "OGW\tSRR7106563\t12S\n",
      "Read 1306919 spots for SRR7106563\n",
      "Written 1306919 spots for SRR7106563\n",
      "KEN\tSRR7106564\t12S\n",
      "Read 550122 spots for SRR7106564\n",
      "Written 550122 spots for SRR7106564\n",
      "LLB\tSRR7106565\t12S\n",
      "Read 515571 spots for SRR7106565\n",
      "Written 515571 spots for SRR7106565\n"
     ]
    }
   ],
   "source": [
    "\n",
    "%%bash\n",
    "\n",
    "for s in $(cat ../supplementary_data/Sample_accessions.tsv | sed 's/\\t/,/g' | grep \"SRA_accession\" -v)\n",
    "do \n",
    "    id=$(echo $s | cut -d \",\" -f 1)\n",
    "    accession=$(echo $s | cut -d \",\" -f 3)\n",
    "    marker=$(echo $s | cut -d \",\" -f 2)\n",
    "    echo -e \"$id\\t$accession\\t$marker\"\n",
    "    fastq-dump --split-files --gzip --defline-seq '@$ac-$sn/$ri' --defline-qual '+' $accession\n",
    "    mv $accession\\_1.fastq.gz $id-$marker\\_R1.fastq.gz\n",
    "    mv $accession\\_2.fastq.gz $id-$marker\\_R2.fastq.gz\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Double check if all files are there.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BET-12S_R1.fastq.gz  LLB-12S_R1.fastq.gz  PAD-12S_R1.fastq.gz\r\n",
      "BET-12S_R2.fastq.gz  LLB-12S_R2.fastq.gz  PAD-12S_R2.fastq.gz\r\n",
      "CAM-12S_R1.fastq.gz  LLG-12S_R1.fastq.gz  PEN-12S_R1.fastq.gz\r\n",
      "CAM-12S_R2.fastq.gz  LLG-12S_R2.fastq.gz  PEN-12S_R2.fastq.gz\r\n",
      "CWE-12S_R1.fastq.gz  MAP-12S_R1.fastq.gz  TRA-12S_R1.fastq.gz\r\n",
      "CWE-12S_R2.fastq.gz  MAP-12S_R2.fastq.gz  TRA-12S_R2.fastq.gz\r\n",
      "FEN-12S_R1.fastq.gz  OGW-12S_R1.fastq.gz  WLF-12S_R1.fastq.gz\r\n",
      "FEN-12S_R2.fastq.gz  OGW-12S_R2.fastq.gz  WLF-12S_R2.fastq.gz\r\n",
      "KEN-12S_R1.fastq.gz  OSS-12S_R1.fastq.gz\r\n",
      "KEN-12S_R2.fastq.gz  OSS-12S_R2.fastq.gz\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}