notebooks/sr2_2018-05-04-Gonad-Methylation-Full-Samples-Bismark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bismark\n",
    "\n",
    "In this notebook I'll use Bismark on the full extent of my data. The steps from the [user guide](https://rawgit.com/FelixKrueger/Bismark/master/Docs/Bismark_User_Guide.html#i-bismark-genome-preparation) are as follows:\n",
    "\n",
    "1. Genome Preparation (already done in [this Jupyter notebook](https://github.com/RobertsLab/project-virginica-oa/blob/master/notebooks/2018-04-27-Gonad-Methylation-Bismark.ipynb))\n",
    "2. Alignment\n",
    "3. Deduplication\n",
    "4. Methlyation Extractor\n",
    "5. HTML Processing Report\n",
    "6. Summary Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T16:05:24.309733Z",
     "start_time": "2018-05-16T16:05:24.304096Z"
    }
   },
   "outputs": [],
   "source": [
    "bmk=\"/Applications/bioinfo/Bismark_v0.19.0/\"\n",
    "gnm=\"/Users/sr320/Dropbox/wd/18-03-15/genome\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Genome Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step was already completed in [this Jupyter notebook](https://github.com/RobertsLab/project-virginica-oa/blob/master/notebooks/2018-04-27-Gonad-Methylation-Bismark.ipynb). The genome only needs to be prepared once. I will move on to the second step, alignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Alignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " The trimmed *C. virginica* sequences I need to use are in [this OWL folder](http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/). To run the `find` + `xargs` commands I want, I need to specify this path. I'm going to run this piece of code first just to confirm that I'm selecting the right files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create data subdirectory that will ignore in GitHub - then curl files to said directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T13:50:55.559983Z",
     "start_time": "2018-05-16T13:50:55.437339Z"
    }
   },
   "outputs": [],
   "source": [
    "mkdir data/dignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T13:51:53.891679Z",
     "start_time": "2018-05-16T13:51:53.885376Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore\n"
     ]
    }
   ],
   "source": [
    "cd data/dignore\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T14:54:22.739093Z",
     "start_time": "2018-05-16T14:50:11.117537Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[1/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_1_s1_R1_val_1.fq.gz --> zr2096_1_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_1_s1_R1_val_1.fq.gz\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1089M  100 1089M    0     0  38.9M      0  0:00:28  0:00:28 --:--:-- 54.0M\n",
      "\n",
      "[2/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_2_s1_R1_val_1.fq.gz --> zr2096_2_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_2_s1_R1_val_1.fq.gz\n",
      "100 1096M  100 1096M    0     0  42.1M      0  0:00:26  0:00:26 --:--:-- 46.9M4  706M    0     0  50.4M      0  0:00:21  0:00:14  0:00:07 54.8M\n",
      "\n",
      "[3/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_3_s1_R1_val_1.fq.gz --> zr2096_3_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_3_s1_R1_val_1.fq.gz\n",
      "100 1100M  100 1100M    0     0  45.8M      0  0:00:24  0:00:24 --:--:-- 31.2M\n",
      "\n",
      "[4/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_4_s1_R1_val_1.fq.gz --> zr2096_4_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_4_s1_R1_val_1.fq.gz\n",
      "100  883M  100  883M    0     0  30.4M      0  0:00:29  0:00:29 --:--:-- 16.5M 883M   87  777M    0     0  35.3M      0  0:00:25  0:00:22  0:00:03 32.3M\n",
      "\n",
      "[5/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_5_s1_R1_val_1.fq.gz --> zr2096_5_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_5_s1_R1_val_1.fq.gz\n",
      "100 1171M  100 1171M    0     0  41.8M      0  0:00:28  0:00:28 --:--:-- 15.0M\n",
      "\n",
      "[6/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_6_s1_R1_val_1.fq.gz --> zr2096_6_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_6_s1_R1_val_1.fq.gz\n",
      "100  868M  100  868M    0     0  41.3M      0  0:00:21  0:00:21 --:--:-- 49.1M\n",
      "\n",
      "[7/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_7_s1_R1_val_1.fq.gz --> zr2096_7_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_7_s1_R1_val_1.fq.gz\n",
      "100 1115M  100 1115M    0     0  46.4M      0  0:00:24  0:00:24 --:--:-- 47.7M\n",
      "\n",
      "[8/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_8_s1_R1_val_1.fq.gz --> zr2096_8_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_8_s1_R1_val_1.fq.gz\n",
      "100 1104M  100 1104M    0     0  50.2M      0  0:00:22  0:00:22 --:--:-- 52.3M\n",
      "\n",
      "[9/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_9_s1_R1_val_1.fq.gz --> zr2096_9_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_9_s1_R1_val_1.fq.gz\n",
      "100 1313M  100 1313M    0     0  46.9M      0  0:00:28  0:00:28 --:--:-- 68.8M\n",
      "\n",
      "[10/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_10_s1_R1_val_1.fq.gz --> zr2096_10_s1_R1_val_1.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_10_s1_R1_val_1.fq.gz\n",
      "100  722M  100  722M    0     0  45.1M      0  0:00:16  0:00:16 --:--:-- 56.8M  0  42.7M      0  0:00:16  0:00:08  0:00:08 42.7M\n"
     ]
    }
   ],
   "source": [
    "!curl -O http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_[1-10]_s1_R1_val_1.fq.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T15:04:29.361673Z",
     "start_time": "2018-05-16T14:57:26.584463Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[1/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_1_s1_R2_val_2.fq.gz --> zr2096_1_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_1_s1_R2_val_2.fq.gz\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1107M  100 1107M    0     0  35.7M      0  0:00:31  0:00:31 --:--:-- 22.8M 998M    0     0  39.9M      0  0:00:27  0:00:25  0:00:02 32.8M\n",
      "\n",
      "[2/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_2_s1_R2_val_2.fq.gz --> zr2096_2_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_2_s1_R2_val_2.fq.gz\n",
      "100 1095M  100 1095M    0     0  35.3M      0  0:00:31  0:00:31 --:--:-- 33.1M1095M    9  108M    0     0  54.2M      0  0:00:20  0:00:02  0:00:18 47.3M\n",
      "\n",
      "[3/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_3_s1_R2_val_2.fq.gz --> zr2096_3_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_3_s1_R2_val_2.fq.gz\n",
      "100 1110M  100 1110M    0     0  37.0M      0  0:00:30  0:00:30 --:--:-- 34.8M\n",
      "\n",
      "[4/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_4_s1_R2_val_2.fq.gz --> zr2096_4_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_4_s1_R2_val_2.fq.gz\n",
      "100  883M  100  883M    0     0  42.0M      0  0:00:21  0:00:21 --:--:-- 60.0M\n",
      "\n",
      "[5/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_5_s1_R2_val_2.fq.gz --> zr2096_5_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_5_s1_R2_val_2.fq.gz\n",
      "100 1175M  100 1175M    0     0  41.9M      0  0:00:28  0:00:28 --:--:-- 57.5M50.8M\n",
      "\n",
      "[6/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_6_s1_R2_val_2.fq.gz --> zr2096_6_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_6_s1_R2_val_2.fq.gz\n",
      "100  873M  100  873M    0     0  31.2M      0  0:00:28  0:00:28 --:--:-- 18.3M  577M    0     0  32.0M      0  0:00:27  0:00:18  0:00:09 36.9M\n",
      "\n",
      "[7/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_7_s1_R2_val_2.fq.gz --> zr2096_7_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_7_s1_R2_val_2.fq.gz\n",
      "100 1114M  100 1114M    0     0  16.6M      0  0:01:07  0:01:07 --:--:-- 17.3M4M   35  400M    0     0  16.7M      0  0:01:06  0:00:24  0:00:42 16.3M\n",
      "\n",
      "[8/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_8_s1_R2_val_2.fq.gz --> zr2096_8_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_8_s1_R2_val_2.fq.gz\n",
      "100 1109M  100 1109M    0     0  16.0M      0  0:01:09  0:01:09 --:--:-- 18.2M    0     0  15.0M      0  0:01:13  0:00:44  0:00:29 18.2M\n",
      "\n",
      "[9/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_9_s1_R2_val_2.fq.gz --> zr2096_9_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_9_s1_R2_val_2.fq.gz\n",
      "100 1274M  100 1274M    0     0  17.4M      0  0:01:13  0:01:13 --:--:-- 18.1M  18.9M      0  0:01:07  0:00:44  0:00:23 25.1M\n",
      "\n",
      "[10/10]: http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_10_s1_R2_val_2.fq.gz --> zr2096_10_s1_R2_val_2.fq.gz\n",
      "--_curl_--http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_10_s1_R2_val_2.fq.gz\n",
      "100  709M  100  709M    0     0  18.1M      0  0:00:39  0:00:39 --:--:-- 22.7M\n"
     ]
    }
   ],
   "source": [
    "\n",
    "!curl -O http://owl.fish.washington.edu/Athaliana/20180411_trimgalore_10bp_Cvirginica_MBD/zr2096_[1-10]_s1_R2_val_2.fq.gz\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T15:49:53.813114Z",
     "start_time": "2018-05-16T15:49:53.648869Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 43250048\r\n",
      "-rw-r--r--  1 sr320  staff   723M May 16 07:54 zr2096_10_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   709M May 16 08:04 zr2096_10_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:50 zr2096_1_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:57 zr2096_1_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:51 zr2096_2_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:58 zr2096_2_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:51 zr2096_3_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:59 zr2096_3_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   884M May 16 07:51 zr2096_4_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   884M May 16 07:59 zr2096_4_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:52 zr2096_5_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:59 zr2096_5_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   869M May 16 07:52 zr2096_6_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   874M May 16 08:00 zr2096_6_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:53 zr2096_7_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 08:01 zr2096_7_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 07:53 zr2096_8_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.1G May 16 08:02 zr2096_8_s1_R2_val_2.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.3G May 16 07:54 zr2096_9_s1_R1_val_1.fq.gz\r\n",
      "-rw-r--r--  1 sr320  staff   1.2G May 16 08:03 zr2096_9_s1_R2_val_2.fq.gz\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What I need for this command:\n",
    "\n",
    "1. Find all the *C. virginica* sequence files using `find`\n",
    "2. Isolate the base name from each of these files using `xargs` that I will then pipe into...\n",
    "\n",
    "*drumroll*\n",
    "\n",
    "1. ...The path to `bismark`\n",
    "2. --non_directional: See [this issue](https://github.com/RobertsLab/resources/issues/216)\n",
    "3. -p 4: Number of simultaneous threads to run\n",
    "4. --genome + path to the folder with the .fa genome, which also has all of the bisulfite genome directories\n",
    "5. -1 + Path to first paired file in the Athaliana folder\n",
    "6. -2 + Path to second paired file in the Athaliana folder\n",
    "7. Path to redirect standard error output.\n",
    "\n",
    "Note that I'm not including a `-score_min` option, since I want to just run this alignment with the default setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T15:59:43.363741Z",
     "start_time": "2018-05-16T15:56:31.766684Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t185\n",
      "Mapping efficiency:\t18.5%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t707\n",
      "Sequence pairs did not map uniquely:\t108\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t50\t((converted) top strand)\n",
      "GA/CT/CT:\t48\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t49\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t38\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t91\n",
      "Mapping efficiency:\t9.1%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t864\n",
      "Sequence pairs did not map uniquely:\t45\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t24\t((converted) top strand)\n",
      "GA/CT/CT:\t21\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t23\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t23\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t176\n",
      "Mapping efficiency:\t17.6%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t717\n",
      "Sequence pairs did not map uniquely:\t107\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t47\t((converted) top strand)\n",
      "GA/CT/CT:\t28\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t46\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t55\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t194\n",
      "Mapping efficiency:\t19.4%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t684\n",
      "Sequence pairs did not map uniquely:\t122\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t46\t((converted) top strand)\n",
      "GA/CT/CT:\t57\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t46\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t45\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t195\n",
      "Mapping efficiency:\t19.5%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t676\n",
      "Sequence pairs did not map uniquely:\t129\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t47\t((converted) top strand)\n",
      "GA/CT/CT:\t49\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t50\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t49\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t182\n",
      "Mapping efficiency:\t18.2%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t722\n",
      "Sequence pairs did not map uniquely:\t96\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t43\t((converted) top strand)\n",
      "GA/CT/CT:\t46\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t44\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t49\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t217\n",
      "Mapping efficiency:\t21.7%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t664\n",
      "Sequence pairs did not map uniquely:\t119\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t67\t((converted) top strand)\n",
      "GA/CT/CT:\t44\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t52\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t54\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t211\n",
      "Mapping efficiency:\t21.1%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t695\n",
      "Sequence pairs did not map uniquely:\t94\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t73\t((converted) top strand)\n",
      "GA/CT/CT:\t62\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t38\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t38\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t175\n",
      "Mapping efficiency:\t17.5%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t725\n",
      "Sequence pairs did not map uniquely:\t100\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t45\t((converted) top strand)\n",
      "GA/CT/CT:\t41\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t47\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t42\t((converted) bottom strand)\n",
      "\n",
      "FastQ format assumed (by default)\n",
      "Each Bowtie 2 instance is going to be run with 4 threads. Please monitor performance closely and tune down if needed!\n",
      "chr NC_035780.1 (65668440 bp)\n",
      "chr NC_035781.1 (61752955 bp)\n",
      "chr NC_035782.1 (77061148 bp)\n",
      "chr NC_035783.1 (59691872 bp)\n",
      "chr NC_035784.1 (98698416 bp)\n",
      "chr NC_035785.1 (51258098 bp)\n",
      "chr NC_035786.1 (57830854 bp)\n",
      "chr NC_035787.1 (75944018 bp)\n",
      "chr NC_035788.1 (104168038 bp)\n",
      "chr NC_035789.1 (32650045 bp)\n",
      "chr NC_007175.2 (17244 bp)\n",
      "\n",
      "Number of paired-end alignments with a unique best hit:\t164\n",
      "Mapping efficiency:\t16.4%\n",
      "\n",
      "Sequence pairs with no alignments under any condition:\t765\n",
      "Sequence pairs did not map uniquely:\t71\n",
      "Sequence pairs which were discarded because genomic sequence could not be extracted:\t0\n",
      "\n",
      "Number of sequence pairs with unique best (first) alignment came from the bowtie output:\n",
      "CT/GA/CT:\t50\t((converted) top strand)\n",
      "GA/CT/CT:\t32\t(complementary to (converted) top strand)\n",
      "GA/CT/GA:\t28\t(complementary to (converted) bottom strand)\n",
      "CT/GA/GA:\t54\t((converted) bottom strand)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "find zr2096_*R1*.fq.gz \\\n",
    "| xargs basename -s _s1_R1_val_1.fq.gz | xargs -I{} /Applications/bioinfo/Bismark_v0.19.0/bismark \\\n",
    "--non_directional \\\n",
    "-u 1000 \\\n",
    "-p 4 \\\n",
    "--genome /Users/sr320/Dropbox/wd/18-03-15/genome \\\n",
    "-1 {}_s1_R1_val_1.fq.gz \\\n",
    "-2 {}_s1_R2_val_2.fq.gz \\\n",
    "2> ../../analyses/0516_bismark.err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T16:04:59.660078Z",
     "start_time": "2018-05-16T16:04:59.509490Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zr2096_10_s1_R1_val_1.fq.gz\r\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_10_s1_R2_val_2.fq.gz\r\n",
      "zr2096_1_s1_R1_val_1.fq.gz\r\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_1_s1_R2_val_2.fq.gz\r\n",
      "zr2096_2_s1_R1_val_1.fq.gz\r\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_2_s1_R2_val_2.fq.gz\r\n",
      "zr2096_3_s1_R1_val_1.fq.gz\r\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_3_s1_R2_val_2.fq.gz\r\n",
      "zr2096_4_s1_R1_val_1.fq.gz\r\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_4_s1_R2_val_2.fq.gz\r\n",
      "zr2096_5_s1_R1_val_1.fq.gz\r\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_5_s1_R2_val_2.fq.gz\r\n",
      "zr2096_6_s1_R1_val_1.fq.gz\r\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_6_s1_R2_val_2.fq.gz\r\n",
      "zr2096_7_s1_R1_val_1.fq.gz\r\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_7_s1_R2_val_2.fq.gz\r\n",
      "zr2096_8_s1_R1_val_1.fq.gz\r\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_8_s1_R2_val_2.fq.gz\r\n",
      "zr2096_9_s1_R1_val_1.fq.gz\r\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_PE_report.txt\r\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.bam\r\n",
      "zr2096_9_s1_R2_val_2.fq.gz\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The alignment is done! Now to deduplication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deduplication, Sorting, and Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3a. Deduplication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step wasn't explicitly laid out in the User Guide. The deduplication step removes any PCR artifacts from the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what I need to run this command:\n",
    "\n",
    "1. Path to `deduplicate_bismark`\n",
    "2. -p: Input data is paired\n",
    "3. --bam: Write the output as a .bam file\n",
    "4. Path to input files. It's important to specify only the files from the alignment! If I just used .bam, the process would become recursive, and the command will start to deduplicate the .deduplicated.bam files that are already deduplicated!\n",
    "5. Path to standard error output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T16:07:50.128224Z",
     "start_time": "2018-05-16T16:07:28.079809Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Now testing Bismark result file zr2096_10_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_10_s1_R1_val_1.fq.gz -2 zr2096_10_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_1_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_1_s1_R1_val_1.fq.gz -2 zr2096_1_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_2_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_2_s1_R1_val_1.fq.gz -2 zr2096_2_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_3_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_3_s1_R1_val_1.fq.gz -2 zr2096_3_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_4_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_4_s1_R1_val_1.fq.gz -2 zr2096_4_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_5_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_5_s1_R1_val_1.fq.gz -2 zr2096_5_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_6_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_6_s1_R1_val_1.fq.gz -2 zr2096_6_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_7_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_7_s1_R1_val_1.fq.gz -2 zr2096_7_s1_R2_val_2.fq.gz\"\n",
      "\n",
      "Now testing Bismark result file zr2096_8_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_8_s1_R1_val_1.fq.gz -2 zr2096_8_s1_R2_val_2.fq.gz\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Now testing Bismark result file zr2096_9_s1_R1_val_1_bismark_bt2_pe.bam for positional sorting (which would be bad...)\tskipping header line:\t@HD\tVN:1.0\tSO:unsorted\n",
      "skipping header line:\t@SQ\tSN:NC_035780.1\tLN:65668440\n",
      "skipping header line:\t@SQ\tSN:NC_035781.1\tLN:61752955\n",
      "skipping header line:\t@SQ\tSN:NC_035782.1\tLN:77061148\n",
      "skipping header line:\t@SQ\tSN:NC_035783.1\tLN:59691872\n",
      "skipping header line:\t@SQ\tSN:NC_035784.1\tLN:98698416\n",
      "skipping header line:\t@SQ\tSN:NC_035785.1\tLN:51258098\n",
      "skipping header line:\t@SQ\tSN:NC_035786.1\tLN:57830854\n",
      "skipping header line:\t@SQ\tSN:NC_035787.1\tLN:75944018\n",
      "skipping header line:\t@SQ\tSN:NC_035788.1\tLN:104168038\n",
      "skipping header line:\t@SQ\tSN:NC_035789.1\tLN:32650045\n",
      "skipping header line:\t@SQ\tSN:NC_007175.2\tLN:17244\n",
      "skipping header line:\t@PG\tID:Bismark\tVN:v0.19.0\tCL:\"bismark --non_directional -u 1000 -p 4 --genome /Users/sr320/Dropbox/wd/18-03-15/genome -1 zr2096_9_s1_R1_val_1.fq.gz -2 zr2096_9_s1_R2_val_2.fq.gz\"\n"
     ]
    }
   ],
   "source": [
    "\n",
    "! $bmk/deduplicate_bismark \\\n",
    "-p \\\n",
    "--bam \\\n",
    "zr2096_*_s1_R1_val_1_bismark_bt2_pe.bam \\\n",
    "2> ../../analyses/0516_dedup.err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T19:17:58.031171Z",
     "start_time": "2018-05-16T19:17:57.904577Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam\r\n"
     ]
    }
   ],
   "source": [
    "ls *deduplicated.bam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Methylation Extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "1. Path to `bismark_methylation_extractor`\n",
    "2. -p: Indicate I have paired reads\n",
    "3. --bedGraph: Create `bedGraph` file with results\n",
    "4. --counts\n",
    "5. --scaffolds: Bypass limitation of open filehandles\n",
    "6. --multicore 8\n",
    "7. --gzip: Gunzip files to save space\n",
    "8. Path to .bam files generated in Step 2\n",
    "9. Path for standard error output file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T21:37:01.110488Z",
     "start_time": "2018-05-16T21:34:13.708415Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Now testing Bismark result file zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n",
      "\n",
      "Now testing Bismark result file zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.bam for positional sorting (which would be bad...)\tzr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.1.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.2.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.3.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.4.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.5.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.6.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.7.mbias\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated_splitting_report.txt.8.mbias\n",
      "\n",
      "/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OT_zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOT_zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_CTOB_zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\t/Users/sr320/Documents/GitHub/project-virginica-oa/data/dignore/CpG_OB_zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.txt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! $bmk/bismark_methylation_extractor \\\n",
    "-p \\\n",
    "--bedGraph \\\n",
    "--counts \\\n",
    "--scaffolds \\\n",
    "--multicore 8 \\\n",
    "*deduplicated.bam \\\n",
    "2> ../../analyses/0516_bme.err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-16T21:54:01.495025Z",
     "start_time": "2018-05-16T21:54:01.368087Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zr2096_10_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_1_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_2_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_3_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_4_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_5_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_6_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_7_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_8_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n",
      "zr2096_9_s1_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz\r\n"
     ]
    }
   ],
   "source": [
    "!ls *bedGraph*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}