notebooks/sam/20170622_oly_pacbio_data_management.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Jun 22 14:26:20 PDT 2017\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Software:\n",
      "\n",
      "    System Software Overview:\n",
      "\n",
      "      System Version: Mac OS X 10.7.5 (11G63)\n",
      "      Kernel Version: Darwin 11.4.2\n",
      "      Boot Volume: SSD2\n",
      "      Boot Mode: Normal\n",
      "      Computer Name: greenbird (2)\n",
      "      User Name: Sam (Sam)\n",
      "      Secure Virtual Memory: Enabled\n",
      "      64-bit Kernel and Extensions: No\n",
      "      Time since boot: 38 days 3:04\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "system_profiler SPSoftwareDataType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "      Model Name: Mac Pro\n",
      "      Model Identifier: MacPro1,1\n",
      "      Processor Name: Dual-Core Intel Xeon\n",
      "      Processor Speed: 3 GHz\n",
      "      Number of Processors: 2\n",
      "      Total Number of Cores: 4\n",
      "      L2 Cache (per Processor): 4 MB\n",
      "      Memory: 14 GB\n",
      "      Bus Speed: 1.33 GHz\n",
      "      Boot ROM Version: MP11.005C.B08\n",
      "      SMC Version (system): 1.7f10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "#Uses grep to exclude lines that display serial number and hardware UUID\n",
    "system_profiler SPHardwareDataType | grep -v [SH][ea]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "The goal of this notebook is to copy the PacBio fastq.gz files for the oly genome sequencing project to the ```/owl/nightingales/O_lurida``` folder to be in compliance with our data management plan. Since these files do not have unique names (they're all named the same thing, but are stored in different subdirectories), they need to be renamed. Additionally, to confirm that the files were copied and renamed correctly, an md5 checkusm verification needs to take place.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create md5 checksums of original gzipped fastq files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd /Volumes/owl/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use find to locate all the gzipped fastq files, create an md5 checksum, and write the output from each file to a new file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t0m0.011s\n",
      "user\t0m0.001s\n",
      "sys\t0m0.005s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time find . -maxdepth 2 -name \"*.fastq.gz\" -exec md5 {} + > md5checksums_fastq.gz.md5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that didn't work. I think it didn't change directories. Should have caught this after executing cell #1 above, since there was no output listed. Let's see where I am..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/Sam/GitRepos/LabDocs/jupyter_nbs/sam\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yep, didn't change directories. I think it's because I used the ```bash``` shell magics for the ```cd``` command. Annoying!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Volumes/owl/nightingales/O_lurida/20170323_pacbio\n"
     ]
    }
   ],
   "source": [
    "cd /Volumes/owl/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use find to locate all the gzipped fastq files, create an md5 checksum, and write the output from each file to a new file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t3m56.425s\n",
      "user\t0m27.842s\n",
      "sys\t0m43.239s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time find . -maxdepth 2 -name \"*.fastq.gz\" -exec md5 {} + > md5checksums_fastq.gz.md5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD5 (./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/filtered_subreads.fastq.gz) = cc30d6e17499a36960dab140991af9b7\n",
      "MD5 (./170228_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz) = c0f549d4867c2d45f0a4c5ebb65a7163\n",
      "MD5 (./170228_PCB-CC_AL_20kb_P6v2_D01_1/filtered_subreads.fastq.gz) = 601c182bceec111f286b1a75f9276613\n",
      "MD5 (./170228_PCB-CC_AL_20kb_P6v2_E01_1/filtered_subreads.fastq.gz) = 1d8c89d18d976d82f977f53f11657c96\n",
      "MD5 (./170307_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz) = c000bb09dd10ea85028cd73ba1f8ce51\n",
      "MD5 (./170307_PCB-CC_AL_20kb_P6v2_C02_1/filtered_subreads.fastq.gz) = 7f387f334adad2c11f71c4216b0d8fe2\n",
      "MD5 (./170314_PCB-CC_20kb_P6v2_A01_1/filtered_subreads.fastq.gz) = c21ce30b76f57485a3cbe8712a623bd5\n",
      "MD5 (./170314_PCB-CC_20kb_P6v2_A02_1/filtered_subreads.fastq.gz) = 20b0b0743fe0214bde319d2d40a40067\n",
      "MD5 (./170314_PCB-CC_20kb_P6v2_A03_1/filtered_subreads.fastq.gz) = 511866d0fc60fe3e8bb6dea4ce87a64f\n",
      "MD5 (./170314_PCB-CC_20kb_P6v2_A04_1/filtered_subreads.fastq.gz) = 206b30a4b231541a85d4707f0b907153\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat md5checksums_fastq.gz.md5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy, rename, and generate new md5 checksums"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Contents of xml_array:\n",
      "./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/m170211_224036_42134_c101073082550000001823236402101737_s1_X0.metadata.xml\n",
      "./170228_PCB-CC_AL_20kb_P6v2_C01_1/m170301_100013_42134_c101174162550000001823269408211761_s1_p0.metadata.xml\n",
      "./170228_PCB-CC_AL_20kb_P6v2_D01_1/m170301_162825_42134_c101174162550000001823269408211762_s1_p0.metadata.xml\n",
      "./170228_PCB-CC_AL_20kb_P6v2_E01_1/m170301_225711_42134_c101174162550000001823269408211763_s1_p0.metadata.xml\n",
      "./170307_PCB-CC_AL_20kb_P6v2_C01_1/m170308_163922_42134_c101174252550000001823269408211742_s1_p0.metadata.xml\n",
      "./170307_PCB-CC_AL_20kb_P6v2_C02_1/m170308_230815_42134_c101174252550000001823269408211743_s1_p0.metadata.xml\n",
      "./170314_PCB-CC_20kb_P6v2_A01_1/m170315_001112_42134_c101169372550000001823273008151717_s1_p0.metadata.xml\n",
      "./170314_PCB-CC_20kb_P6v2_A02_1/m170315_063041_42134_c101169382550000001823273008151700_s1_p0.metadata.xml\n",
      "./170314_PCB-CC_20kb_P6v2_A03_1/m170315_124938_42134_c101169382550000001823273008151701_s1_p0.metadata.xml\n",
      "./170314_PCB-CC_20kb_P6v2_A04_1/m170315_190851_42134_c101169382550000001823273008151702_s1_p0.metadata.xml\n",
      "\n",
      "-------------\n",
      "Contents of xml_array:\n",
      "./170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1/filtered_subreads.fastq.gz\n",
      "./170228_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz\n",
      "./170228_PCB-CC_AL_20kb_P6v2_D01_1/filtered_subreads.fastq.gz\n",
      "./170228_PCB-CC_AL_20kb_P6v2_E01_1/filtered_subreads.fastq.gz\n",
      "./170307_PCB-CC_AL_20kb_P6v2_C01_1/filtered_subreads.fastq.gz\n",
      "./170307_PCB-CC_AL_20kb_P6v2_C02_1/filtered_subreads.fastq.gz\n",
      "./170314_PCB-CC_20kb_P6v2_A01_1/filtered_subreads.fastq.gz\n",
      "./170314_PCB-CC_20kb_P6v2_A02_1/filtered_subreads.fastq.gz\n",
      "./170314_PCB-CC_20kb_P6v2_A03_1/filtered_subreads.fastq.gz\n",
      "./170314_PCB-CC_20kb_P6v2_A04_1/filtered_subreads.fastq.gz\n",
      "\n",
      "-------------\n",
      "Contents of fastq_nopath_array:\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "filtered_subreads.fastq.gz\n",
      "\n",
      "-------------\n",
      "Contents of xml_nopath_array:\n",
      "m170211_224036_42134_c101073082550000001823236402101737_s1_X0.metadata.xml\n",
      "m170301_100013_42134_c101174162550000001823269408211761_s1_p0.metadata.xml\n",
      "m170301_162825_42134_c101174162550000001823269408211762_s1_p0.metadata.xml\n",
      "m170301_225711_42134_c101174162550000001823269408211763_s1_p0.metadata.xml\n",
      "m170308_163922_42134_c101174252550000001823269408211742_s1_p0.metadata.xml\n",
      "m170308_230815_42134_c101174252550000001823269408211743_s1_p0.metadata.xml\n",
      "m170315_001112_42134_c101169372550000001823273008151717_s1_p0.metadata.xml\n",
      "m170315_063041_42134_c101169382550000001823273008151700_s1_p0.metadata.xml\n",
      "m170315_124938_42134_c101169382550000001823273008151701_s1_p0.metadata.xml\n",
      "m170315_190851_42134_c101169382550000001823273008151702_s1_p0.metadata.xml\n",
      "\n",
      "-------------\n",
      "Contents of xml_nosuffix_array:\n",
      "m170211_224036_42134_c101073082550000001823236402101737_s1_X0\n",
      "m170301_100013_42134_c101174162550000001823269408211761_s1_p0\n",
      "m170301_162825_42134_c101174162550000001823269408211762_s1_p0\n",
      "m170301_225711_42134_c101174162550000001823269408211763_s1_p0\n",
      "m170308_163922_42134_c101174252550000001823269408211742_s1_p0\n",
      "m170308_230815_42134_c101174252550000001823269408211743_s1_p0\n",
      "m170315_001112_42134_c101169372550000001823273008151717_s1_p0\n",
      "m170315_063041_42134_c101169382550000001823273008151700_s1_p0\n",
      "m170315_124938_42134_c101169382550000001823273008151701_s1_p0\n",
      "m170315_190851_42134_c101169382550000001823273008151702_s1_p0\n",
      "\n",
      "-------------\n",
      "/Volumes/owl/nightingales/O_lurida/m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170211_224036_42134_c101073082550000001823236402101737_s1_X0_filtered_subreads.fastq.gz) = cc30d6e17499a36960dab140991af9b7\n",
      "/Volumes/owl/nightingales/O_lurida/m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170301_100013_42134_c101174162550000001823269408211761_s1_p0_filtered_subreads.fastq.gz) = c0f549d4867c2d45f0a4c5ebb65a7163\n",
      "/Volumes/owl/nightingales/O_lurida/m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170301_162825_42134_c101174162550000001823269408211762_s1_p0_filtered_subreads.fastq.gz) = 601c182bceec111f286b1a75f9276613\n",
      "/Volumes/owl/nightingales/O_lurida/m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170301_225711_42134_c101174162550000001823269408211763_s1_p0_filtered_subreads.fastq.gz) = 1d8c89d18d976d82f977f53f11657c96\n",
      "/Volumes/owl/nightingales/O_lurida/m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170308_163922_42134_c101174252550000001823269408211742_s1_p0_filtered_subreads.fastq.gz) = c000bb09dd10ea85028cd73ba1f8ce51\n",
      "/Volumes/owl/nightingales/O_lurida/m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170308_230815_42134_c101174252550000001823269408211743_s1_p0_filtered_subreads.fastq.gz) = 7f387f334adad2c11f71c4216b0d8fe2\n",
      "/Volumes/owl/nightingales/O_lurida/m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170315_001112_42134_c101169372550000001823273008151717_s1_p0_filtered_subreads.fastq.gz) = c21ce30b76f57485a3cbe8712a623bd5\n",
      "/Volumes/owl/nightingales/O_lurida/m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170315_063041_42134_c101169382550000001823273008151700_s1_p0_filtered_subreads.fastq.gz) = 20b0b0743fe0214bde319d2d40a40067\n",
      "/Volumes/owl/nightingales/O_lurida/m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170315_124938_42134_c101169382550000001823273008151701_s1_p0_filtered_subreads.fastq.gz) = 511866d0fc60fe3e8bb6dea4ce87a64f\n",
      "/Volumes/owl/nightingales/O_lurida/m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq.gz\n",
      "MD5 (/Volumes/owl/nightingales/O_lurida/m170315_190851_42134_c101169382550000001823273008151702_s1_p0_filtered_subreads.fastq.gz) = 206b30a4b231541a85d4707f0b907153\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Find all xml files and store results in array\n",
    "xml_array=($(find . -maxdepth 2 -name \"*.xml\"))\n",
    "\n",
    "# Print contents of array, with some formatting for easier reading.\n",
    "echo \"Contents of xml_array:\"\n",
    "printf '%s\\n' \"${xml_array[@]}\"\n",
    "echo \"\"\n",
    "echo \"-------------\"\n",
    "\n",
    "# Find all fastq.gz files and store results in array\n",
    "fastq_array=($(find . -maxdepth 2 -name \"*.fastq.gz\"))\n",
    "echo \"Contents of xml_array:\"\n",
    "printf '%s\\n' \"${fastq_array[@]}\"\n",
    "echo \"\"\n",
    "echo \"-------------\"\n",
    "\n",
    "# Use parameter expansion to remove path from each component in fastq_array.\n",
    "# Store results in new array.\n",
    "fastq_nopath_array=($(echo \"${fastq_array[@]##*/}\"))\n",
    "\n",
    "# Print contents of array, with some formatting for easier reading.\n",
    "echo \"Contents of fastq_nopath_array:\"\n",
    "printf '%s\\n' \"${fastq_nopath_array[@]}\"\n",
    "echo \"\"\n",
    "echo \"-------------\"\n",
    "\n",
    "# Use parameter expansion to remove path from each component in xml_array.\n",
    "# Store results in new array.\n",
    "xml_nopath_array=($(echo \"${xml_array[@]##*/}\"))\n",
    "\n",
    "# Print contents of array, with some formatting for easier reading.\n",
    "echo \"Contents of xml_nopath_array:\"\n",
    "printf '%s\\n' \"${xml_nopath_array[@]}\"\n",
    "echo \"\"\n",
    "echo \"-------------\"\n",
    "\n",
    "# Use parameter expansion to remove the suffix (.xml) from each component in xml_nopath_array.\n",
    "# Store results in new array.\n",
    "xml_nosuffix_array=($(echo \"${xml_nopath_array[@]%%.*}\"))\n",
    "\n",
    "# Print contents of array, with some formatting for easier reading.\n",
    "echo \"Contents of xml_nosuffix_array:\"\n",
    "printf '%s\\n' \"${xml_nosuffix_array[@]}\"\n",
    "echo \"\"\n",
    "echo \"-------------\"\n",
    "\n",
    "# Loop through each index (i.e. the number corresponding to each element in the array).\n",
    "# Using the paths to each fastq.gz stored in the fastq_array, copy the fastq.gz file to the O_lurida nigthtingales folder &\n",
    "# use array elements to provide new name to copied file.\n",
    "# List the newly copied/named file to verify it got copied/renamed.\n",
    "# Create md5 checksums for each newly copied/renamed file and append to checksums.md5 file.\n",
    "# Use grep to verify info was written to checksums.md5 file\n",
    "for item in \"${!fastq_array[@]}\"; do\n",
    "    cp \"${fastq_array[$item]}\" /Volumes/owl/nightingales/O_lurida/\"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}\"\n",
    "    ls /Volumes/owl/nightingales/O_lurida/\"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}\"\n",
    "    md5 /Volumes/owl/nightingales/O_lurida/\"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}\" >> \\\n",
    "    /Volumes/owl/nightingales/O_lurida/checksums.md5\n",
    "    grep \"${xml_nosuffix_array[$item]}_${fastq_nopath_array[$item]}\" /Volumes/owl/nightingales/O_lurida/checksums.md5\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! I'm impressed with myself! It nearly all came out as intended!! Forgot to change the second ```echo``` statement to read \"Contents of fastq_array:\". Other than that, it all worked. Feels good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare md5 checksums\n",
    "\n",
    "#### Need to compare original checksums with that of the copied/renamed files to ensure file integrity didn't change during the copying process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quick test of using ```grep``` and ```awk``` to isolate just the checksum values..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-8-0e82f528bf91>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-8-0e82f528bf91>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    grep \"filtered_subreads.fastq.gz\" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print \"$4\"}'\u001b[0m\n\u001b[0m                                    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "grep \"filtered_subreads.fastq.gz\" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print \"$4\"}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n",
      "$4\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep \"filtered_subreads.fastq.gz\" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print \"$4\"}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cc30d6e17499a36960dab140991af9b7\n",
      "c0f549d4867c2d45f0a4c5ebb65a7163\n",
      "601c182bceec111f286b1a75f9276613\n",
      "1d8c89d18d976d82f977f53f11657c96\n",
      "c000bb09dd10ea85028cd73ba1f8ce51\n",
      "7f387f334adad2c11f71c4216b0d8fe2\n",
      "c21ce30b76f57485a3cbe8712a623bd5\n",
      "20b0b0743fe0214bde319d2d40a40067\n",
      "511866d0fc60fe3e8bb6dea4ce87a64f\n",
      "206b30a4b231541a85d4707f0b907153\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep \"filtered_subreads.fastq.gz\" /Volumes/owl/nightingales/O_lurida/checksums.md5 | awk '{print $4}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, I just looked through some previously used code and realized I didn't need to perform the above grep/awk test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "bash: line 2: unexpected EOF while looking for matching `)'\n",
      "bash: line 8: syntax error: unexpected end of file\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "original_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/20170323_pacbio/md5checksums_fastq.gz.md5))\n",
    "current_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/checksums.md5\n",
    "for ((i=0;i<=$count;++i))\n",
    "    do\n",
    "        printf \"%s\\n\" \"${original_md5[$i]}\"\n",
    "        printf \"%s\\n\\n\" \"${current_md5[$i]}\"\n",
    "    done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cc30d6e17499a36960dab140991af9b7\n",
      "cc30d6e17499a36960dab140991af9b7\n",
      "\n",
      "c0f549d4867c2d45f0a4c5ebb65a7163\n",
      "c0f549d4867c2d45f0a4c5ebb65a7163\n",
      "\n",
      "601c182bceec111f286b1a75f9276613\n",
      "601c182bceec111f286b1a75f9276613\n",
      "\n",
      "1d8c89d18d976d82f977f53f11657c96\n",
      "1d8c89d18d976d82f977f53f11657c96\n",
      "\n",
      "c000bb09dd10ea85028cd73ba1f8ce51\n",
      "c000bb09dd10ea85028cd73ba1f8ce51\n",
      "\n",
      "7f387f334adad2c11f71c4216b0d8fe2\n",
      "7f387f334adad2c11f71c4216b0d8fe2\n",
      "\n",
      "c21ce30b76f57485a3cbe8712a623bd5\n",
      "c21ce30b76f57485a3cbe8712a623bd5\n",
      "\n",
      "20b0b0743fe0214bde319d2d40a40067\n",
      "20b0b0743fe0214bde319d2d40a40067\n",
      "\n",
      "511866d0fc60fe3e8bb6dea4ce87a64f\n",
      "511866d0fc60fe3e8bb6dea4ce87a64f\n",
      "\n",
      "206b30a4b231541a85d4707f0b907153\n",
      "206b30a4b231541a85d4707f0b907153\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "original_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/20170323_pacbio/md5checksums_fastq.gz.md5))\n",
    "current_md5=($(awk '/filtered_subreads.fastq.gz/{print $4}' /Volumes/owl/nightingales/O_lurida/checksums.md5))\n",
    "count=$(( ${#original_md5[@]} - 1 ))\n",
    "for ((i=0;i<=$count;++i))\n",
    "    do\n",
    "        printf \"%s\\n\" \"${original_md5[$i]}\"\n",
    "        printf \"%s\\n\\n\" \"${current_md5[$i]}\"\n",
    "    done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Visual inspection indicates the MD5 checkums have not changed during the copying/renaming process!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation of the above command:\n",
    "\n",
    "##### The gist is that the output from the awk command is saved to an array. Then a for loop is run which iterates over the array and prints the element at each position.\n",
    "\n",
    "---\n",
    "###### Break down the first line:\n",
    "\n",
    "```original_md5=()``` - This is an empty array called \"original_md5\".\n",
    "\n",
    "```$()``` - This is an empty command substitution. The stdout of commands within the parentheses are stored.\n",
    "\n",
    "```awk '/filtered_subreads.fastq.gz/{print $4}' md5_file``` - Awk looks for any lines from the input file (md5_file) with \"filtered_subreads.fastq.gz\" in them. If a line contains \"filtered_subreads.fastq.gz\", awk prints the fourth field (i.e. the fourth column).\n",
    "\n",
    "Summary - The output from each result printed by awk is saved in an auto-incrementing fashion in the array called \"original_md5\".\n",
    "\n",
    "---\n",
    "###### Break down the 3rd line:\n",
    "\n",
    "```count=$(())``` - A variable called \"count\". This is a combination of empty command substitution and bash arithmeetic. Double parentheses are required for bash arithmetic.\n",
    "\n",
    "\n",
    "```${#current_md5[@]} - 1``` - This prints the number of indices (#) in the array called \"original_md5\" and subtracts 1 from that number. Subtraction of one is necessary because bash is a zero-based language (e.g. the array starts at index 0).\n",
    "\n",
    "Summary - The length of the array minus one is saved to the variable called \"count\".\n",
    "\n",
    "---\n",
    "###### Break down the for loop:\n",
    "\n",
    "```((i=0;i<=$count;++i))``` - Sets variable \"i\" to 0. Then, the loop evaluates whether or not the value of \"i\" is less than/equal to the value in the variable \"count\". If that condition is met, the loop increases the value stored in \"i\" by 1 and continues through the loop.\n",
    "\n",
    "```printf \"%s\\n\" \"${original_md5[$i]}\"``` - Prints the value at the array index designated by the value currently stored in \"i\" (the printing is specified by the \"%s\", which means string). This is followed by printing a new line (\\n).\n",
    "\n",
    "Summary - This prints the value at each position within the array and uses printf to improve legibility of output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compress the original UW PacBio folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Volumes/owl/nightingales/O_lurida\n"
     ]
    }
   ],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tar: Removing leading '/' from member names\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/data.items.json: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/data/filtered_regions: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/workflow: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1/filter/log: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: Write errorWrite errorWrite errorFailed to clean up compressor\n",
      "tar: Error exit delayed from previous errors.\n",
      "\n",
      "real\t360m51.774s\n",
      "user\t201m23.761s\n",
      "sys\t17m51.178s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well... That's a bummer. I think I actually need to run this again, as I'm not entirely sure how to tell if the tarball completed properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 970, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 233, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 267, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1049, in getinnerframes\n",
      "    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1009, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 454, in getsourcefile\n",
      "    if hasattr(getmodule(object, filename), '__loader__'):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 483, in getmodule\n",
      "    file = getabsfile(object, _filename)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 467, in getabsfile\n",
      "    return os.path.normcase(os.path.abspath(_filename))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py\", line 364, in abspath\n",
      "    cwd = os.getcwd()\n",
      "OSError: [Errno 2] No such file or directory\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "\n",
      "Unfortunately, your original traceback can not be constructed.\n",
      "\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not iterable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mrun_code\u001b[0;34m(self, code_obj, result)\u001b[0m\n\u001b[1;32m   3081\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3082\u001b[0m                 \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merror_in_exec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3083\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3084\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3085\u001b[0m             \u001b[0moutflag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mshowtraceback\u001b[0;34m(self, exc_tuple, filename, tb_offset, exception_only)\u001b[0m\n\u001b[1;32m   1878\u001b[0m                     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1879\u001b[0m                         stb = self.InteractiveTB.structured_traceback(etype,\n\u001b[0;32m-> 1880\u001b[0;31m                                             value, tb, tb_offset=tb_offset)\n\u001b[0m\u001b[1;32m   1881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1882\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_showtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1240\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1241\u001b[0m         return FormattedTB.structured_traceback(\n\u001b[0;32m-> 1242\u001b[0;31m             self, etype, value, tb, tb_offset, number_of_lines_of_context)\n\u001b[0m\u001b[1;32m   1243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1244\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1148\u001b[0m             \u001b[0;31m# Verbose modes need a full traceback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1149\u001b[0m             return VerboseTB.structured_traceback(\n\u001b[0;32m-> 1150\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1151\u001b[0m             )\n\u001b[1;32m   1152\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1000\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m         formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,\n\u001b[0;32m-> 1002\u001b[0;31m                                                                tb_offset)\n\u001b[0m\u001b[1;32m   1003\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1004\u001b[0m         \u001b[0mcolors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mColors\u001b[0m  \u001b[0;31m# just a shorthand + quicker name lookup\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_exception_as_a_whole\u001b[0;34m(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)\u001b[0m\n\u001b[1;32m    949\u001b[0m         \u001b[0mrecords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    950\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 951\u001b[0;31m         \u001b[0mframes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    952\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrecords\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    953\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_records\u001b[0;34m(self, records)\u001b[0m\n\u001b[1;32m    722\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    723\u001b[0m         \u001b[0mabspath\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabspath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlines\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrecords\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    725\u001b[0m             \u001b[0;31m#print '*** record:',file,lnum,func,lines,index  # dbg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    726\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not iterable"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\n",
      "pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 970, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 233, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 267, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1049, in getinnerframes\n",
      "    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1009, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 454, in getsourcefile\n",
      "    if hasattr(getmodule(object, filename), '__loader__'):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 483, in getmodule\n",
      "    file = getabsfile(object, _filename)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 467, in getabsfile\n",
      "    return os.path.normcase(os.path.abspath(_filename))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py\", line 364, in abspath\n",
      "    cwd = os.getcwd()\n",
      "OSError: [Errno 2] No such file or directory\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "\n",
      "Unfortunately, your original traceback can not be constructed.\n",
      "\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not iterable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mrun_code\u001b[0;34m(self, code_obj, result)\u001b[0m\n\u001b[1;32m   3081\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3082\u001b[0m                 \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merror_in_exec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3083\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3084\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3085\u001b[0m             \u001b[0moutflag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mshowtraceback\u001b[0;34m(self, exc_tuple, filename, tb_offset, exception_only)\u001b[0m\n\u001b[1;32m   1878\u001b[0m                     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1879\u001b[0m                         stb = self.InteractiveTB.structured_traceback(etype,\n\u001b[0;32m-> 1880\u001b[0;31m                                             value, tb, tb_offset=tb_offset)\n\u001b[0m\u001b[1;32m   1881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1882\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_showtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1240\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1241\u001b[0m         return FormattedTB.structured_traceback(\n\u001b[0;32m-> 1242\u001b[0;31m             self, etype, value, tb, tb_offset, number_of_lines_of_context)\n\u001b[0m\u001b[1;32m   1243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1244\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1148\u001b[0m             \u001b[0;31m# Verbose modes need a full traceback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1149\u001b[0m             return VerboseTB.structured_traceback(\n\u001b[0;32m-> 1150\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1151\u001b[0m             )\n\u001b[1;32m   1152\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1000\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m         formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,\n\u001b[0;32m-> 1002\u001b[0;31m                                                                tb_offset)\n\u001b[0m\u001b[1;32m   1003\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1004\u001b[0m         \u001b[0mcolors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mColors\u001b[0m  \u001b[0;31m# just a shorthand + quicker name lookup\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_exception_as_a_whole\u001b[0;34m(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)\u001b[0m\n\u001b[1;32m    949\u001b[0m         \u001b[0mrecords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    950\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 951\u001b[0;31m         \u001b[0mframes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    952\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrecords\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    953\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_records\u001b[0;34m(self, records)\u001b[0m\n\u001b[1;32m    722\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    723\u001b[0m         \u001b[0mabspath\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabspath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlines\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrecords\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    725\u001b[0m             \u001b[0;31m#print '*** record:',file,lnum,func,lines,index  # dbg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    726\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not iterable"
     ]
    }
   ],
   "source": [
    "cd /Volumes/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 970, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 233, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.py\", line 267, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1049, in getinnerframes\n",
      "    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 1009, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 454, in getsourcefile\n",
      "    if hasattr(getmodule(object, filename), '__loader__'):\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 483, in getmodule\n",
      "    file = getabsfile(object, _filename)\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/inspect.py\", line 467, in getabsfile\n",
      "    return os.path.normcase(os.path.abspath(_filename))\n",
      "  File \"/Users/Sam/Applications/anaconda/lib/python2.7/posixpath.py\", line 364, in abspath\n",
      "    cwd = os.getcwd()\n",
      "OSError: [Errno 2] No such file or directory\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "\n",
      "Unfortunately, your original traceback can not be constructed.\n",
      "\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not iterable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mrun_code\u001b[0;34m(self, code_obj, result)\u001b[0m\n\u001b[1;32m   3081\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3082\u001b[0m                 \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merror_in_exec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3083\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3084\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3085\u001b[0m             \u001b[0moutflag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mshowtraceback\u001b[0;34m(self, exc_tuple, filename, tb_offset, exception_only)\u001b[0m\n\u001b[1;32m   1878\u001b[0m                     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1879\u001b[0m                         stb = self.InteractiveTB.structured_traceback(etype,\n\u001b[0;32m-> 1880\u001b[0;31m                                             value, tb, tb_offset=tb_offset)\n\u001b[0m\u001b[1;32m   1881\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1882\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_showtraceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1240\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1241\u001b[0m         return FormattedTB.structured_traceback(\n\u001b[0;32m-> 1242\u001b[0;31m             self, etype, value, tb, tb_offset, number_of_lines_of_context)\n\u001b[0m\u001b[1;32m   1243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1244\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1148\u001b[0m             \u001b[0;31m# Verbose modes need a full traceback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1149\u001b[0m             return VerboseTB.structured_traceback(\n\u001b[0;32m-> 1150\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0metype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1151\u001b[0m             )\n\u001b[1;32m   1152\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mstructured_traceback\u001b[0;34m(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[1;32m   1000\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m         formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,\n\u001b[0;32m-> 1002\u001b[0;31m                                                                tb_offset)\n\u001b[0m\u001b[1;32m   1003\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1004\u001b[0m         \u001b[0mcolors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mColors\u001b[0m  \u001b[0;31m# just a shorthand + quicker name lookup\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_exception_as_a_whole\u001b[0;34m(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)\u001b[0m\n\u001b[1;32m    949\u001b[0m         \u001b[0mrecords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0metb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    950\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 951\u001b[0;31m         \u001b[0mframes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat_records\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    952\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrecords\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    953\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/Sam/Applications/anaconda/lib/python2.7/site-packages/IPython/core/ultratb.pyc\u001b[0m in \u001b[0;36mformat_records\u001b[0;34m(self, records)\u001b[0m\n\u001b[1;32m    722\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    723\u001b[0m         \u001b[0mabspath\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabspath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 724\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlines\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrecords\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    725\u001b[0m             \u001b[0;31m#print '*** record:',file,lnum,func,lines,index  # dbg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    726\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not iterable"
     ]
    }
   ],
   "source": [
    "cd ~/Downloads/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok... It seems like the notebook is just totally jacked up. Guess I'll restart it. Ugh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Jun 29 06:20:30 PDT 2017\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Volumes/owl/nightingales/O_lurida\n"
     ]
    }
   ],
   "source": [
    "cd /Volumes/owl/nightingales/O_lurida/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw-  1 Sam  staff  116176977920 Jun 28 17:31 20170323_pacbio.tar.gz\n",
      "\n",
      "20170323_pacbio:\n",
      "total 48\n",
      "-rw-rw-rw-@ 1 Sam  staff   401 Jun 28 11:26 readme.txt\n",
      "-rw-rw-rw-  1 Sam  staff  1022 Jun 28 08:53 md5checksums_fastq.gz.md5\n",
      "-rw-rw-rw-@ 1 Sam  staff  7112 Jun 28 08:20 all_md5.txt\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  5 10:01 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A04_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  3 07:27 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A03_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  3 08:07 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A02_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Mar 23 18:47 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A01_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  5 10:04 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C02_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Mar 23 17:32 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Mar 23 17:00 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_E01_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Mar 23 16:32 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_D01_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  3 08:01 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\n",
      "drwxrwxrwx  1 Sam  staff   264 Apr  3 08:01 \u001b[30m\u001b[43m170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1\u001b[m\u001b[m/\n"
     ]
    }
   ],
   "source": [
    "ls -lr 20170323_pacbio*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw-  1 Sam  staff   108G Jun 28 17:31 20170323_pacbio.tar.gz\r\n",
      "\r\n",
      "20170323_pacbio:\r\n",
      "total 48\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:01 \u001b[30m\u001b[43m170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:01 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 16:32 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_D01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 17:00 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_E01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 17:32 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  5 10:04 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C02_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 18:47 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:07 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A02_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 07:27 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A03_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  5 10:01 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A04_1\u001b[m\u001b[m/\r\n",
      "-rw-rw-rw-@ 1 Sam  staff   6.9K Jun 28 08:20 all_md5.txt\r\n",
      "-rw-rw-rw-  1 Sam  staff   1.0K Jun 28 08:53 md5checksums_fastq.gz.md5\r\n",
      "-rw-rw-rw-@ 1 Sam  staff   401B Jun 28 11:26 readme.txt\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh 20170323*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, I still have no way of knowing whether or not that tarball is legit. Guess I'll delete it and re-run the compression command. Sigh..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t0m1.360s\n",
      "user\t0m0.001s\n",
      "sys\t0m0.004s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time rm 20170323_pacbio.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 48\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:01 \u001b[30m\u001b[43m170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:01 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 16:32 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_D01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 17:00 \u001b[30m\u001b[43m170228_PCB-CC_AL_20kb_P6v2_E01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 17:32 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  5 10:04 \u001b[30m\u001b[43m170307_PCB-CC_AL_20kb_P6v2_C02_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Mar 23 18:47 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A01_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 08:07 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A02_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  3 07:27 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A03_1\u001b[m\u001b[m/\r\n",
      "drwxrwxrwx  1 Sam  staff   264B Apr  5 10:01 \u001b[30m\u001b[43m170314_PCB-CC_20kb_P6v2_A04_1\u001b[m\u001b[m/\r\n",
      "-rw-rw-rw-@ 1 Sam  staff   6.9K Jun 28 08:20 all_md5.txt\r\n",
      "-rw-rw-rw-  1 Sam  staff   1.0K Jun 28 08:53 md5checksums_fastq.gz.md5\r\n",
      "-rw-rw-rw-@ 1 Sam  staff   401B Jun 28 11:26 readme.txt\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh 20170323*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tar: Removing leading '/' from member names\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/data.items.json: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: Write errorWrite errorWrite errorFailed to clean up compressor\n",
      "tar: Error exit delayed from previous errors.\n",
      "\n",
      "real\t213m55.180s\n",
      "user\t120m32.603s\n",
      "sys\t10m36.751s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dude! Lost the connection to Owl. AGAIN! What is going on??!! I vaguely remember Sean mentioning some data transfer issues involving Owl and Hyak (mox). I wonder if this is related. Will discuss with Steven to see if he has experienced any weird network issues this week.\n",
    "\n",
    "Will have to consider another approach to generating the tarball (likely ssh in as admin and run ```tar``` directly on Owl - not over the network.\n",
    "\n",
    "In the meantime, I'm going to delete this incomplete tarball.\n",
    "\n",
    "I'll remount Owl outside of this notebook and then proceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-f44d0c6507fa>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-f44d0c6507fa>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    $$bash\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "$$bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Jun 29 14:37:21 PDT 2017\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-rw-  1 Sam  staff    68G Jun 29 10:01 /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t0m1.684s\n",
      "user\t0m0.001s\n",
      "sys\t0m0.004s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time rm /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz: No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "ls -lh /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, I'm just going to try to run this again. Third time's the charm?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Volumes/web/nightingales/O_lurida\n"
     ]
    }
   ],
   "source": [
    "cd /Volumes/web/nightingales/O_lurida/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tar: /Volumes/owl/nightingales/O_lurida/20170323_pacbio: Cannot stat: No such file or directory\n",
      "tar: Error exit delayed from previous errors.\n",
      "\n",
      "real\t0m0.074s\n",
      "user\t0m0.001s\n",
      "sys\t0m0.008s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time tar -zcf 20170323_pacbio.tar.gz /Volumes/owl/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tar: Removing leading '/' from member names\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subread_summary.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_subreads.fasta: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk003of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/control_results_by_movie.csv: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/data.items.json: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk001of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/filtered_regions.fofn: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C02_1/filter/data/post_control_regions.chunk002of003: Cannot stat: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170307_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_E01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170228_PCB-CC_AL_20kb_P6v2_C01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: /Volumes/web/nightingales/O_lurida/20170323_pacbio/170210_PCB-CC_MS_EEE_20kb_P6v2_D01_1: Couldn't visit directory: Socket is not connected\n",
      "tar: Write errorWrite errorWrite errorFailed to clean up compressor\n",
      "tar: Error exit delayed from previous errors.\n",
      "\n",
      "real\t216m49.377s\n",
      "user\t125m13.016s\n",
      "sys\t9m52.843s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time tar -zcf 20170323_pacbio.tar.gz /Volumes/web/nightingales/O_lurida/20170323_pacbio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, this is annoying. The problem is caused by the connection to Owl getting lost/unmounted (see below). This is an issue that Sean had previously experienced as well. He had some conversations with the people who run Hyak (mox), they did some quick tests and experienced a similar problem with moving data to/from Owl. According to Sean, either the Hyak (mox) people and/or UW IT indicated that there's a wiring issue in FTR that's causing this problem and it's not a problem related to Owl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\r\n",
      "ls: Volumes/web/nightingales/O_lurida: No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "ls Volumes/web/nightingales/O_lurida"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've reconnected to Owl (via Finder) in order to remove the incomplete tarball."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jul  3 05:57:38 PDT 2017\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz: No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "ls /Volumes/web/nightingales/O_lurida/20170323_pacbio.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}