notebooks/2016-09-19_jbw_stackoverflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stackoverflow\n",
    "\n",
    "Stackoverflow has been my friend for **years**. I've received help from so many questions/answers/comments it's hard to even estimate the time saved. It's a place that can be a bit cranky at times, too. For example, I asked a question over four years ago [Reproduce the Unix cat command in Python](http://stackoverflow.com/questions/11532980/reproduce-the-unix-cat-command-in-python).\n",
    "\n",
    "For veterans, the immediate reaction is \"why the hell would you want to reproduce an already existing tool that's bulletproof?\" Of course, that's coming at it from the mindset of an expert or at least someone far from the beginning of their computer programming journey. I'd encourage people to take a bit more charitable mindset -- and I'll peel back my thinking with the benefit of 4+ years of learning to bridge the gap between why a young and naive Jonathan was thinking and saying versus what what actually said.\n",
    "\n",
    "I didn't actually want to **reproduce `cat`**, I wanted python's `readlines` to give me what I expected -- which `cat` appeared to do without trying. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First let's create two example files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting fort.13\n"
     ]
    }
   ],
   "source": [
    "%%writefile fort.13\n",
    "   FeII     11.22274     1.6907577ac    5.1691ac  -3.84E-06QA      0.00   1.00E+00  0 !    1\n",
    "   FeII     11.89638     1.6908290ad    3.6138ad  -3.84E-06QA      0.00   1.00E+00  0 !    2\n",
    "   FeII     12.13440     1.6909416ae    8.4330ae  -3.84E-06QA      0.00   1.00E+00  0 !    3\n",
    "   FeII     12.66998     1.6909743af    2.5432af  -3.84E-06qa      0.00   1.00E+00  0 !    4\n",
    "   FeII     11.84478     1.6910729ag   10.1314ag  -3.84E-06QA      0.00   1.00E+00  0 !    5\n",
    "   FeII     11.11314     1.6910954ah    0.7396ah  -3.84E-06QA      0.00   1.00E+00  0 !    6\n",
    "   FeII     12.27689     1.6911837ai    3.2614ai  -3.84E-06QA      0.00   1.00E+00  0 !    7\n",
    "   FeII     12.50262     1.6912225aj    4.6212aj  -3.84E-06QA      0.00   1.00E+00  0 !    8\n",
    "   FeII     11.89564     1.6913030ca    5.0580ca  -3.84E-06QA      0.00   1.00E+00  0 !    9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting command.info\n"
     ]
    }
   ],
   "source": [
    "%%writefile command.info\n",
    "! printf \"f\\nil\\ncs\\n5.e-7 50.0 5.e-7\\nn\\n0.003\\nb\\n0.02\\nz\\n1.e-6\\nx4\\n5.e-7\\n\\n\\nP.1.69.starting.alpha.fort.13\\nn\\nn\\n\\n\" | vpfit\n",
    "   *\n",
    "jwnorAq2217.dat  1   4328.1400   4331.5300 vsig=2.00  !  1  1.000   261 2012/03/26 FeII 1608\n",
    "jwnorAq2217.dat  1   4495.5500   4499.8000 vsig=2.11  !  2  0.000   3272 AlII 1670\n",
    "jwnorAq2217.dat  1   6310.5000   6311.5000 vsig=2.11  !  3  0.180    703 FeII 2344\n",
    "jwnorAq2217.dat  1   6312.2500   6312.7000 vsig=2.11  !  4  0.827    324 FeII 2344\n",
    "jwnorAq2217.dat  1   6389.3700   6391.4500 vsig=2.21  !  5  0.973   1465 FeII 2374\n",
    "jwnorAq2217.dat  1   6391.5498   6394.2400 vsig=2.21  !  6  0.483   1906 FeII 2374 \n",
    "jwnorAq2217.dat  1   6411.2000   6417.2000 vsig=2.20  !  7  0.000   4237 FeII 2382\n",
    "jwnorAq2217.dat  1   6959.6000   6961.5000 vsig=2.05  !  8  0.002   1348 FeII 2586\n",
    "jwnorAq2217.dat  1   6996.7000   7000.5000 vsig=2.07  !  10 0.000   267  10 FeII 2600\n",
    "jwnorAq2217.dat  1   7001.5500   7002.7900 vsig=2.07  !  11 0.029    88  11 FeII 2600\n",
    "  *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First thing that I tried\n",
    "\n",
    "I was trying to read in two files into memory, manipulate them some, then print them back out. Here was roughly how I was doing that piece of logic (I'm printing the results instead of storing them here).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "! printf \"f\\nil\\ncs\\n5.e-7 50.0 5.e-7\\nn\\n0.003\\nb\\n0.02\\nz\\n1.e-6\\nx4\\n5.e-7\\n\\n\\nP.1.69.starting.alpha.fort.13\\nn\\nn\\n\\n\" | vpfit\n",
      "\n",
      "   *\n",
      "\n",
      "jwnorAq2217.dat  1   4328.1400   4331.5300 vsig=2.00  !  1  1.000   261 2012/03/26 FeII 1608\n",
      "\n",
      "jwnorAq2217.dat  1   4495.5500   4499.8000 vsig=2.11  !  2  0.000   3272 AlII 1670\n",
      "\n",
      "jwnorAq2217.dat  1   6310.5000   6311.5000 vsig=2.11  !  3  0.180    703 FeII 2344\n",
      "\n",
      "jwnorAq2217.dat  1   6312.2500   6312.7000 vsig=2.11  !  4  0.827    324 FeII 2344\n",
      "\n",
      "jwnorAq2217.dat  1   6389.3700   6391.4500 vsig=2.21  !  5  0.973   1465 FeII 2374\n",
      "\n",
      "jwnorAq2217.dat  1   6391.5498   6394.2400 vsig=2.21  !  6  0.483   1906 FeII 2374 \n",
      "\n",
      "jwnorAq2217.dat  1   6411.2000   6417.2000 vsig=2.20  !  7  0.000   4237 FeII 2382\n",
      "\n",
      "jwnorAq2217.dat  1   6959.6000   6961.5000 vsig=2.05  !  8  0.002   1348 FeII 2586\n",
      "\n",
      "jwnorAq2217.dat  1   6996.7000   7000.5000 vsig=2.07  !  10 0.000   267  10 FeII 2600\n",
      "\n",
      "jwnorAq2217.dat  1   7001.5500   7002.7900 vsig=2.07  !  11 0.029    88  11 FeII 2600\n",
      "\n",
      "  *\n",
      "   FeII     11.22274     1.6907577ac    5.1691ac  -3.84E-06QA      0.00   1.00E+00  0 !    1\n",
      "\n",
      "   FeII     11.89638     1.6908290ad    3.6138ad  -3.84E-06QA      0.00   1.00E+00  0 !    2\n",
      "\n",
      "   FeII     12.13440     1.6909416ae    8.4330ae  -3.84E-06QA      0.00   1.00E+00  0 !    3\n",
      "\n",
      "   FeII     12.66998     1.6909743af    2.5432af  -3.84E-06qa      0.00   1.00E+00  0 !    4\n",
      "\n",
      "   FeII     11.84478     1.6910729ag   10.1314ag  -3.84E-06QA      0.00   1.00E+00  0 !    5\n",
      "\n",
      "   FeII     11.11314     1.6910954ah    0.7396ah  -3.84E-06QA      0.00   1.00E+00  0 !    6\n",
      "\n",
      "   FeII     12.27689     1.6911837ai    3.2614ai  -3.84E-06QA      0.00   1.00E+00  0 !    7\n",
      "\n",
      "   FeII     12.50262     1.6912225aj    4.6212aj  -3.84E-06QA      0.00   1.00E+00  0 !    8\n",
      "\n",
      "   FeII     11.89564     1.6913030ca    5.0580ca  -3.84E-06QA      0.00   1.00E+00  0 !    9\n"
     ]
    }
   ],
   "source": [
    "with open('fort.13', 'r') as fort13, open('command.info', 'r') as com:\n",
    "    for line in com.readlines():\n",
    "        print line\n",
    "    for line in fort13.readlines():\n",
    "        print line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# My confusion\n",
    "\n",
    "The question I had: why does this think these files are double-spaced? Of course, searching for _that_ doesn't find anything useful (even to this day). The real 'problem' is that the newline is captured at the end of the line, so when printing line by line an extra return gets in there. Contrasting with `cat` which just works as expected, led to this question. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# My 'fix'\n",
    "\n",
    "This worked for my specific use case, but was clearly not the Right Way to do this. Leading to my question, which I pared down to a MWE (Minimum Working Example) that actually worked before I even asked the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('command.fort.13', 'w') as outFile:\n",
    "    with open('fort.13', 'r') as fort13, open('command.info', 'r') as com:\n",
    "        for line in com.read().split('\\n'):\n",
    "            if line.strip() != '':\n",
    "                print >>outFile, line\n",
    "        for line in fort13.read().split('\\n'):\n",
    "            if line.strip() != '':\n",
    "                print >>outFile, line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "! printf \"f\\nil\\ncs\\n5.e-7 50.0 5.e-7\\nn\\n0.003\\nb\\n0.02\\nz\\n1.e-6\\nx4\\n5.e-7\\n\\n\\nP.1.69.starting.alpha.fort.13\\nn\\nn\\n\\n\" | vpfit\r\n",
      "   *\r\n",
      "jwnorAq2217.dat  1   4328.1400   4331.5300 vsig=2.00  !  1  1.000   261 2012/03/26 FeII 1608\r\n",
      "jwnorAq2217.dat  1   4495.5500   4499.8000 vsig=2.11  !  2  0.000   3272 AlII 1670\r\n",
      "jwnorAq2217.dat  1   6310.5000   6311.5000 vsig=2.11  !  3  0.180    703 FeII 2344\r\n",
      "jwnorAq2217.dat  1   6312.2500   6312.7000 vsig=2.11  !  4  0.827    324 FeII 2344\r\n",
      "jwnorAq2217.dat  1   6389.3700   6391.4500 vsig=2.21  !  5  0.973   1465 FeII 2374\r\n",
      "jwnorAq2217.dat  1   6391.5498   6394.2400 vsig=2.21  !  6  0.483   1906 FeII 2374 \r\n",
      "jwnorAq2217.dat  1   6411.2000   6417.2000 vsig=2.20  !  7  0.000   4237 FeII 2382\r\n",
      "jwnorAq2217.dat  1   6959.6000   6961.5000 vsig=2.05  !  8  0.002   1348 FeII 2586\r\n",
      "jwnorAq2217.dat  1   6996.7000   7000.5000 vsig=2.07  !  10 0.000   267  10 FeII 2600\r\n",
      "jwnorAq2217.dat  1   7001.5500   7002.7900 vsig=2.07  !  11 0.029    88  11 FeII 2600\r\n",
      "  *\r\n",
      "   FeII     11.22274     1.6907577ac    5.1691ac  -3.84E-06QA      0.00   1.00E+00  0 !    1\r\n",
      "   FeII     11.89638     1.6908290ad    3.6138ad  -3.84E-06QA      0.00   1.00E+00  0 !    2\r\n",
      "   FeII     12.13440     1.6909416ae    8.4330ae  -3.84E-06QA      0.00   1.00E+00  0 !    3\r\n",
      "   FeII     12.66998     1.6909743af    2.5432af  -3.84E-06qa      0.00   1.00E+00  0 !    4\r\n",
      "   FeII     11.84478     1.6910729ag   10.1314ag  -3.84E-06QA      0.00   1.00E+00  0 !    5\r\n",
      "   FeII     11.11314     1.6910954ah    0.7396ah  -3.84E-06QA      0.00   1.00E+00  0 !    6\r\n",
      "   FeII     12.27689     1.6911837ai    3.2614ai  -3.84E-06QA      0.00   1.00E+00  0 !    7\r\n",
      "   FeII     12.50262     1.6912225aj    4.6212aj  -3.84E-06QA      0.00   1.00E+00  0 !    8\r\n",
      "   FeII     11.89564     1.6913030ca    5.0580ca  -3.84E-06QA      0.00   1.00E+00  0 !    9\r\n"
     ]
    }
   ],
   "source": [
    "cat command.fort.13"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To answer the question that I was getting at over four years ago\n",
    "\n",
    "You want to `read` and then use `splitlines()` if you want to get the behavior that you're hoping for (including keeping extra blank lines that are in the file, which my previous naive solution would mercilessly remove)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "! printf \"f\\nil\\ncs\\n5.e-7 50.0 5.e-7\\nn\\n0.003\\nb\\n0.02\\nz\\n1.e-6\\nx4\\n5.e-7\\n\\n\\nP.1.69.starting.alpha.fort.13\\nn\\nn\\n\\n\" | vpfit\n",
      "   *\n",
      "jwnorAq2217.dat  1   4328.1400   4331.5300 vsig=2.00  !  1  1.000   261 2012/03/26 FeII 1608\n",
      "jwnorAq2217.dat  1   4495.5500   4499.8000 vsig=2.11  !  2  0.000   3272 AlII 1670\n",
      "jwnorAq2217.dat  1   6310.5000   6311.5000 vsig=2.11  !  3  0.180    703 FeII 2344\n",
      "jwnorAq2217.dat  1   6312.2500   6312.7000 vsig=2.11  !  4  0.827    324 FeII 2344\n",
      "jwnorAq2217.dat  1   6389.3700   6391.4500 vsig=2.21  !  5  0.973   1465 FeII 2374\n",
      "jwnorAq2217.dat  1   6391.5498   6394.2400 vsig=2.21  !  6  0.483   1906 FeII 2374 \n",
      "jwnorAq2217.dat  1   6411.2000   6417.2000 vsig=2.20  !  7  0.000   4237 FeII 2382\n",
      "jwnorAq2217.dat  1   6959.6000   6961.5000 vsig=2.05  !  8  0.002   1348 FeII 2586\n",
      "jwnorAq2217.dat  1   6996.7000   7000.5000 vsig=2.07  !  10 0.000   267  10 FeII 2600\n",
      "jwnorAq2217.dat  1   7001.5500   7002.7900 vsig=2.07  !  11 0.029    88  11 FeII 2600\n",
      "  *\n",
      "   FeII     11.22274     1.6907577ac    5.1691ac  -3.84E-06QA      0.00   1.00E+00  0 !    1\n",
      "   FeII     11.89638     1.6908290ad    3.6138ad  -3.84E-06QA      0.00   1.00E+00  0 !    2\n",
      "   FeII     12.13440     1.6909416ae    8.4330ae  -3.84E-06QA      0.00   1.00E+00  0 !    3\n",
      "   FeII     12.66998     1.6909743af    2.5432af  -3.84E-06qa      0.00   1.00E+00  0 !    4\n",
      "   FeII     11.84478     1.6910729ag   10.1314ag  -3.84E-06QA      0.00   1.00E+00  0 !    5\n",
      "   FeII     11.11314     1.6910954ah    0.7396ah  -3.84E-06QA      0.00   1.00E+00  0 !    6\n",
      "   FeII     12.27689     1.6911837ai    3.2614ai  -3.84E-06QA      0.00   1.00E+00  0 !    7\n",
      "   FeII     12.50262     1.6912225aj    4.6212aj  -3.84E-06QA      0.00   1.00E+00  0 !    8\n",
      "   FeII     11.89564     1.6913030ca    5.0580ca  -3.84E-06QA      0.00   1.00E+00  0 !    9\n"
     ]
    }
   ],
   "source": [
    "with open('fort.13', 'r') as fort13, open('command.info', 'r') as com:\n",
    "    for line in com.read().splitlines():\n",
    "        print line\n",
    "    for line in fort13.read().splitlines():\n",
    "        print line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py2]",
   "language": "python",
   "name": "conda-env-py2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}