novice/python/06-cmdline.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Command-Line Programs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The IPython Notebook and other interactive tools are great for prototyping code and exploring data,\n",
      "but sooner or later we will want to use our program in a pipeline\n",
      "or run it in a shell script to process thousands of data files.\n",
      "In order to do that,\n",
      "we need to make our programs work like other Unix command-line tools.\n",
      "For example,\n",
      "we may want a program that reads a data set\n",
      "and prints the average inflammation per patient:\n",
      "\n",
      "~~~\n",
      "$ python readings.py --mean inflammation-01.csv\n",
      "5.45\n",
      "5.425\n",
      "6.1\n",
      "...\n",
      "6.4\n",
      "7.05\n",
      "5.9\n",
      "~~~\n",
      "\n",
      "but we might also want to look at the minimum of the first four lines\n",
      "\n",
      "~~~\n",
      "$ head -4 inflammation-01.csv | python readings.py --min\n",
      "~~~\n",
      "\n",
      "or the maximum inflammations in several files one after another:\n",
      "\n",
      "~~~\n",
      "$ python readings.py --max inflammation-*.csv\n",
      "~~~\n",
      "\n",
      "Our overall requirements are:\n",
      "\n",
      "1. If no filename is given on the command line, read data from [standard input](../../gloss.html#standard-input).\n",
      "2. If one or more filenames are given, read data from them and report statistics for each file separately.\n",
      "3. Use the `--min`, `--mean`, or `--max` flag to determine what statistic to print.\n",
      "\n",
      "To make this work,\n",
      "we need to know how to handle command-line arguments in a program,\n",
      "and how to get at standard input.\n",
      "We'll tackle these questions in turn below."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "objectives"
      ]
     },
     "source": [
      "#### Objectives\n",
      "\n",
      "*   Use the values of command-line arguments in a program.\n",
      "*   Handle flags and files separately in a command-line program.\n",
      "*   Read data from standard input in a program so that it can be used in a pipeline."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Command-Line Arguments"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Using the text editor of your choice,\n",
      "save the following in a text file:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat sys-version.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "print 'version is', sys.version\r\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The first line imports a library called `sys`,\n",
      "which is short for \"system\".\n",
      "It defines values such as `sys.version`,\n",
      "which describes which version of Python we are running.\n",
      "We can run this script from within the IPython Notebook like this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run sys-version.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "version is 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20) \n",
        "[GCC 4.0.1 (Apple Inc. build 5493)]\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "or like this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython sys-version.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "version is 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20) \r\n",
        "[GCC 4.0.1 (Apple Inc. build 5493)]\r\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The first method, `%run`,\n",
      "uses a special command in the IPython Notebook to run a program in a `.py` file.\n",
      "The second method is more general:\n",
      "the exclamation mark `!` tells the Notebook to run a shell command,\n",
      "and it just so happens that the command we run is `ipython` with the name of the script."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Here's another script that does something more interesting:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat argv-list.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "print 'sys.argv is', sys.argv\r\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The strange name `argv` stands for \"argument values\".\n",
      "Whenever Python runs a program,\n",
      "it takes all of the values given on the command line\n",
      "and puts them in the list `sys.argv`\n",
      "so that the program can determine what they were.\n",
      "If we run this program with no arguments:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython argv-list.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sys.argv is ['/Users/gwilson/s/bc/python/novice/argv-list.py']\r\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "the only thing in the list is the full path to our script,\n",
      "which is always `sys.argv[0]`.\n",
      "If we run it with a few arguments, however:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython argv-list.py first second third"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sys.argv is ['/Users/gwilson/s/bc/python/novice/argv-list.py', 'first', 'second', 'third']\r\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "then Python adds each of those arguments to that magic list."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "With this in hand,\n",
      "let's build a version of `readings.py` that always prints the per-patient mean of a single data file.\n",
      "The first step is to write a function that outlines our implementation,\n",
      "and a placeholder for the function that does the actual work.\n",
      "By convention this function is usually called `main`,\n",
      "though we can call it whatever we want:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat readings-01.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def main():\r\n",
        "    script = sys.argv[0]\r\n",
        "    filename = sys.argv[1]\r\n",
        "    data = np.loadtxt(filename, delimiter=',')\r\n",
        "    for m in data.mean(axis=1):\r\n",
        "        print m\r\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "This function gets the name of the script from `sys.argv[0]`,\n",
      "because that's where it's always put,\n",
      "and the name of the file to process from `sys.argv[1]`.\n",
      "Here's a simple test:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run readings-01.py inflammation-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "There is no output because we have defined a function,\n",
      "but haven't actually called it.\n",
      "Let's add a call to `main`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat readings-02.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def main():\r\n",
        "    script = sys.argv[0]\r\n",
        "    filename = sys.argv[1]\r\n",
        "    data = np.loadtxt(filename, delimiter=',')\r\n",
        "    for m in data.mean(axis=1):\r\n",
        "        print m\r\n",
        "\r\n",
        "main()\r\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "and run that:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run readings-02.py inflammation-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "5.45\n",
        "5.425\n",
        "6.1\n",
        "5.9\n",
        "5.55\n",
        "6.225\n",
        "5.975\n",
        "6.65\n",
        "6.625\n",
        "6.525\n",
        "6.775\n",
        "5.8\n",
        "6.225\n",
        "5.75\n",
        "5.225\n",
        "6.3\n",
        "6.55\n",
        "5.7\n",
        "5.85\n",
        "6.55\n",
        "5.775\n",
        "5.825\n",
        "6.175\n",
        "6.1\n",
        "5.8\n",
        "6.425\n",
        "6.05\n",
        "6.025\n",
        "6.175\n",
        "6.55\n",
        "6.175\n",
        "6.35\n",
        "6.725\n",
        "6.125\n",
        "7.075\n",
        "5.725\n",
        "5.925\n",
        "6.15\n",
        "6.075\n",
        "5.75\n",
        "5.975\n",
        "5.725\n",
        "6.3\n",
        "5.9\n",
        "6.75\n",
        "5.925\n",
        "7.225\n",
        "6.15\n",
        "5.95\n",
        "6.275\n",
        "5.7\n",
        "6.1\n",
        "6.825\n",
        "5.975\n",
        "6.725\n",
        "5.7\n",
        "6.25\n",
        "6.4\n",
        "7.05\n",
        "5.9\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> #### The Right Way to Do It\n",
      ">\n",
      "> If our programs can take complex parameters or multiple filenames,\n",
      "> we shouldn't handle `sys.argv` directly.\n",
      "> Instead,\n",
      "> we should use Python's `argparse` library,\n",
      "> which handles common cases in a systematic way,\n",
      "> and also makes it easy for us to provide sensible error messages for our users."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "challenges"
      ]
     },
     "source": [
      "#### Challenges\n",
      "\n",
      "1.  Write a command-line program that does addition and subtraction:\n",
      "\n",
      "    ~~~\n",
      "    $ python arith.py 1 + 2\n",
      "    3\n",
      "    $ python arith.py 3 - 4\n",
      "    -1\n",
      "    ~~~\n",
      "\n",
      "    What goes wrong if you try to add multiplication using '*' to the program?\n",
      "\n",
      "2.  Using the `glob` module introduced [03-loop.ipynb](earlier),\n",
      "    write a simple version of `ls` that shows files in the current directory with a particular suffix:\n",
      "    \n",
      "    ~~~\n",
      "    $ python my_ls.py py\n",
      "    left.py\n",
      "    right.py\n",
      "    zero.py\n",
      "    ~~~"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Handling Multiple Files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The next step is to teach our program how to handle multiple files.\n",
      "Since 60 lines of output per file is a lot to page through,\n",
      "we'll start by creating three smaller files,\n",
      "each of which has three days of data for two patients:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ls small-*.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "small-01.csv small-02.csv small-03.csv\r\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0,0,1\r\n",
        "0,1,2\r\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run readings-02.py small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.333333333333\n",
        "1.0\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Using small data files as input also allows us to check our results more easily:\n",
      "here,\n",
      "for example,\n",
      "we can see that our program is calculating the mean correctly for each line,\n",
      "whereas we were really taking it on faith before.\n",
      "This is yet another rule of programming:\n",
      "\"[test the simple things first](../../rules.html#test-simple-first)\".\n",
      "\n",
      "We want our program to process each file separately,\n",
      "so we need a looop that executes once for each filename.\n",
      "If we specify the files on the command line,\n",
      "the filenames will be in `sys.argv`,\n",
      "but we need to be careful:\n",
      "`sys.argv[0]` will always be the name of our script,\n",
      "rather than the name of a file.\n",
      "We also need to handle an unknown number of filenames,\n",
      "since our program could be run for any number of files.\n",
      "\n",
      "The solution to both problems is to loop over the contents of `sys.argv[1:]`.\n",
      "The '1' tells Python to start the slice at location 1,\n",
      "so the program's name isn't included;\n",
      "since we've left off the upper bound,\n",
      "the slice runs to the end of the list,\n",
      "and includes all the filenames.\n",
      "Here's our changed program:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat readings-03.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def main():\r\n",
        "    script = sys.argv[0]\r\n",
        "    for filename in sys.argv[1:]:\r\n",
        "        data = np.loadtxt(filename, delimiter=',')\r\n",
        "        for m in data.mean(axis=1):\r\n",
        "            print m\r\n",
        "\r\n",
        "main()\r\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "and here it is in action:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run readings-03.py small-01.csv small-02.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.333333333333\n",
        "1.0\n",
        "13.6666666667\n",
        "11.0\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Note:\n",
      "at this point,\n",
      "we have created three versions of our script called `readings-01.py`,\n",
      "`readings-02.py`, and `readings-03.py`.\n",
      "We wouldn't do this in real life:\n",
      "instead,\n",
      "we would have one file called `readings.py` that we committed to version control\n",
      "every time we got an enhancement working.\n",
      "For teaching,\n",
      "though,\n",
      "we need all the successive versions side by side."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "challenges"
      ]
     },
     "source": [
      "#### Challenges\n",
      "\n",
      "1.  Write a program called `check.py` that takes the names of one or more inflammation data files as arguments\n",
      "    and checks that all the files have the same number of rows and columns.\n",
      "    What is the best way to test your program?"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Handling Command-Line Flags"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The next step is to teach our program to pay attention to the `--min`, `--mean`, and `--max` flags.\n",
      "These always appear before the names of the files,\n",
      "so we could just do this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat readings-04.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def main():\r\n",
        "    script = sys.argv[0]\r\n",
        "    action = sys.argv[1]\r\n",
        "    filenames = sys.argv[2:]\r\n",
        "\r\n",
        "    for f in filenames:\r\n",
        "        data = np.loadtxt(f, delimiter=',')\r\n",
        "\r\n",
        "        if action == '--min':\r\n",
        "            values = data.min(axis=1)\r\n",
        "        elif action == '--mean':\r\n",
        "            values = data.mean(axis=1)\r\n",
        "        elif action == '--max':\r\n",
        "            values = data.max(axis=1)\r\n",
        "\r\n",
        "        for m in values:\r\n",
        "            print m\r\n",
        "\r\n",
        "main()\r\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "This works:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run readings-04.py --max small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.0\n",
        "2.0\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "but there are seveal things wrong with it:\n",
      "\n",
      "1.  `main` is too large to read comfortably.\n",
      "\n",
      "2.  If `action` isn't one of the three recognized flags,\n",
      "    the program loads each file but does nothing with it\n",
      "    (because none of the branches in the conditional match).\n",
      "    [Silent failures](../../gloss.html#silent-failure) like this\n",
      "    are always hard to debug.\n",
      "\n",
      "This version pulls the processing of each file out of the loop into a function of its own.\n",
      "It also checks that `action` is one of the allowed flags\n",
      "before doing any processing,\n",
      "so that the program fails fast:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat readings-05.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def main():\r\n",
        "    script = sys.argv[0]\r\n",
        "    action = sys.argv[1]\r\n",
        "    filenames = sys.argv[2:]\r\n",
        "    assert action in ['--min', '--mean', '--max'], \\\r\n",
        "           'Action is not one of --min, --mean, or --max: ' + action\r\n",
        "    for f in filenames:\r\n",
        "        process(f, action)\r\n",
        "\r\n",
        "def process(filename, action):\r\n",
        "    data = np.loadtxt(filename, delimiter=',')\r\n",
        "\r\n",
        "    if action == '--min':\r\n",
        "        values = data.min(axis=1)\r\n",
        "    elif action == '--mean':\r\n",
        "        values = data.mean(axis=1)\r\n",
        "    elif action == '--max':\r\n",
        "        values = data.max(axis=1)\r\n",
        "\r\n",
        "    for m in values:\r\n",
        "        print m\r\n",
        "\r\n",
        "main()\r\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "This is four lines longer than its predecessor,\n",
      "but broken into more digestible chunks of 8 and 12 lines."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Python has a module named [argparse](http://docs.python.org/dev/library/argparse.html)\n",
      "that helps handle complex command-line flags. We will not cover this module in this lesson\n",
      "but you can go to Tshepang Lekhonkhobe's [Argparse tutorial](http://docs.python.org/dev/howto/argparse.html)\n",
      "that is part of Python's Official Documentation."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "challenges"
      ]
     },
     "source": [
      "#### Challenges\n",
      "\n",
      "1.  Rewrite this program so that it uses `-n`, `-m`, and `-x` instead of `--min`, `--mean`, and `--max` respectively.\n",
      "    Is the code easier to read?\n",
      "    Is the program easier to understand?\n",
      "\n",
      "2.  Separately,\n",
      "    modify the program so that if no parameters are given\n",
      "    (i.e., no action is specified and no filenames are given),\n",
      "    it prints a message explaining how it should be used.\n",
      "\n",
      "3.  Separately,\n",
      "    modify the program so that if no action is given\n",
      "    it displays the means of the data."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Handling Standard Input"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "The next thing our program has to do is read data from standard input if no filenames are given\n",
      "so that we can put it in a pipeline,\n",
      "redirect input to it,\n",
      "and so on.\n",
      "Let's experiment in another script:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat count-stdin.py"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "import sys\r\n",
        "\r\n",
        "count = 0\r\n",
        "for line in sys.stdin:\r\n",
        "    count += 1\r\n",
        "\r\n",
        "print count, 'lines in standard input'\r\n"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "This little program reads lines from a special \"file\" called `sys.stdin`,\n",
      "which is automatically connected to the program's standard input.\n",
      "We don't have to open it&mdash;Python and the operating system\n",
      "take care of that when the program starts up&mdash;\n",
      "but we can do almost anything with it that we could do to a regular file.\n",
      "Let's try running it as if it were a regular command-line program:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython count-stdin.py < small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2 lines in standard input\r\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "What if we run it using `%run`?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run count-stdin.py < small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0 lines in standard input\n"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "As you can see,\n",
      "`%run` doesn't understand file redirection:\n",
      "that's a shell thing.\n",
      "\n",
      "A common mistake is to try to run something that reads from standard input like this:\n",
      "\n",
      "~~~\n",
      "!ipython count_stdin.py small-01.csv\n",
      "~~~\n",
      "\n",
      "i.e., to forget the `<` character that redirect the file to standard input.\n",
      "In this case,\n",
      "there's nothing in standard input,\n",
      "so the program waits at the start of the loop for someone to type something on the keyboard.\n",
      "Since there's no way for us to do this,\n",
      "our program is stuck,\n",
      "and we have to halt it using the `Interrupt` option from the `Kernel` menu in the Notebook.\n",
      "\n",
      "We now need to rewrite the program so that it loads data from `sys.stdin` if no filenames are provided.\n",
      "Luckily,\n",
      "`numpy.loadtxt` can handle either a filename or an open file as its first parameter,\n",
      "so we don't actually need to change `process`.\n",
      "That leaves `main`:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "~~~\n",
      "def main():\n",
      "    script = sys.argv[0]\n",
      "    action = sys.argv[1]\n",
      "    filenames = sys.argv[2:]\n",
      "    assert action in ['--min', '--mean', '--max'], \\\n",
      "           'Action is not one of --min, --mean, or --max: ' + action\n",
      "    if len(filenames) == 0:\n",
      "        process(sys.stdin, action)\n",
      "    else:\n",
      "        for f in filenames:\n",
      "            process(f, action)\n",
      "~~~"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Let's try it out\n",
      "(we'll see in a moment why we send the output through `head`):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython readings-06.py --mean < small-01.csv | head -10"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[TerminalIPythonApp] CRITICAL | Bad config encountered during initialization:\r\n",
        "[TerminalIPythonApp] CRITICAL | Unrecognized flag: '--mean'\r\n",
        "=========\r\n",
        " IPython\r\n",
        "=========\r\n",
        "\r\n",
        "Tools for Interactive Computing in Python\r\n",
        "=========================================\r\n",
        "\r\n",
        "    A Python shell with automatic history (input and output), dynamic object\r\n",
        "    introspection, easier configuration, command completion, access to the\r\n",
        "    system shell and more.  IPython can also be embedded in running programs.\r\n"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "Whoops:\n",
      "why are we getting IPython's help rather than the line-by-line average of our data?\n",
      "The answer is that IPython has a hard time telling\n",
      "which command-line arguments are meant for it,\n",
      "and which are meant for the program it's running.\n",
      "To make our meaning clear,\n",
      "we have to use `--` (a double dash)\n",
      "to separate the two:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!ipython readings-06.py -- --mean < small-01.csv"
     ],
     "language": "python",
     "metadata": {
      "cell_tags": []
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.333333333333\r\n",
        "1.0\r\n"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": []
     },
     "source": [
      "That's better.\n",
      "In fact,\n",
      "that's done:\n",
      "the program now does everything we set out to do."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "challenges"
      ]
     },
     "source": [
      "#### Challenges\n",
      "\n",
      "1.  Write a program called `line-count.py` that works like the Unix `wc` command:\n",
      "    *   If no filenames are given, it reports the number of lines in standard input.\n",
      "    *   If one or more filenames are given, it reports the number of lines in each, followed by the total number of lines."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "cell_tags": [
       "keypoints"
      ]
     },
     "source": [
      "#### Key Points\n",
      "\n",
      "*   The `sys` library connects a Python program to the system it is running on.\n",
      "*   The list `sys.argv` contains the command-line arguments that a program was run with.\n",
      "*   Avoid silent failures.\n",
      "*   The \"file\" `sys.stdin` connects to a program's standard input.\n",
      "*   The \"file\" `sys.stdout` connects to a program's standard output."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}