examples/IPython Kernel/Script Magics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Scripts from IPython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IPython has a `%%script` cell magic, which lets you run a cell in\n",
    "a subprocess of any interpreter on your system, such as: bash, ruby, perl, zsh, R, etc.\n",
    "\n",
    "It can even be a script of your own, which expects input on stdin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use it, simply pass a path or shell command to the program you want to run on the `%%script` line,\n",
    "and the rest of the cell will be run by that script, and stdout/err from the subprocess are captured and displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello from Python 2.7.9 (default, Jan 29 2015, 06:27:40) \n",
      "[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]\n"
     ]
    }
   ],
   "source": [
    "%%script python2\n",
    "import sys\n",
    "print 'hello from Python %s' % sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello from Python: 3.4.2 |Continuum Analytics, Inc.| (default, Oct 21 2014, 17:42:20) \n",
      "[GCC 4.2.1 (Apple Inc. build 5577)]\n"
     ]
    }
   ],
   "source": [
    "%%script python3\n",
    "import sys\n",
    "print('hello from Python: %s' % sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IPython also creates aliases for a few common interpreters, such as bash, ruby, perl, etc.\n",
    "\n",
    "These are all equivalent to `%%script <name>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello from Ruby 2.0.0\n"
     ]
    }
   ],
   "source": [
    "%%ruby\n",
    "puts \"Hello from Ruby #{RUBY_VERSION}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello from /usr/local/bin/bash\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"hello from $BASH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capturing output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also capture stdout/err from these subprocesses into Python variables, instead of letting them go directly to stdout/err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi, stdout\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "hello, stderr\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"hi, stdout\"\n",
    "echo \"hello, stderr\" >&2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash --out output --err error\n",
    "echo \"hi, stdout\"\n",
    "echo \"hello, stderr\" >&2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello, stderr\n",
      "\n",
      "hi, stdout\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(error)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background Scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These scripts can be run in the background, by adding the `--bg` flag.\n",
    "\n",
    "When you do this, output is discarded unless you use the `--out/err`\n",
    "flags to store output as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting job # 0 in a separate thread.\n"
     ]
    }
   ],
   "source": [
    "%%ruby --bg --out ruby_lines\n",
    "for n in 1...10\n",
    "    sleep 1\n",
    "    puts \"line #{n}\"\n",
    "    STDOUT.flush\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you do store output of a background thread, these are the stdout/err *pipes*,\n",
    "rather than the text of the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<open file '<fdopen>', mode 'rb' at 0x10a4be660>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ruby_lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line 1\n",
      "line 2\n",
      "line 3\n",
      "line 4\n",
      "line 5\n",
      "line 6\n",
      "line 7\n",
      "line 8\n",
      "line 9\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(ruby_lines.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arguments to subcommand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass arguments the subcommand as well,\n",
    "such as  this example instructing Python to use integer division from Python 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.333333333333\n"
     ]
    }
   ],
   "source": [
    "%%script python2 -Qnew\n",
    "print 1/3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can really specify *any* program for `%%script`,\n",
    "for instance here is a 'program' that echos the lines of stdin, with delays between each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting job # 2 in a separate thread.\n"
     ]
    }
   ],
   "source": [
    "%%script --bg --out bashout bash -c \"while read line; do echo $line; sleep 1; done\"\n",
    "line 1\n",
    "line 2\n",
    "line 3\n",
    "line 4\n",
    "line 5\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, since the output of a background script is just the stdout pipe,\n",
    "you can read it as lines become available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0s: line 1\n",
      "1.0s: line 2\n",
      "2.0s: line 3\n",
      "3.0s: line 4\n",
      "4.0s: line 5\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "tic = time.time()\n",
    "line = True\n",
    "while True:\n",
    "    line = bashout.readline()\n",
    "    if not line:\n",
    "        break\n",
    "    sys.stdout.write(\"%.1fs: %s\" %(time.time()-tic, line))\n",
    "    sys.stdout.flush()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the default ScriptMagics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list of aliased script magics is configurable.\n",
    "\n",
    "The default is to pick from a few common interpreters, and use them if found, but you can specify your own in ipython_config.py:\n",
    "\n",
    "    c.ScriptMagics.scripts = ['R', 'pypy', 'myprogram']\n",
    "\n",
    "And if any of these programs do not apear on your default PATH, then you would also need to specify their location with:\n",
    "\n",
    "    c.ScriptMagics.script_paths = {'myprogram': '/opt/path/to/myprogram'}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}