shallowWater-threaded.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive monitoring of a shallow water wave parallel simulation with Proteus and IPython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Brian E. Granger(1), Chris E. Kees(2), Fernando Perez(3), Benjamin Ragan-Kelley(3) and Jose Unpingco (4).\n",
    "\n",
    "(1) California Polytechnic State University, San Luis Obispo, CA.\n",
    "\n",
    "(2) Army Engineer Research and Development Center, Vicksburg, MS.\n",
    "\n",
    "(3) University of California, Berkeley, CA.\n",
    "\n",
    "(4) DRC Inc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this technical report, we will illustrate how the IPython parallel computing capabilities can be combined with the interactive IPython notebook, to introspect and monitor the evolution of an existing HPC code that uses MPI for its core communication needs.  \n",
    "\n",
    "The [Proteus toolkit](http://proteus.usace.army.mil) is a modeling package for the numerical simulation of continuous processes such as the shallow water wave equations.  Proteus was designed to run in traditional HPC environments, using the MPI libraries for efficient inter-node communication.  The [IPython project](http://ipython.org) provides tools for interactive computing that include a web-based interface called the IPython Notebook and a set of high-level libraries for parallel computing that complement MPI.\n",
    "\n",
    "We will construct a simulation with Proteus that will run in any traditional HPC environment and will monitor its evolution interactively from the web notebook, using the notebook's facilities for data visualization.  The interactive monitoring will be performed without stopping or slowing down the parallel simulation.\n",
    "\n",
    "This will be accomplished without requiring any changes to the architecture of Proteus and only a trivial, minimal change to how the code would have been executed in batch mode.  Importantly, this change does not introduce an IPython dependency into the simulation and has no discernible performance impact. This allows for the same code to be used in production runs absent interactive monitoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load IPython support for working with MPI tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rest of this notebook assumes that you have already started an IPython cluster with the `mpi` profile.  If you have not done so yet, use [the cluster tab in the Dashboard](/#tab2) to start your `mpi` cluster.  If you do not have a cluster configuration for MPI use, see the IPython documentation for [detailed instructions](http://ipython.org/ipython-doc/rel-0.12.1/parallel/parallel_process.html#using-ipcluster-in-mpiexec-mpirun-mode) on how to create one.\n",
    "\n",
    "We begin by creating a cluster client that gives us a local handle on the engines running in the (possibly remote) MPI cluster.  From the client we make a `view` object, which we set to use blocking mode by default as it is more convenient for interactive control.  Since the real computation will be done over MPI without IPython intervention, setting the default behavior to be blocking will have no significant performance impact:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "from IPython.parallel import Client, error\n",
    "cluster = Client(profile=\"mpi\")\n",
    "view = cluster[:]\n",
    "view.block = True\n",
    "%load_ext parallelmagic\n",
    "view.activate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we load the MPI libraries into the engine namespaces, and do a simple printing of their MPI rank information to verify that all nodes are operational and they match our cluster's real capacity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] MPI rank: 1/4\n",
      "[stdout:1] MPI rank: 3/4\n",
      "[stdout:2] MPI rank: 2/4\n",
      "[stdout:3] MPI rank: 0/4\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "from mpi4py import MPI\n",
    "mpi = MPI.COMM_WORLD\n",
    "bcast = mpi.bcast\n",
    "barrier = mpi.barrier\n",
    "rank = mpi.rank\n",
    "print \"MPI rank: %i/%i\" % (mpi.rank,mpi.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Proteus libraries and set simulation parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the Proteus objects that parametrize our example, except for the main simulation object that will be created later.  That will make it more convenient to recreate a fresh one by only re-executing a single cell without re-defining all parameters.\n",
    "\n",
    "In this example, we will use Proteus to solve the shallow water equations for cylindrically symmetric flow impacting the center of a flat container with square shape and perfectly reflecting walls:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "# Required imports\n",
    "from proteus.iproteus import * \n",
    "from proteus import default_n, default_s, default_so\n",
    "import sw_2d_p, sw_2d_n\n",
    "from proteus import Comm\n",
    "from petsc4py import PETSc\n",
    "\n",
    "# Simulation parameters\n",
    "nSteps = 101\n",
    "sw_2d_n.tnList = [i*sw_2d_p.T/float(nSteps-1) for i in range(nSteps)]\n",
    "pList = [sw_2d_p]\n",
    "nList = [sw_2d_n]\n",
    "so = default_so\n",
    "so.name = pList[0].name = \"sw_2d\"\n",
    "so.sList = [default_s]\n",
    "so.tnList = sw_2d_n.tnList\n",
    "nList[0].multilevelLinearSolver = default_n.KSP_petsc4py\n",
    "\n",
    "# PETSc solver configuration\n",
    "OptDB = PETSc.Options()\n",
    "OptDB.setValue(\"ksp_type\", \"bcgsl\")\n",
    "OptDB.setValue(\"pc_type\", \"asm\")\n",
    "OptDB.setValue(\"pc_asm_type\", \"basic\")\n",
    "OptDB.setValue(\"sub_pc_type\", \"jacobi\")\n",
    "OptDB.setValue(\"ksp_rtol\", 0.0)\n",
    "OptDB.setValue(\"ksp_atol\", 1.0e-8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a simulation object that can be monitored interactively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we create a numerical simulation object, `ns`, that we will use to compute the desired solution.  The key idea for performing interactive monitoring of an ongoing MPI simulation is to call the main solution method (in this case `ns.calculateSolution(...)`) in a *secondary worker thread*, leaving the main thread available to answer for IPython requests.  This separation allows all computation and MPI communications to happen in one thread, while the other is available for communication with an interactive control client (typically, but not necessarily, a human inspecting the results).  While we have chosen to put the IPython code in the main thread and the MPI code in the secondary one, these roles can be reveresed without ill effects.\n",
    "\n",
    "In cases where the numerical code has manual control of the time-stepping, one can very easily write a small helper class that advances the simulation step by step from the thread.  Such a configuration will then permit not only monitorig and inspection, but also pausing and restarting of the simulation (even with adjustment of parameters).  But for now, the following simple code suffices that uses the default `Thread` class without modifications.\n",
    "\n",
    "Note: we run this cell in non-blocking mode only to silence a lot of verbose terminal output from all nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] \n",
      "TriangleBaseMesh nbase=1 baseFlags= pAq30Dena0.000800 \n",
      "2  nSpace_global\n",
      "[stdout:1] \n",
      "TriangleBaseMesh nbase=1 baseFlags= pAq30Dena0.000800 \n",
      "2  nSpace_global\n",
      "[stdout:2] \n",
      "TriangleBaseMesh nbase=1 baseFlags= pAq30Dena0.000800 \n",
      "2  nSpace_global\n",
      "[stdout:3] \n",
      "TriangleBaseMesh nbase=1 baseFlags= pAq30Dena0.000800 \n",
      "2  nSpace_global\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "from threading import Thread\n",
    "# Create simulation object (but don't start the computation yet)\n",
    "ns = NumericalSolution.NS_base(so, pList, nList, so.sList, opts)\n",
    "# Create a thread wrapper for the simulation.  The target must be an argument-less\n",
    "# function so we wrap the call to `calculateSolution` in a simple lambda:\n",
    "simulation_thread = Thread(target = lambda : ns.calculateSolution('run1'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] <Thread(Thread-2, initial)>\n",
      "[stdout:1] <Thread(Thread-2, initial)>\n",
      "[stdout:2] <Thread(Thread-2, initial)>\n",
      "[stdout:3] <Thread(Thread-2, initial)>\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "print simulation_thread"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now define a local (to this notebook) plotting function that makes a remote call in the engines to populate some variables in the global namespace.  Once it has retrieved the current state of the relevant variables, it produces and returns a figure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.core.display import clear_output\n",
    "\n",
    "def plot_current_results(in_place=True):\n",
    "    \"\"\"Makes a blocking call to retrieve remote data and displays the solution mesh\n",
    "    as a contour plot.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    in_place : bool\n",
    "        By default it calls clear_output so that new plots replace old ones.  Set\n",
    "        to False to allow keeping of all previous outputs.\n",
    "    \"\"\"\n",
    "    \n",
    "    # We make a blocking call to load the remote data from the simulation into simple named \n",
    "    # variables we can read from the engine namespaces\n",
    "    view.apply_sync(load_simulation_globals)\n",
    "    # And now we can use the view to read these variables from all the engines.  Then we\n",
    "    # concatenate all of them into single arrays for local plotting\n",
    "    x = np.concatenate(view['x'])\n",
    "    y = np.concatenate(view['y'])\n",
    "    u = np.concatenate(view['u'])\n",
    "    shifts = numpy.cumsum([0]+view['nn'][:-1])\n",
    "    flat_triangles = np.concatenate([ tri + shift for tri,shift in zip(view['triangles'], shifts) ])\n",
    "    # We can now call the matplotlib plotting function we need\n",
    "    fig, ax = plt.subplots(subplot_kw=dict(aspect='equal'))\n",
    "    ax.tricontourf(x, y, flat_triangles, u)\n",
    "    # We clear the notebook output before plotting this if in-place plot updating is requested\n",
    "    if in_place:\n",
    "        clear_output()\n",
    "    display(fig)\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the function that will be called *on the engines* by the plotting code to load the relevant data structures into named variables in the global namespace that we can just read off:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_simulation_globals():\n",
    "    \"\"\"Put some variables we need in engine namespace.\n",
    "\n",
    "    These can then be retrieved by clients for inspection, visualization, etc.\n",
    "    \"\"\"\n",
    "    global nn, x, y, u, triangles\n",
    "    model = ns.modelList[0].levelModelList[-1]\n",
    "    nodes = model.mesh.nodeArray\n",
    "    triangles = model.mesh.elementNodesArray\n",
    "    x = nodes[:,0]\n",
    "    y = nodes[:,1]\n",
    "    u = model.u[0].dof\n",
    "    nn = len(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also define a utility to check whether any of our engines is still busy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def simulation_alive():\n",
    "    \"\"\"Return True if the simulation thread is still running on any engine.\n",
    "    \"\"\"\n",
    "    return any(view.apply_sync(lambda : simulation_thread.is_alive()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, this is a convenience wrapper around the plotting code so that we can interrupt monitoring at any point, and that will provide basic timing information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def monitor_simulation(refresh=5.0, plots_in_place=True):\n",
    "    \"\"\"Monitor the simulation progress and call plotting routine.\n",
    "\n",
    "    Supress KeyboardInterrupt exception if interrupted, ensure that the last \n",
    "    figure is always displayed and provide basic timing and simulation status.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    refresh : float\n",
    "      Refresh interval between calls to retrieve and plot data.  The default\n",
    "      is 5s, adjust depending on the desired refresh rate, but be aware that \n",
    "      very short intervals will start having a significant impact.\n",
    "\n",
    "    plots_in_place : bool\n",
    "       If true, every new figure replaces the last one, producing a (slow)\n",
    "       animation effect in the notebook.  If false, all frames are plotted\n",
    "       in sequence and appended in the output area.\n",
    "    \"\"\"\n",
    "    import datetime as dt, time\n",
    "    \n",
    "    if not simulation_alive():\n",
    "        plot_current_results(in_place=plots_in_place)\n",
    "        plt.close('all')\n",
    "        print 'Simulation has already finished, no monitoring to do.'\n",
    "        return\n",
    "    \n",
    "    t0 = dt.datetime.now()\n",
    "    fig = None\n",
    "    try:\n",
    "        while simulation_alive():\n",
    "            fig = plot_current_results(in_place=plots_in_place)\n",
    "            plt.close('all') # prevent re-plot of old figures\n",
    "            time.sleep(refresh) # so we don't hammer the server too fast\n",
    "    except (KeyboardInterrupt, error.TimeoutError):\n",
    "        msg = 'Monitoring interrupted, simulation is ongoing!'\n",
    "    else:\n",
    "        msg = 'Simulation completed!'\n",
    "    tmon = dt.datetime.now() - t0\n",
    "    if plots_in_place and fig is not None:\n",
    "        clear_output()\n",
    "        display(fig)\n",
    "    print msg\n",
    "    print 'Monitored for: %s.' % tmon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start simulation and monitor its evolution interactively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start the simulation by calling the `start()` method of our thread wrapper on all engines.  They will then run all the communications related to the numerical computation over MPI, while listening on the IPython sockets for data requests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px \n",
    "simulation_thread.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The monitoring cell below can be interrupted any time by using the `Kernel Interrupt` menu option, the 'stop' icon in the toolbar or the interrupt keybinding (`Control-m-i`).  It can be interrupted and restarted as many times as desired, as it will simply read and display the currently available data.  Calling the function with `plots_in_place` set to `False` will keep the entire sequence of plots visible in the output area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQcAAAEACAYAAAC+rrMfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEPlJREFUeJzt3U+IHOedxvHnt45y8sEMBi2xJvggGWIkWF1kYbF47CxG\n0kEGY4gFUcCXyAaRsBeFDQLL4Mv6FLxxLB3s4FVAOiRCOweJeIml4BVYbNayoknk4MEEZC3RZWzj\nWBeL/HKYnpmemre7/nT9ed/q7wcGqrrf6Xqr36qn36p6u9rcXQCQ9Q9dVwBAnAgHAEGEA4AgwgFA\nEOEAIIhwABA0NhzMbNbMLprZH8xswcx+MKLcq2b2kZldM7OdzVQVQJu+lvP8V5L+1d0/MLN7Jf2f\nmf23u99YKWBm+yVtdfdtZvaIpNcl7W6uygDaMLbn4O5/cfcPBtN/lXRD0jcyxQ5IemtQ5oqk+8xs\ncwN1BdCiwucczOxBSTslXck89YCkm0Pzn0jaMmnFAHSrUDgMDil+KemHgx7EhiKZecZkA4nLO+cg\nM9sk6VeSfuHu5wJFbkmaHZrfMngs+zoEBtARd89+gOcaGw5mZpLekPRHd//JiGLzko5IOmNmuyV9\n5u63w0XfK1u/0R5/ZG362Nrkk0/M17eMEhaPn9bW4wc7WXbT+rxuUnfr9/Y7B9ZmXh564mL2yH1S\n1a4P5PUc9kj6rqTfm9nVwWM/lvRNSXL3k+5+3sz2m9mipC8lPVepJgCiMjYc3P1/VOC8hLsfqa1G\nZR3LLwKgvPRHSL6cX6QNM3Pbu65CY/q8blL/16+q9MMhEjNzO7quQmP6vG5S/9evqjTDYcTJSAD1\nSTMcgL6J8EOuV+HQ1WVMoFbDPeMO9SocANSHcAA6EntPN3f4dHQmGBl5WCcaqNB6J/V848tAdW1s\nA1mFt4ljiubSvCRZW79bsfzdihqGTwfCYVQwdLEhNGUaQqdP7TVKqB2bH0a9u9J3K9IKh4K9hmnY\nyJC2bEg0GxDVwiG9w4qMOoPh6YULlf7v7PZ9lZeJOFRt+xVlt4HDOrEuIJ58Yn59QEQgnZ5DzuHE\nuFCYtOFT0JeA6mtbjWuflZBorvfQ18OK7DXfgsHQ140M6QsFRW5ASBOERB/DoUIwEApIRTYkhg8z\nVkOiloDoUziERojlBMPEofDKZP8enaNaW6ejI8q8Mua5cSZ9r7LLTP29r/IeDhkOiUIBIZUMiVTC\noezQ0BFXJUoHQ+obINJVIDwqBcQoG4IjhXB4vOSyJg0GAgGxGRMUuQEhVRskddF6FA6Zb6iVCgYC\nASkYERKFAkIq2ZNIIRx+U35ZBEP7Lp9aP7/nUDf1mAqBkCgcEEV9u2fhMG5wE8EwmezOLy0HQOjx\nqgiUEkoEhFQhJPoUDgRDverc6csIBcTlUwRHUJMBUTEcWh8+PWqYaOWvrxIMQV0FQpE6EBABJS8r\nh/aXt985sOHxSYZkt9pzeNL/q/T/je01EAzrxBAIZRAQGSV7D0W9bU9V6jlEfbMXvl0JrGl7f4g6\nHIYxLLp/UuvpNC6ynnC04ZCbkpG9kUATuvxQjDYc0H+ccwjI+dBr89CCcOiRVHa2PYfSqes0SyIc\nON9QXOw7Xez1w5okwmEDzjeMFeMnc4x1SkVXH45phgMKiWFnJBTSFeUNZhnfUJ/sjtnG5UPCYEI5\noyWzN6dtSpThgOYM77h1BQVh0E+EwxRjp8Y4nHMAEEQ4AAgiHAAEEQ4AgggHAEGEAxCjCEYBEw4A\ngtILhwgSFehSG6MjpRTDAUArCAcAQYQDgCDCAUBQlOEw9oRLiR/+AFBdbjiY2ZtmdtvMro94fs7M\nPjezq4O/Y6FyteFqBdCKIl/Z/rmk/5D0n2PK/Nbdq//uFoDo5PYc3P1dSZ/mFCv9U1sA4lbHOQeX\n9KiZXTOz82b2cA2vCaBjddwJ6n1Js+5+x8z2STon6aFQwcXjp1enZ+a2a2ZuRw2LBzBs6dJ1LV1a\nmPh1Jg4Hd/9iaPqCmf3MzGbcfSlbduvxg5MuDkCOmbkd6z54P37pTKXXmfiwwsw2m5kNpndJslAw\nAEhLbs/BzE5LekzS/WZ2U9KLkjZJkruflPSMpBfM7K6kO5Keba66ANqSGw7uPvZYwN1fk/RabTUC\nEIUoR0gC6B7hACCIcAAQRDgACCIcAAQRDgCCCAcAQemFAzd7AVqRXjgAaEV64cCdoDBlzm7f18ly\n0wsHAK0gHAAEEQ5A5J5euLA63dZP4UmEA4ARCAcgcpyQBLAmgvE8hAOAoGjDYfjES1fdKmCaRRsO\nALqVXjhEcCwGTIP0wgFAKwgHAEGEA4AgwgFAEOEAIIhwAFp2+dTyX+wIBwBBub+VCaBeew51XYNi\n6DkACCIcAAT18rBi+GRPKl049Ffo5GMK22Xveg7ZhkjlzDD6KeVtr3fhMErKjYT+SWF77FU45L3h\nKTQIpkfs22NvwiH2NxrTJ/VtsjfhUFTqDYY0NLGdtXlbemkKwwFAMYQD0KGYe7JTGQ4xNwjS15ft\nayrDAUA+wgFAEOEAIIhwAGrUl/MNUuTh0PZ1XQBrog4HAN2Z2nDoU/cPaMLUhgNQt7594OSGg5m9\naWa3zez6mDKvmtlHZnbNzHbWW0UAXSjSc/i5pL2jnjSz/ZK2uvs2Sd+X9HpNdQPQodxwcPd3JX06\npsgBSW8Nyl6RdJ+Zba6jcod1olC5vnXngBjUcc7hAUk3h+Y/kbSlhtcF0KG6bjBrmXkPFVo8fnp1\nemZuu2bmdtSy8D2H6D0AK5YuXdfSpYWJX6eOcLglaXZofsvgsQ22Hj9Yw+IAjDMzt2PdB+/HL52p\n9Dp1HFbMS/qeJJnZbkmfufvtGl6XEZJAh3J7DmZ2WtJjku43s5uSXpS0SZLc/aS7nzez/Wa2KOlL\nSc81WWEA7cgNB3fPPRZw9yP1VAdALBghCSCIcABq0sRP3J3dvq/+Fy2IcAAi9vTChc6WHXU4FB0h\nCaQs1jE6UYdDk1L4lWOgS1MbDkAsYv2gIhyAGsW6o1dBOAAIIhwABPUmHPrUnUPaymyLMW+3vQmH\nMmJuEGCUti/tT2U4AE3rwwfQ1IVDHxoNaMNUhQPBgDalvr1NTTik3lBA23oVDqMCgGBAV1Le9noV\nDlLajYF+Cm2Tew7Fv63WdffpqMT+pmP6rNwhPaVts3c9ByBWKQWDRDgAGIFwABBEOAAIIhwABBEO\nAILSC4dXuq4AMB3SCwcArSAcAASlFw5Hu64AMB2SCIcuf/UHmFZJhAOA9hEOAILSCwcuZQKtSC8c\ngCnW5h2oCQcAQdGGQ9v36AewXrThAEy1CM6tEQ5A5Loa50M4AJE7u31fJ8slHAAEEQ4AgggHIHKc\ncwAQFcIBQBDhACCIcAAQRDgACCIcAATlhoOZ7TWzD83sIzP7UeD5OTP73MyuDv6ONVNVAG362rgn\nzeweST+V9C+Sbkn6XzObd/cbmaK/dfcDDdURQAfyeg67JC26+5/d/StJZyQ9FShntdcMQKfywuEB\nSTeH5j8ZPDbMJT1qZtfM7LyZPVxnBQF0Y+xhhZZ3/DzvS5p19ztmtk/SOUkPhQouHj+9Oj0zt10z\nczuK1hNAQUuXrmvp0sLEr5MXDrckzQ7Nz2q597DK3b8Ymr5gZj8zsxl3X8q+2NbjByepK4ACZuZ2\nrPvg/filM5VeJ++w4neStpnZg2b2dUnfkTQ/XMDMNpuZDaZ3SbJQMABIy9ieg7vfNbMjkn4t6R5J\nb7j7DTM7PHj+pKRnJL1gZncl3ZH0bMN1BtCCvMMKufsFSRcyj50cmn5N0mv1V22Eo4ri/npA36U3\nQpJgAFoRZThwW3qge1GGA4DuEQ4AgggHAEGEA4AgwgFITFsn7AkHAEHphcPRrisATIf0wgFAKwgH\nIEYR9JAJBwBBhAOAoNxvZWL6XD41+rk9h9qrB7pFOEyxcSFQ9X8Ij/6IMhxO6nm+mdmQKoFQ5fUJ\nieac1POtLCfKcEC9mg6EccskJCqI4EqFlGo4cDeoQroIhbw6EBblnd2+r5PlJnG1oqs3J1WXT8UR\nDCEx1w3rJREO6B9CIn6EQ8+ktsOlVt9pEm04tHVGtk/Y0Xog52Rkm/tFtOGQteG8QyRndIEmdXm+\nLepwoPdQTopXAlKsc2Mi6jVIkYdDFr2HfCnsbHsOrf0hXuZe5Ie0a1iQmes3+ct68on5DY8Nj5Z8\neuHChucZ8xAW6zkIQiEg8EE3/GGY7TW8/c6B4q/9bZO7W9kqRRcOK7IhQUDUp8nQyO74w8siFEZo\nMhik/oWDNDogCIfJjdtpGfrcohGHxivhMHEwSImEw+M5yzoWfng4JMYGhERIIB0lgmFkKLxcYDkX\n+xAOwzJBsRIQuYcXKwgJxGySYCgSCMN6Fw4rhkKidEBIhATiUODKWm4wlA2FFUmEg94bX+jxR8KP\nBwJCKnCIMQqBgSaVvMQ+6uTjajCMC4WLVwosYXcPwmFYKCgGIREKCKlCSDQpG0CMyYjTqA+Kltqr\nUjAUCoRhfQuHFdmQyAkIKbKQAAKyA/qaCwapv+EgrQ+InEOMLIJiTZvj9HnfNxr1/hc6x1ApFFb0\nORykiQIiq8sNlxvXtK/t9i7TxrnBMFEorOh7OEi1BgTQpbGDm2rrMayYhnCQCgXEimkLipi/xbrS\nFmXr2Mc2DL0HI88zdBgOad5gtqBYbnEf807blqrvQdX/i6Hdh41bj5GjH2sJhurS6zlIpXoP4xzW\nidVGy25M7NAIqdoDGqXZw4kV9BxKG25gwgBFTNN2ktTNXvJU+sYa0JHYt9dehQPQCx2fa1hBOAAI\nSjMcIklWoDZVv3HZoDTDYViEbyrQB7nhYGZ7zexDM/vIzH40osyrg+evmdnO+qsJoG1jw8HM7pH0\nU0l7JT0s6aCZfStTZr+kre6+TdL3Jb3eUF2jtnTpetdVaEyf103q//pVlddz2CVp0d3/7O5fSToj\n6alMmQOS3pIkd78i6T4z21x7TSO3dGmh6yo0ps/rJvV//arKC4cHJN0cmv9k8FhemS2TVw1Al/LC\noejY6uzQzHbGZANozNjvVpjZbknH3X3vYP7fJP3N3f99qMwJSZfc/cxg/kNJj7n77cxrERhAR5r4\nbsXvJG0zswcl/b+k70g6mCkzL+mIpDODMPksGwxVKwegO2PDwd3vmtkRSb+WdI+kN9z9hpkdHjx/\n0t3Pm9l+M1uU9KWk5xqvNYDGtfaVbQBpqX2EZJ8HTeWtm5nNmdnnZnZ18DfiB/7iY2ZvmtltMxt5\n0T/VdpPy1y/ltpMkM5s1s4tm9gczWzCzH4woV7wN3b22Py0feixKelDSJkkfSPpWpsx+SecH049I\neq/OOjT1V3Dd5iTNd13Xiuv3z5J2Sro+4vkk263E+iXbdoP6/6OkfxpM3yvpT5Pue3X3HPo8aKrI\nukkbL+smwd3flfTpmCKptpukQusnJdp2kuTuf3H3DwbTf5V0Q9I3MsVKtWHd4dDnQVNF1s0lPTro\nsp03s4dbq13zUm23onrTdoOrizslZb++XKoN675NXJ8HTRWp4/uSZt39jpntk3RO0kPNVqtVKbZb\nUb1oOzO7V9IvJf1w0IPYUCQzP7IN6+453JI0OzQ/q+V0Gldmy+Cx2OWum7t/4e53BtMXJG0ys5n2\nqtioVNutkD60nZltkvQrSb9w93OBIqXasO5wWB00ZWZf1/KgqewtoeclfU9aHYEZHDQVodx1M7PN\nZmaD6V1avlS81H5VG5FquxWSetsN6v6GpD+6+09GFCvVhrUeVniPB00VWTdJz0h6wczuSroj6dnO\nKlySmZ2W9Jik+83spqQXtXxVJul2W5G3fkq47Qb2SPqupN+b2dXBYz+W9E2pWhsyCApAUPq3iQPQ\nCMIBQBDhACCIcAAQRDgACCIcAAQRDgCCCAcAQX8HGTSJD13z0eMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3f710c7d50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulation completed!\n",
      "Monitored for: 0:06:14.339446.\n"
     ]
    }
   ],
   "source": [
    "monitor_simulation(refresh=5.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison with a direct, non-monitored run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to perform a measurement of the entire simulation running directly without any IPython interactive monitoring or thread overhead, you can rerun this notebook skipping the two cells above and instead executing the following one.  The output value will be the total simulation time in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = view.apply_async(lambda : ns.calculateSolution('run1'))\n",
    "r.wait()\n",
    "r.wall_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our experiments so far, running the monitored version with an update interval of 5s produces a negligible overhead of ~ 0.5-1%, while updating every 1s raises it to ~20%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This report shows how IPython can be used to provide interactive introspection, monitoring and control capabilities to existing Python HPC codes that use MPI, requiring only trivial and non-intrusive modifications to the codes that don't introduce any permanent IPython dependency.  We have illustrated this by monitoring a shallow water simulation performed with the Proteus toolkit.\n",
    "\n",
    "The only assumption made was that the HPC code has a \"doit\" method or function that runs the main body of the code.  If the code has fine-grained time-stepping control, we can use IPython to manually move the time step forward, pause and restart the simulation, and modify intermediate variables.  But even if only a global \"do it all\" capability exists, we can still introspect and monitor the simulation's evolution, visualizing intermediate results as they are generated in the HPC nodes but using our local visualization capabilities.  This allows for human analysis of the simulation as the process is ongoing, without having to add visualization support to the main code or having to slow down the simulation for interactive introspection.\n",
    "\n",
    "We know of no other system that provides interactive monitoring and visualization of a traditional MPI code with such low development effort, changes to the existing code and performance impact."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}