magics.txt

.. _parallel_magics:

=======================
Parallel Magic Commands
=======================

We provide a few IPython magic commands
that make it a bit more pleasant to execute Python commands on the engines interactively.
These are mainly shortcuts to :meth:`.DirectView.execute`
and :meth:`.AsyncResult.display_outputs` methods repsectively.

These magics will automatically become available when you create a Client:

.. sourcecode:: ipython

    In [2]: rc = parallel.Client()

The initially active View will have attributes ``targets='all', block=True``,
which is a blocking view of all engines, evaluated at request time
(adding/removing engines will change where this view's tasks will run).

The Magics
==========

%px
---

The %px magic executes a single Python command on the engines
specified by the :attr:`targets` attribute of the :class:`DirectView` instance:

.. sourcecode:: ipython

    # import numpy here and everywhere
    In [25]: with rc[:].sync_imports():
       ....:    import numpy
    importing numpy on engine(s)

    In [27]: %px a = numpy.random.rand(2,2)
    Parallel execution on engines: [0, 1, 2, 3]

    In [28]: %px numpy.linalg.eigvals(a)
    Parallel execution on engines: [0, 1, 2, 3]
    Out [0:68]: array([ 0.77120707, -0.19448286])
    Out [1:68]: array([ 1.10815921,  0.05110369])
    Out [2:68]: array([ 0.74625527, -0.37475081])
    Out [3:68]: array([ 0.72931905,  0.07159743])
    
    In [29]: %px print 'hi'
    Parallel execution on engine(s): all
    [stdout:0] hi
    [stdout:1] hi
    [stdout:2] hi
    [stdout:3] hi


Since engines are IPython as well, you can even run magics remotely:

.. sourcecode:: ipython

    In [28]: %px %pylab inline
    Parallel execution on engine(s): all
    [stdout:0] 
    Welcome to pylab, a matplotlib-based Python environment...
    For more information, type 'help(pylab)'.
    [stdout:1] 
    Welcome to pylab, a matplotlib-based Python environment...
    For more information, type 'help(pylab)'.
    [stdout:2] 
    Welcome to pylab, a matplotlib-based Python environment...
    For more information, type 'help(pylab)'.
    [stdout:3] 
    Welcome to pylab, a matplotlib-based Python environment...
    For more information, type 'help(pylab)'.

And once in pylab mode with the inline backend,
you can make plots and they will be displayed in your frontend
if it suports the inline figures (e.g. notebook or qtconsole):

.. sourcecode:: ipython

    In [40]: %px plot(rand(100))
    Parallel execution on engine(s): all
    <plot0>
    <plot1>
    <plot2>
    <plot3>
    Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
    Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
    Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
    Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]


%%px Cell Magic
---------------

%%px can be used as a Cell Magic, which accepts some arguments for controlling
the execution.


Targets and Blocking
********************

%%px accepts ``--targets`` for controlling which engines on which to run,
and ``--[no]block`` for specifying the blocking behavior of this cell,
independent of the defaults for the View.

.. sourcecode:: ipython

    In [6]: %%px --targets ::2
       ...: print "I am even"
       ...: 
    Parallel execution on engine(s): [0, 2]
    [stdout:0] I am even
    [stdout:2] I am even

    In [7]: %%px --targets 1
       ...: print "I am number 1"
       ...: 
    Parallel execution on engine(s): 1
    I am number 1

    In [8]: %%px
       ...: print "still 'all' by default"
       ...: 
    Parallel execution on engine(s): all
    [stdout:0] still 'all' by default
    [stdout:1] still 'all' by default
    [stdout:2] still 'all' by default
    [stdout:3] still 'all' by default

    In [9]: %%px --noblock
       ...: import time
       ...: time.sleep(1)
       ...: time.time()
       ...: 
    Async parallel execution on engine(s): all
    Out[9]: <AsyncResult: execute>

    In [10]: %pxresult
    Out[0:12]: 1339454561.069116
    Out[1:10]: 1339454561.076752
    Out[2:12]: 1339454561.072837
    Out[3:10]: 1339454561.066665


.. seealso::

    :ref:`%pxconfig` accepts these same arguments for changing the *default*
    values of targets/blocking for the active View.


Output Display
**************


%%px also accepts a ``--group-outputs`` argument,
which adjusts how the outputs of multiple engines are presented.

.. seealso::

    :meth:`.AsyncResult.display_outputs` for the grouping options.

.. sourcecode:: ipython

    In [50]: %%px --block --group-outputs=engine
       ....: import numpy as np
       ....: A = np.random.random((2,2))
       ....: ev = numpy.linalg.eigvals(A)
       ....: print ev
       ....: ev.max()
       ....:
    Parallel execution on engine(s): all
    [stdout:0] [ 0.60640442  0.95919621]
    Out [0:73]: 0.9591962130899806
    [stdout:1] [ 0.38501813  1.29430871]
    Out [1:73]: 1.2943087091452372
    [stdout:2] [-0.85925141  0.9387692 ]
    Out [2:73]: 0.93876920456230284
    [stdout:3] [ 0.37998269  1.24218246]
    Out [3:73]: 1.2421824618493817


%pxresult
---------

If you are using %px in non-blocking mode, you won't get output.
You can use %pxresult to display the outputs of the latest command,
just as is done when %px is blocking:

.. sourcecode:: ipython

    In [39]: dv.block = False
    
    In [40]: %px print 'hi'
    Async parallel execution on engine(s): all
    
    In [41]: %pxresult
    [stdout:0] hi
    [stdout:1] hi
    [stdout:2] hi
    [stdout:3] hi

%pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
It accepts the same output-grouping arguments as %%px, so you can use it to view
a result in different ways.


%autopx
-------

The %autopx magic switches to a mode where everything you type is executed
on the engines until you do %autopx again.

.. sourcecode:: ipython

    In [30]: dv.block=True

    In [31]: %autopx
    %autopx enabled

    In [32]: max_evals = []

    In [33]: for i in range(100):
       ....:     a = numpy.random.rand(10,10)
       ....:     a = a+a.transpose()
       ....:     evals = numpy.linalg.eigvals(a)
       ....:     max_evals.append(evals[0].real)
       ....:

    In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
    [stdout:0] Average max eigenvalue is: 10.193101
    [stdout:1] Average max eigenvalue is: 10.064508
    [stdout:2] Average max eigenvalue is: 10.055724
    [stdout:3] Average max eigenvalue is: 10.086876

    In [35]: %autopx
    Auto Parallel Disabled

%pxconfig
---------

The default targets and blocking behavior for the magics are governed by the :attr:`block`
and :attr:`targets` attribute of the active View.  If you have a handle for the view,
you can set these attributes directly, but if you don't, you can change them with
the %pxconfig magic:

.. sourcecode:: ipython

    In [3]: %pxconfig --block

    In [5]: %px print 'hi'
    Parallel execution on engine(s): all
    [stdout:0] hi
    [stdout:1] hi
    [stdout:2] hi
    [stdout:3] hi

    In [6]: %pxconfig --targets ::2

    In [7]: %px print 'hi'
    Parallel execution on engine(s): [0, 2]
    [stdout:0] hi
    [stdout:2] hi

    In [8]: %pxconfig --noblock

    In [9]: %px print 'are you there?'
    Async parallel execution on engine(s): [0, 2]
    Out[9]: <AsyncResult: execute>

    In [10]: %pxresult
    [stdout:0] are you there?
    [stdout:2] are you there?


Multiple Active Views
=====================

The parallel magics are associated with a particular :class:`~.DirectView` object.
You can change the active view by calling the :meth:`~.DirectView.activate` method
on any view.

.. sourcecode:: ipython

    In [11]: even = rc[::2]

    In [12]: even.activate()

    In [13]: %px print 'hi'
    Async parallel execution on engine(s): [0, 2]
    Out[13]: <AsyncResult: execute>

    In [14]: even.block = True

    In [15]: %px print 'hi'
    Parallel execution on engine(s): [0, 2]
    [stdout:0] hi
    [stdout:2] hi

When activating a View, you can also specify a *suffix*, so that a whole different
set of magics are associated with that view, without replacing the existing ones.

.. sourcecode:: ipython

    # restore the original DirecView to the base %px magics
    In [16]: rc.activate()
    Out[16]: <DirectView all>

    In [17]: even.activate('_even')

    In [18]: %px print 'hi all'
    Parallel execution on engine(s): all
    [stdout:0] hi all
    [stdout:1] hi all
    [stdout:2] hi all
    [stdout:3] hi all

    In [19]: %px_even print "We aren't odd!"
    Parallel execution on engine(s): [0, 2]
    [stdout:0] We aren't odd!
    [stdout:2] We aren't odd!

This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.

For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
which creates a DirectView with block=True, activates it, and returns the new View.

The initial magics registered when you create a client are the result of a call to
:meth:`rc.activate` with default args.


Engines as Kernels
==================

Engines are really the same object as the Kernels used elsewhere in IPython,
with the minor exception that engines connect to a controller, while regular kernels
bind their sockets, listening for connections from a QtConsole or other frontends.

Sometimes for debugging or inspection purposes, you would like a QtConsole connected
to an engine for more direct interaction.  You can do this by first instructing
the Engine to *also* bind its kernel, to listen for connections:

.. sourcecode:: ipython

    In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()

Then, if your engines are local, you can start a qtconsole right on the engine(s):

.. sourcecode:: ipython

    In [51]: %px %qtconsole

Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!

Or you can view just the connection info, and work out the right way to connect to the engines,
depending on where they live and where you are:

.. sourcecode:: ipython

    In [51]: %px %connect_info
    Parallel execution on engine(s): all
    [stdout:0] 
    {
      "stdin_port": 60387, 
      "ip": "127.0.0.1", 
      "hb_port": 50835, 
      "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", 
      "shell_port": 55328, 
      "iopub_port": 58264
    }

    Paste the above JSON into a file, and connect with:
        $> ipython <app> --existing <file>
    or, if you are local, you can connect with just:
        $> ipython <app> --existing kernel-60125.json 
    or even just:
        $> ipython <app> --existing 
    if this is the most recent IPython session you have started.
    [stdout:1] 
    {
      "stdin_port": 61869,
    ...

.. note::

    ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
    so you can often skip that first step.