Skip to content

Commit

Permalink
update ipcluster docs w/ latest PBS/SGE/LSF changes
Browse files Browse the repository at this point in the history
  • Loading branch information
@jtriley @satra
jtriley authored and satra committed Aug 1, 2010
1 parent 09567ca commit 1505a87
Showing 1 changed file with 39 additions and 58 deletions.
97 changes: 39 additions & 58 deletions docs/source/parallel/parallel_process.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ The :command:`ipcluster` command provides a simple way of starting a controller
with most MPI [MPI]_ implementations
3. When engines are started using the PBS [PBS]_ batch system.
4. When engines are started using the SGE [SGE]_ batch system.
5. When the controller is started on localhost and the engines are started on
5. When engines are started using the LSF [LSF]_ batch system.
6. When the controller is started on localhost and the engines are started on
remote nodes using :command:`ssh`.

.. note::
Expand Down Expand Up @@ -127,44 +128,21 @@ More details on using MPI with IPython can be found :ref:`here <parallelmpi>`.
Using :command:`ipcluster` in PBS mode
--------------------------------------

The PBS mode uses the Portable Batch System [PBS]_ to start the engines. To use this mode, you first need to create a PBS script template that will be used to start the engines. Here is a sample PBS script template:
The PBS mode uses the Portable Batch System [PBS]_ to start the engines.

.. sourcecode:: bash

#PBS -N ipython
#PBS -j oe
#PBS -l walltime=00:10:00
#PBS -l nodes=${n/4}:ppn=4
#PBS -q parallel

cd $$PBS_O_WORKDIR
export PATH=$$HOME/usr/local/bin
export PYTHONPATH=$$HOME/usr/local/lib/python2.4/site-packages
/usr/local/bin/mpiexec -n ${n} ipengine --logfile=$$PBS_O_WORKDIR/ipengine

There are a few important points about this template:

1. This template will be rendered at runtime using IPython's :mod:`Itpl`
template engine.
To start an ipcluster using the Portable Batch System::

2. Instead of putting in the actual number of engines, use the notation
``${n}`` to indicate the number of engines to be started. You can also uses
expressions like ``${n/4}`` in the template to indicate the number of
nodes.
$ ipcluster pbs -n 12

3. Because ``$`` is a special character used by the template engine, you must
escape any ``$`` by using ``$$``. This is important when referring to
environment variables in the template.
The above command will launch a PBS job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:

4. Any options to :command:`ipengine` should be given in the batch script
template.
$ ipcluster pbs -n 12 -q hpcqueue

5. Depending on the configuration of you system, you may have to set
environment variables in the script template.
By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:

Once you have created such a script, save it with a name like :file:`pbs.template`. Now you are ready to start your job::
$ ipcluster pbs -n 12 -q hpcqueue -s mypbscript.sh

$ ipcluster pbs -n 128 --pbs-script=pbs.template
NOTE: ipcluster relies on using PBS job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#PBS -t 1-N) to your script.

Additional command line options for this mode can be found by doing::

Expand All @@ -173,46 +151,48 @@ Additional command line options for this mode can be found by doing::
Using :command:`ipcluster` in SGE mode
--------------------------------------

The SGE mode uses the Sun Grid Engine [SGE]_ to start the engines. To use this mode, you first need to create a SGE script template that will be used to start the engines. Here is a sample SGE script template:
The SGE mode uses the Sun Grid Engine [SGE]_ to start the engines.

.. sourcecode:: bash
To start an ipcluster using Sun Grid Engine::

#!/bin/bash
#$ -V
#$ -m n
#$ -N ipengine-${eid}
#$ -r y
#$ -q sub
#$ -S /bin/bash
$ ipcluster sge -n 12

cd $$HOME/sge
ipengine --logfile=ipengine${eid}
The above command will launch an SGE job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:

There are a few important points about this template:
$ ipcluster sge -n 12 -q hpcqueue

1. This template will be rendered at runtime using IPython's :mod:`Itpl`
template engine.
By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:

2. Instead of putting in the actual id of engines, use the notation
``${eid}`` to indicate where engine id should be inserted.
$ ipcluster sge -n 12 -q hpcqueue -s mysgescript.sh

3. Because ``$`` is a special character used by the template engine, you must
escape any ``$`` by using ``$$``. This is important when referring to
environment variables in the template.
NOTE: ipcluster relies on using SGE job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#$ -t 1-N) to your script.

4. Any options to :command:`ipengine` should be given in the batch script
template.
Additional command line options for this mode can be found by doing::

5. Depending on the configuration of you system, you may have to set
environment variables in the script template.
$ ipcluster sge -h

Once you have created such a script, save it with a name like :file:`sge.template`. Now you are ready to start your job::
Using :command:`ipcluster` in LSF mode
--------------------------------------

$ ipcluster sge -n 12 --sge-script=sge.template
The LSF mode uses the Load Sharing Facility [LSF]_ to start the engines.

To start an ipcluster using the Load Sharing Facility::

$ ipcluster lsf -n 12

The above command will launch an LSF job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:

$ ipcluster lsf -n 12 -q hpcqueue

By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:

$ ipcluster lsf -n 12 -q hpcqueue -s mysgescript.sh

NOTE: ipcluster relies on using LSF job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#BSUB -J ipengine[1-N]) to your script.

Additional command line options for this mode can be found by doing::

$ ipcluster sge -h
$ ipcluster lsf -h

Using :command:`ipcluster` in SSH mode
--------------------------------------
Expand Down Expand Up @@ -394,4 +374,5 @@ the log files to us will often help us to debug any problems.

.. [PBS] Portable Batch System. http://www.openpbs.org/
.. [SGE] Sun Grid Engine. http://www.sun.com/software/sge/
.. [LSF] Load Sharing Facility. http://www.platform.com/
.. [SSH] SSH-Agent http://en.wikipedia.org/wiki/Ssh-agent

0 comments on commit 1505a87

Please sign in to comment.