Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/history.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
History
=======

This package came out of the Dask_Distributed_ project with help from the
This package came out of the `Dask Distributed`_ project with help from the
Pangeo_ collaboration. The original code was contained in the ``distributed.cli.dask_mpi``
module and the original tests were contained in the ``distributed.cli.tests.test_dask_mpi``
module. The impetus for pulling Dask-MPI out of Dask-Distributed was provided by feedback
on the Dask Distributted `Issue 2402 <https://github.com/dask/distributed/issues/2402>`_.

Development history for these original files was preserved.

.. _Dask_Distributed: https://github.com/dask/distributed
.. _`Dask Distributed`: https://github.com/dask/distributed
.. _Pangeo: https://pangeo.io
69 changes: 44 additions & 25 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,66 +8,85 @@ The Dask-MPI project makes it easy to deploy Dask from within an existing MPI
environment, such as one created with the common MPI command-line launchers
``mpirun`` or ``mpiexec``. Such environments are commonly found in high performance
supercomputers, academic research institutions, and other clusters where MPI
has already been installed. Dask-MPI provides a convenient interface for
launching your cluster either from within a batch script or directly from the
command-line.
has already been installed.

Example:
--------
Dask-MPI provides two convenient interfaces to launch Dask, either from within
a batch script or directly from the command-line.

You can launch a Dask cluster directly from the command-line using the ``dask-mpi``
command and specifying a scheduler JSON file.
Batch Script Example
--------------------

You can turn your batch Python script into an MPI executable
with the ``dask_mpi.initialize`` function.

.. code-block:: python

from dask_mpi import initialize
initialize()

from dask.distributed import Client
client = Client() # Connect this local process to remote workers

This makes your Python script launchable directly with ``mpirun`` or ``mpiexec``.

.. code-block:: bash

mpirun -np 4 dask-mpi --scheduler-file /path/to/scheduler.json
mpirun -np 4 python my_client_script.py

You can then access this cluster from a batch script or an interactive session
(such as a Jupyter Notebook) by referencing the scheduler file.
This deploys the Dask scheduler and workers as well as the user's Client
process within a single cohesive MPI computation.

.. code-block:: python

from dask.distributed import Client
client = Client(scheduler_file='/path/to/scheduler.json')
Command Line Example
--------------------

Alternatively you can launch a Dask cluster directly from the command-line
using the ``dask-mpi`` command and specifying a scheduler file where Dask can
write connection information.

Example:
--------
.. code-block:: bash

Alternatively, you can turn your batch Python script into an MPI executable
simply by using the ``initialize`` function.
mpirun -np 4 dask-mpi --scheduler-file ~/dask-scheduler.json

.. code-block:: python
You can then access this cluster either from a separate batch script or from an
interactive session (such as a Jupyter Notebook) by referencing the same scheduler
file that ``dask-mpi`` created.

from dask_mpi import initialize
initialize()
.. code-block:: python

from dask.distributed import Client
client = Client() # Connect this local process to remote workers
client = Client(scheduler_file='~/dask-scheduler.json')

which makes your Python script launchable directly with ``mpirun`` or ``mpiexec``.

.. code-block:: bash
Use Job Queuing System Directly
-------------------------------

You can also use `Dask Jobqueue <https://jobqueue.dask.org>`_ to deploy Dask
directly on a job queuing system like SLURM, SGE, PBS, LSF, Torque, or others.
This can be especially nice when you want to dynamically scale your cluster
during your computation, or for interactive use.

mpirun -np 4 python my_client_script.py

.. toctree::
:maxdepth: 1
:hidden:
:caption: Getting Started

install
interactive
batch
interactive

.. toctree::
:maxdepth: 1
:hidden:
:caption: Detailed use

cli
api

.. toctree::
:maxdepth: 1
:hidden:
:caption: Help & Reference

howitworks
Expand Down
6 changes: 1 addition & 5 deletions docs/source/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,7 @@ To install Dask-MPI from source, clone the repository from `github

git clone https://github.com/dask/dask-mpi.git
cd dask-mpi
python setup.py install

or use ``pip`` locally if you want to install all dependencies as well::

pip install -e .
pip install .

You can also install directly from git master branch::

Expand Down