From c86e9ee0db45abe47c7fcec060cf79ed41c3df15 Mon Sep 17 00:00:00 2001 From: Matthew Rocklin Date: Wed, 2 Jan 2019 19:36:45 -0800 Subject: [PATCH 1/4] Update index This elevates the batch processing approach above interactive, and does some slight modification of language. --- docs/source/index.rst | 62 +++++++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 26 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index b2b9cce..b42b188 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -8,59 +8,68 @@ The Dask-MPI project makes it easy to deploy Dask from within an existing MPI environment, such as one created with the common MPI command-line launchers ``mpirun`` or ``mpiexec``. Such environments are commonly found in high performance supercomputers, academic research institutions, and other clusters where MPI -has already been installed. Dask-MPI provides a convenient interface for -launching your cluster either from within a batch script or directly from the -command-line. +has already been installed. -Example: --------- +Dask-MPI provides two convenient interfaces to launch Dask, either from within +a batch script or directly from the command-line. -You can launch a Dask cluster directly from the command-line using the ``dask-mpi`` -command and specifying a scheduler JSON file. +Batch Script Example +-------------------- -.. code-block:: bash - - mpirun -np 4 dask-mpi --scheduler-file /path/to/scheduler.json - -You can then access this cluster from a batch script or an interactive session -(such as a Jupyter Notebook) by referencing the scheduler file. +You can turn your batch Python script into an MPI executable +with the ``dask_mpi.initialize`` function. .. code-block:: python + from dask_mpi import initialize + initialize() + from dask.distributed import Client - client = Client(scheduler_file='/path/to/scheduler.json') + client = Client() # Connect this local process to remote workers +This makes your Python script launchable directly with ``mpirun`` or ``mpiexec``. -Example: --------- +.. code-block:: bash -Alternatively, you can turn your batch Python script into an MPI executable -simply by using the ``initialize`` function. + mpirun -np 4 python my_client_script.py -.. code-block:: python +This deploys the Dask scheduler and workers as well as the user's Client +process within a single cohesive MPI computation. - from dask_mpi import initialize - initialize() - from dask.distributed import Client - client = Client() # Connect this local process to remote workers +Command Line Example +-------------------- -which makes your Python script launchable directly with ``mpirun`` or ``mpiexec``. +Alternatively you can launch a Dask cluster directly from the command-line +using the ``dask-mpi`` command and specifying a scheduler file where Dask can +write connection information. .. code-block:: bash - mpirun -np 4 python my_client_script.py + mpirun -np 4 dask-mpi --scheduler-file ~/dask-scheduler.json + +You can then access this cluster either from a separate batch script or from an +interactive session (such as a Jupyter Notebook) by referencing the same scheduler +file that ``dask-mpi`` created. + +.. code-block:: python + + from dask.distributed import Client + client = Client(scheduler_file='~/dask-scheduler.json') + .. toctree:: :maxdepth: 1 + :hidden: :caption: Getting Started install - interactive batch + interactive .. toctree:: :maxdepth: 1 + :hidden: :caption: Detailed use cli @@ -68,6 +77,7 @@ which makes your Python script launchable directly with ``mpirun`` or ``mpiexec` .. toctree:: :maxdepth: 1 + :hidden: :caption: Help & Reference howitworks From 93dc16c79946a09a301bf944accc0fe8551f639c Mon Sep 17 00:00:00 2001 From: Matthew Rocklin Date: Wed, 2 Jan 2019 19:38:34 -0800 Subject: [PATCH 2/4] use a space in dask distributed link --- docs/source/history.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/history.rst b/docs/source/history.rst index 1838286..9b501be 100644 --- a/docs/source/history.rst +++ b/docs/source/history.rst @@ -1,7 +1,7 @@ History ======= -This package came out of the Dask_Distributed_ project with help from the +This package came out of the `Dask Distributed`_ project with help from the Pangeo_ collaboration. The original code was contained in the ``distributed.cli.dask_mpi`` module and the original tests were contained in the ``distributed.cli.tests.test_dask_mpi`` module. The impetus for pulling Dask-MPI out of Dask-Distributed was provided by feedback @@ -9,5 +9,5 @@ on the Dask Distributted `Issue 2402 Date: Wed, 2 Jan 2019 19:41:41 -0800 Subject: [PATCH 3/4] Only suggest pip install This now seems to be common practice --- docs/source/install.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 97926ad..aa6dd31 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -29,11 +29,7 @@ To install Dask-MPI from source, clone the repository from `github git clone https://github.com/dask/dask-mpi.git cd dask-mpi - python setup.py install - -or use ``pip`` locally if you want to install all dependencies as well:: - - pip install -e . + pip install . You can also install directly from git master branch:: From 58c3eecb2117edbcddec69265ce81c344c65c3c3 Mon Sep 17 00:00:00 2001 From: Matthew Rocklin Date: Wed, 2 Jan 2019 20:00:32 -0800 Subject: [PATCH 4/4] link to dask-jobqueue --- docs/source/index.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index b42b188..620f221 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -58,6 +58,15 @@ file that ``dask-mpi`` created. client = Client(scheduler_file='~/dask-scheduler.json') +Use Job Queuing System Directly +------------------------------- + +You can also use `Dask Jobqueue `_ to deploy Dask +directly on a job queuing system like SLURM, SGE, PBS, LSF, Torque, or others. +This can be especially nice when you want to dynamically scale your cluster +during your computation, or for interactive use. + + .. toctree:: :maxdepth: 1 :hidden: