Skip to content

Commit

Permalink
Add dask-jobqueue and dask-drmaa links to HPC setup docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mrocklin committed Apr 20, 2018
1 parent a842d44 commit cb002d1
Showing 1 changed file with 28 additions and 0 deletions.
28 changes: 28 additions & 0 deletions docs/source/setup/hpc.rst
Expand Up @@ -14,6 +14,34 @@ labs. These systems commonly have the following attributes:
3. A high performance network interconnect, such as Infiniband
4. Little or no node-local storage

Where to start
--------------

Most of this page documents best practices to use Dask on an HPC cluster. This
is technical and aimed both at users with some experience deploying Dask and
also system administrators.

New users may instead prefer to start with one of the following projects, which
provide easy high-level access to Dask using resource managers that are
commonly deployed on HPC systems:

1. `dask-jobqueue <https://dask-jobqueue.readthedocs.io>`_ for use with PBS,
SLURM, and SGE resource managers
2. `dask-drmaa <https://github.com/dask/dask-drmaa>`_ for use with any DRMAA
compliant resource manager

They provide interfaces that look like the following:

.. code-block:: python
from dask_jobqueue import PBSCluster
cluster = PBSCluster(# TODO realistic example)
cluster.start_workers(100) # Start 100 jobs that match the description above
from dask.distributed import Client
client = Client(cluster) # Connect to that cluster
Using a Shared Network File System and a Job Scheduler
------------------------------------------------------
Expand Down

0 comments on commit cb002d1

Please sign in to comment.