Dask Jobqueue Scheduler Daemon

The dask jobqueue is an excellent interface to use dask with typical HPCC job queueing systems like SLURM or PBS.

NOTE: only supports SLURM at the moment. But others should be as well. Make a PR!

However, it doesn’t provide an actual scheduling server that you can connect to for launching jobs. This simple module provides this.

Just install:

pip install git+https://github.com/salotz/dask_scheduler.git

Then from the cluster with the job queue system run this command (for SLURM):

dask_scheduler slurm

You should see something like:

Scheduler address: tcp://10.3.8.48:42089
Dashboard port:

Depending on the network settings you might not get a dashboard

See dask_scheduler slurm --help for all of the options.

This will start it with the default settings and should give you a URL to use.

Then you just need to connect to it with a dask client to submit jobs:

from dask.distributed import Client

client = Client("http://10.3.8.48:42089")

def work_func(b):

    return 3 + b

results = client.submit(work_func, 3).result()

results = []
for result in client.map(work_func, [0,1,2,3,4]):
    results.append(result.result())