dask-worker processes on Marathon in response to load on a Dask
scheduler. This creates a Marathon application of dask-worker processes. It
watches a Dask Scheduler object in the local process and, based on current
requested load, scales the Marathon application up and down.
This project is a proof of concept only. No guarantee of quality or future maintenance is made.
It's not yet clear how to expose all of the necessary options to a command line interface. For now we're doing everything manually.
Make an IOLoop running in a separate thread:
from threading import Thread from tornado.ioloop import IOLoop loop = IOLoop() thread = Thread(target=loop.start); thread.daemon = True thread.start()
Start a Scheduler in that IOLoop
from distributed import Scheduler s = Scheduler(loop=loop) s.start()
Start a dask-marathon cluster. Give it the scheduler, information about the
application, and the address of the Marathon master. This example assumes that
dask-worker is available in the system environment, but see see
marathon.MarathonApp for possible keyword args to define the application,
including docker containers, etc..
from dask_marathon import AdaptiveCluster cluster = AdaptiveCluster(s, marathon_address='http://localhost:8080', cpus=1, mem=1000, executable='dask-worker', **kwargs)
Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.
from distributed import Client c = Client(s.address) future = c.submit(lambda x: x + 1, 10)
- Deploy the scheduler on the cluster
- Support a command line interface
Docker Testing Harness
This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.
- docker version >= 1.11.1
- docker-compose version >= 1.7.1
export DOCKER_IP=127.0.0.1 docker-compose up
- Master and Agents have dask.distributed installed its github repository
- Mesos container names: - mesos_master - mesos_slave_one - mesos_slave_two
- http://localhost:5050/ for Mesos master UI
- http://localhost:5051/ for the first Mesos agent UI
- http://localhost:5052/ for the second Mesos agent UI
- http://localhost:8080/ for Marathon UI
- http://localhost:8888/ for Chronos UI
Mesos Docker-compose solution originally forked from https://github.com/bobrik/mesos-compose
This project was then forked from dask-mesos.