Deploy Dask on Marathon
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ci/scripts
dask_marathon
.gitignore
.travis.yml
README.rst
conftest.py
docker-compose.yml
environment.yml
setup.cfg
setup.py

README.rst

Dask-Marathon

Build Status

Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.

This project is a proof of concept only. No guarantee of quality or future maintenance is made.

Run

It's not yet clear how to expose all of the necessary options to a command line interface. For now we're doing everything manually.

Make an IOLoop running in a separate thread:

from threading import Thread
from tornado.ioloop import IOLoop
loop = IOLoop()
thread = Thread(target=loop.start); thread.daemon = True
thread.start()

Start a Scheduler in that IOLoop

from distributed import Scheduler
s = Scheduler(loop=loop)
s.start()

Start a dask-marathon cluster. Give it the scheduler, information about the application, and the address of the Marathon master. This example assumes that dask-worker is available in the system environment, but see see marathon.MarathonApp for possible keyword args to define the application, including docker containers, etc..

from dask_marathon import AdaptiveCluster
cluster = AdaptiveCluster(s, marathon_address='http://localhost:8080',
                          cpus=1, mem=1000, executable='dask-worker',
                          **kwargs)

Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.

from distributed import Client
c = Client(s.address)

future = c.submit(lambda x: x + 1, 10)

TODO

  • Deploy the scheduler on the cluster
  • Support a command line interface

Docker Testing Harness

This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.

Requires:

  • docker version >= 1.11.1
  • docker-compose version >= 1.7.1
export DOCKER_IP=127.0.0.1
docker-compose up

Run py.test:

py.test dask-marathon

Additional notes

  • Master and Agents have dask.distributed installed its github repository
  • Mesos container names: - mesos_master - mesos_slave_one - mesos_slave_two

Web UIs

History

Mesos Docker-compose solution originally forked from https://github.com/bobrik/mesos-compose

This project was then forked from dask-mesos.