Deploy Dask on Marathon
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ci/scripts
dask_marathon
.gitignore
.travis.yml
README.rst
conftest.py
docker-compose.yml
environment.yml
setup.cfg
setup.py

README.rst

Dask-Marathon

Build Status

Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.

This project is a proof of concept only. No guarantee of quality or future maintenance is made.

Run

It's not yet clear how to expose all of the necessary options to a command line interface. For now we're doing everything manually.

Make an IOLoop running in a separate thread:

from threading import Thread
from tornado.ioloop import IOLoop
loop = IOLoop()
thread = Thread(target=loop.start); thread.daemon = True
thread.start()

Start a Scheduler in that IOLoop

from distributed import Scheduler
s = Scheduler(loop=loop)
s.start()

Start a dask-marathon cluster. Give it the scheduler, information about the application, and the address of the Marathon master. This example assumes that dask-worker is available in the system environment, but see see marathon.MarathonApp for possible keyword args to define the application, including docker containers, etc..

from dask_marathon import AdaptiveCluster
cluster = AdaptiveCluster(s, marathon_address='http://localhost:8080',
                          cpus=1, mem=1000, executable='dask-worker',
                          **kwargs)

Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.

from distributed import Client
c = Client(s.address)

future = c.submit(lambda x: x + 1, 10)

TODO

  • Deploy the scheduler on the cluster
  • Support a command line interface

Docker Testing Harness

This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.

Requires:

  • docker version >= 1.11.1
  • docker-compose version >= 1.7.1
export DOCKER_IP=127.0.0.1
docker-compose up

Run py.test:

py.test dask-marathon

Additional notes

  • Master and Agents have dask.distributed installed its github repository
  • Mesos container names: - mesos_master - mesos_slave_one - mesos_slave_two

Web UIs

History

Mesos Docker-compose solution originally forked from https://github.com/bobrik/mesos-compose

This project was then forked from dask-mesos.