Skip to content
Simple distributed computing in python
Branch: master
Clone or download
Latest commit 518124d Jun 17, 2016
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
myriad
.gitignore
LICENSE.txt
README.md Just a tidyup Nov 17, 2015
setup.cfg Prepare some pypi things Oct 7, 2015
setup.py

README.md

myriad

Some simple distributed computing components using only the python standard library. The motivation for the project was to create a simple distributed map-reduce style interface that can act as a drop-in replacement for python's inbuilt map and the itertools.imap family.

Installation

myriad can be installed in the usual way with

python setup.py install

It should be installed on all computers that will do any processing.

Usage

myriad contains client and server components. The server manages input and output queues, which clients can access. The basic pattern is to start a server, push work to its input queues, launch clients, and finally collect results from the server. An example is given in the myriad.components module, which provides the myriad command-line entry point. Here's a near-full example.

from myriad.components import MyriadServer
from random import randint

host = 'localhost'
port = '12345'
key = 'auth_key'
worker = ... # mysterious function that transforms inputs to outputs

# Create a server
server = MyriadServer(worker, port, key)
# Push some jobs to the server
for _ in xrange(5):
    server.put(randint(1, 3))

# Start a client
subprocess.Popen(['myriad', '--client',
    '--host', 'localhost', '--port', port, '--key', key])
# Collect results from server
for result in server:
    print "Server got back '{}'".format(result)

In the above we start a subprocess running myriad with the --client option. Of course this line could be amended to, for example, submit clients to a cluster. In common with similar packages, the function worker must be pickleable.

You can’t perform that action at this time.