Simple Clojure Job Distribution Service
Clojure Java Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

simple-dist is a very simple job distribution service for Clojure.

simple-dist was written under two assumptions: The jobs are pure functions, i.e. they have no side effects and are only run to produce a result, and they run for at least some non-trivial amount of time, like one second.

simple-dist requires Clojure 1.2.


If you don't have it already, install leiningen. Download dependencies and compile Java code:

lein deps
lein compile-java

With everything in place, run a worker:


The worker defines a few functions, like "fib" to calculate Fibonacci numbers, and provides an interface to call those functions remotely.

Now run a distributor:


A distributor provides the same interface as a worker, but doesn't do the work itself. Instead it distributes the jobs to one or more workers that it connects to.

Now, in a Clojure REPL we can do:

(use '
(def dist (connect "localhost" 1099))

This connects to the distributor, which uses port 1099. Then, let's get a handle on the "fib" function:

(def dist-fib (worker-function dist "fib" (fn [x] 'local)))

The third argument to "worker-function" should actually be the local "fib" function - if the connection dies or the worker doesn't implement the "fib" function, the local function is used instead. Here we use a different function to see whether or not the job is processed remotely or locally.

Let's call it:

(dist-fib 20)
=> 6765

Wonderful! Just to see if it will still work, kill the worker process, and try again:

(dist-fib 20)
=> local

Now restart the worker process and try again:

(dist-fib 20)
=> 6765

Run a few long-running jobs:

(pmap dist-fib (range 45))

and go to http://localhost:8080. There you'll see the simple web interface of the distributor that tells you which workers it knows about and what jobs they're currently executing.

Have fun!