Borges: A Mesos framework for Org Babel

The universe (which others call the Library) is composed of an indefinite and perhaps infinite number of hexagonal galleries… The idealists argue that the hexagonal rooms are a necessary form of absolute space or, at least, of our intuition of space.

• Jorge Luis Borges, The Library of Babel

Motivation

Org Babel is good for scheduling workloads locally or remotely via SSH, and the Borges scheduler is an attempt at taking it to the next level by using Mesos to abstract the resources used for running the workloads. This also makes it possible to orchestrate its execution via header arguments, similarly to how Org Babel does it to enable literate programming.

Usage examples

For example, having a org-file.org document with the following content:

#+TITLE: Code block that should run forever

#+name: run-forever
#+BEGIN_SRC sh 
echo "Started running"
while true; do
  echo "Still running..."
  sleep 1
done
#+END_SRC

We can submit it to the scheduler for execution:

curl -X POST -d@org-file.org 127.0.0.1:8888/org

Though we can specify much more about how to run it, for example by setting :dockerize t, the code block will run within the specified Docker image. Furthermore, we can specify the required resources in order to run it via the :cpu and :mem arguments:

curl -X POST 192.168.0.7:8888/org -d '

#+name: redis-server
#+header: :cpu 1 :mem 128 :dockerize t :image redis
#+BEGIN_SRC sh
redis-server
#+END_SRC

#+name: set-something-in-redis
#+header: :cpu 1 :mem 128
#+BEGIN_SRC sh
while true; do 
  echo "SET timestamp `date \"+%s\"`" | nc 127.0.0.1 6379
  sleep 1
done
#+END_SRC

#+name: get-something-from-redis
#+header: :cpu 1 :mem 128 :dockerize t :image ubuntu
#+BEGIN_SRC sh
while true; do 
  echo "Current value in Redis:"
  echo "GET timestamp" | nc 127.0.0.1 6379
  sleep 1
done
#+END_SRC

' 2>&1

We can confirm that the code blocks are running fine in Mesos web UI:

Or check its logs within the Sandbox:

Running it

Once getting the binary (go build ...), or go get

go get github.com/wallyqs/borges

Then it can be started as follows:

borges -f org/examples/borges-config.org -logtostderr=true

I0301 20:01:23.158484 02578 borges.go:313] Reading #+setupfile:  org/examples/borges-config.org
I0301 20:01:23.164691 02578 scheduler.go:232] Initializing mesos scheduler driver
I0301 20:01:23.165067 02578 scheduler.go:640] Starting the scheduler driver...
I0301 20:01:23.165752 02578 http_transporter.go:275] http transport listening on 192.168.0.7:50717
I0301 20:01:23.165873 02578 borges.go:292] API server running at  192.168.0.7:8888
I0301 20:01:24.166711 02578 scheduler.go:659] Mesos scheduler driver started with PID=scheduler(1)@192.168.0.7:50717
I0301 20:01:24.168792 02578 scheduler.go:808] Scheduler driver running.  Waiting to be stopped.
I0301 20:01:24.181682 02578 scheduler.go:272] New master master@192.168.0.7:5050 detected
I0301 20:01:24.181742 02578 scheduler.go:331] No credentials were provided. Attempting to register scheduler without authentication.
I0301 20:01:24.187725 02578 scheduler.go:441] Framework registered with ID=20150301-191330-117483712-5050-1515-0001
I0301 20:01:24.188582 02578 borges.go:87] [ 20150301-191330-117483712-5050-1515-O1403 ] Received Offer with cpus = 4  mem = 2812

Configuration itself is done via a la Org mode via a #+setupfile: with at least the following:

#+TITLE: Borges Configuration

#+mesos_master: 192.168.0.7:5050
#+borges_bind:  192.168.0.7
#+borges_port:  8888

• #+mesos_master: is the address of a Mesos master
• #+borges_bind: is the address that the schedule will be announcing to Mesos
• #+borges_port: is the port where we can send HTTP requests with Org mode in the body.

Contributing/Status

Currently the project is still very experimental, though feedback is very welcome!