By Greg Hazel and Steven Hazel.
monocle straightens out event-driven code using Python's generators. It aims to be portable between event-driven I/O frameworks, and currently supports Twisted and Tornado.
It's for Python 2.5 and up; the syntax it uses isn't supported in older versions of Python. (Versions before 2.7 require the ordereddict module.)
Here's a simple monocle program that runs two concurrent lightweight processes (called "o-routines") using Tornado's event loop. One is an HTTP server, and the other makes an HTTP request:
import monocle monocle.init("tornado") from monocle import Return from monocle.stack import eventloop from monocle.stack.network import add_service from monocle.stack.network.http import HttpClient, HttpHeaders, HttpServer @monocle.o def hello_http(req): content = "Hello, World!" headers = HttpHeaders() headers['Content-Length'] = len(content) headers['Content-Type'] = 'text/plain' yield Return(200, headers, content) @monocle.o def request(): client = HttpClient() resp = yield client.request('http://127.0.0.1:8088/') print resp.code, resp.body add_service(HttpServer(hello_http, 8088)) monocle.launch(request) eventloop.run()
It's important that code be dapper and well-dressed, so if you prefer,
you can don the monocle and use this handy shortcut for
from monocle import _o @_o def request(): client = HttpClient() resp = yield client.request('http://127.0.0.1:8088/') print resp.code, resp.body
It's true, this violates Python's convention that underscores indicate variables for internal use. But rules are for breaking. Live a little.
Event-driven code can be efficient and easy to reason about, but it often splits up procedures in an unpleasant way. Here's an example of a blocking function to read a request from a user, query a database, and return a result:
def do_cmd(conn): cmd = conn.read_until("\n") if cmd.type == "get-address": user = db.query(cmd.username) conn.write(user.address) else: conn.write("unknown command")
Here's the same thing in event-driven style, using callbacks:
def get_cmd(conn): conn.read_until("\n", callback=handle_cmd) def handle_cmd(conn, cmd): if cmd.type == "get-address": # keep track of the conn so we can write the response back! def callback(result): handle_user_query_result(conn, result) db.query(cmd.username, callback) else: conn.write("unknown command") def handle_user_query_result(conn, user): conn.write(user.address)
What started out as a single function in the blocking code has
expanded here into four functions (counting the
handle_cmd). In real event-driven code,
this kind of thing happens a lot. Any time we want to do I/O, we
have to register a new handler and return back out to the event loop
to let other things happen while we wait for the I/O to finish. It
would be nice if we had some way to tell the event loop to call back
into the middle of our function, so we could just continue where we
Fortunately, Python has a mechanism that lets us do exactly that, called generators. Monocle uses generators to straighten out event-driven code.
Here's the monocle equivalent of the event-based code above:
@_o def do_cmd(conn): cmd = yield conn.read_until("\n") if cmd.type == "get-address": user = yield db.query(cmd.username) yield conn.write(user.address) else: yield conn.write("unknown command")
It's event-driven for efficient concurrency, but otherwise looks a lot
like the original blocking code. Each time you see the word
in the code above, the o-routine is returning back up to the event
loop and waiting to be called back when the I/O it requested
This approach is a kind of cooperative concurrency that makes for simpler code than callback-based event-driven code, but which we think is easier to reason about than multi-threaded code.
In ordinary Python generators, the norm is to think of
yield as in
crops: the generator yields a value. In monocle o-routines, it's
helpful to think of
yield as in traffic.
yield conn.read(10) in
an o-routine means "yield to other o-routines until we finish reading
monocle is similar to, and takes inspiration from:
- Twisted's inlineCallbacks
- BitTorrent's yielddefer (used in the 5.x mainline client)
- Go's goroutines (and CSP generally)
- Haskell's I/O monad