Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
581 lines (361 sloc) 13.9 KB

Build reliable, traceable, distributed systems with ZeroMQ (ZeroRPC) by Jérôme Petazzoni from dotCloud

Presenter: Jérôme Petazzoni ( (@jpetazzo)

PyCon 2012 presentation page:



Video running time: 36:22

ZeroRPC GitHub repo:

This talk is about ZeroRPC, a library for doing RPC using ZeroMQ, that was recently open-sourced by dotCloud.


Why did we build an RPC system?

(00:22 / 36:22 into the video)

  • dotCloud is a PaaS.
    • We deploy, monitor, and scale your apps (in the cloud!)
  • Many moving parts
  • ... On a large distributed cluster

What the architecture looks like

(00:45 / 36:22 into the video)

Easy Requirements

(01:08 / 36:22 into the video)

  • Expose arbitrary code with minimal modification
    • if we can do import foo;
    • we want to be able to do foo = RemoteService(...);
  • Self-documented system
    • Want to see methods, signatures, and docstrings

More Difficult Requirements

(02:09 / 36:22 into the video)

  • Propagate exceptions
  • Language-agnostic
  • Brokerless, highly available, fast, support fan-in/fan/out - Not necessarily all at the same time!
  • We want to trace & profile nested calls

Why not {x}?

(04:06 / 36:22 into the video)

  • Why not HTTP
  • Why not AMQP

What we came up with

(04:48 / 36:22 into the video)


  • Based on ZeroMQ and MessagePack

  • Supports everything we needed!

  • Multiple implementations

    • Internal "reference" implementation in Python

    • Public "alternative" implementation with gevent

      Internal Node.js implementation (so-so)

Using it

Example: unmodified code

(05:45 / 36:22 into the video)

$ zerorpc-client --server --bind tcp:// urllib

Example: calling code

(06:14 / 36:22 into the video)

  • From the command-line (for testing):
$ zerorpc-client tcp:// quote "hello pycon"
connecting to "tcp://"
  • From Python code:
>>> import zerorpc
>>> remote_urllib = zerorpc.Client()
>>> remote_urllib.connect('tcp://')
>>> remote_urllib.quote('hello pycon')

Example: introspection

(06:38 / 36:22 into the video)

We can list methods:

$ zerorpc-client tcp:// | grep ^q
quote                       quote('abc def') -> 'abc%20def'
quote_plus                  Quote the query fragment of a URL; replacing ' ' with '+'

We can see signatures and docstrings:

$ zerorpc-client tcp:// quote_plus -?
connecting to "tcp://"

quote_plus(s, safe='')

Quote the query fragment of a URL; replacing ' ' with '+'

Example: exceptions

(06:47 / 36:22 into the video)

$ zerorpc-client tcp:// quote_plus
connecting to "tcp://"
Traceback (most recent call last):
zerorpc.exceptions.RemoteError: Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/", line 201, in _async_task
    functor.pattern.process_call(self._context, socket, event, functor)
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/", line 74, in process_call
    result = context.middleware_call_procedure(functor, *event.args)
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/", line 88, in middleware_call_procedure
    return procedure(*args, **kwargs)
  File "/Users/marca/dev/git-repos/zerorpc-python/zerorpc/", line 55, in __call__
    return self._functor(*args, **kargs)
TypeError: quote_plus() takes at least 1 argument (0 given)

Example: load balancing

(07:07 / 36:22 into the video)

Start a load balancing hub:

$ cat foo.yml
in: "tcp://*:1111"
out: "tcp://*:2222"
type: queue
$ foo.yml

Start (at least) one worker:

$ zerorpc-client --server tcp://localhost:2222 urllib

Now connect to the "in" side of the hub:

$ zerorpc-client tcp://localhost:1111

Example: high availability

(07:30 / 36:22 into the video)

Start a local HAProxy in TCP mode, dispatching requests to 2 or more remote services or hubs:

$ cat haproxy.cfg
listen zerorpc
    mode tcp
    server backend_a localhost:2222 check
    server backend_b localhost:3333 check
$ haproxy -f haproxy.cfg

Start (at least) one backend:

$ zerorpc-client --server --bind tcp://0:2222 urllib

Now connect to HAProxy:

$ zerorpc-client tcp://localhost:1111

Non-example: PUB/SUB

(08:01 / 36:22 into the video)

Not in public repo -- yet

  • Broadcast a message to a group of nodes
    • But if a node leaves and rejoins, he'll lose messages
  • Send a continuous stream of information
    • But if a speaker or listener leaves and rejoins...

You generally don't want to do this!

Better pattern: ZeroRPC streaming with gevent

Example: streaming

(09:10 / 36:22 into the video)

  • Server code returns an iterator
  • Client code gets an iterator
  • Small messages, high latency? No problem! - Server code will pre-push elements - Client code will notify server if pipeline runs low
  • Huge messages? No problem! - Big data sets can be nicely chunked - They don't have to fit entirely in memory - Don't worry about timeouts anymore
  • Also supports long polling

Example: tracing

(10:15 / 36:22 into the video)

Not in public repo yet

Implementation details

(11:16 / 36:22 into the video)

This will be useful if:

  • You think you might want to use ZeroRPC
  • You think you might want to hack ZeroRPC
  • You want to implement something similar
  • You just happen to love distributed systems


(11:50 / 36:22 into the video)


(13:28 / 36:22 into the video)

  • MessagePack
  • In our tests, msgpack is more efficient than JSON, BSON, YAML:
    • 20-50x faster
    • serialized output is 2x smaller or better
$ pip install msgpack-python
>>> import msgpack
>>> bytes = msgpack.dumps(data)

Wire format

(14:09 / 36:22 into the video)

Request: (headers, method_name, args)

  • headers dict
    • no mandatory header
    • carries the protocol version number
    • used for tracing in our in-house version
  • args
    • list of arguments
    • no named parameters

Response: (headers, ERR|OK|STREAM, value)


(15:20 / 36:22 into the video)

  • 0MQ does not detect disconnections (or rather, it works hard to hide them)
  • You can't know when the remote is gone
  • Original implementation: 30s timeout
  • Published implementation: heartbeat


(16:13 / 36:22 into the video)

  • Expose a few special calls:
    • _zerorpc_list to list calls
    • _zerorpc_name to know who you're talking to
    • _zerorpc_ping (redundant with the previous one)
    • _zerorpc_help to retrieve the docstring of a call
    • _zerorpc_args to retrieve the argspec of a call
    • _zerorpc_inspect to retrieve everything at once


(17:10 / 36:22 into the video)

  • Published implementation does not include any kind of naming/discovery
  • In-house version uses a flat YAML file, mapping service names to 0MQ addresses and socket types, but ashamed to publish this :-)
  • In progress: use DNS records
    • SRV for host+port
    • TXT for 0MQ socket type (not sure about this!)
  • In progress: registration of services

Security: there is none

(18:00 / 36:22 into the video)

  • No security at all in 0MQ
    • assumes that you are on a private, internal network
  • If you need to run "in the wild", use SSL:
    • bind 0MQ socket on localhost
    • run stunnel (with client cert verification)
  • In progress: authentication layer
  • dotCloud API is actually ZeroRPC, exposed through a HTTP/ZeroRPC gateway
  • In progress: standardization of this gateway

Tracing (not published yet)

(19:26 / 36:22 into the video)

How it works: all calls and responses are logged to a central place, along with a trace_id unique to each sequence of calls.

Tracing: trace_id

(21:14 / 36:22 into the video)

  • Each call has a trace_id
  • The trace_id is propagated to subcalls
  • The trace_id is bound to a local context (think thread local storage)
  • When making a call:
    • If there is a local trace_id, use it
    • If there is none ("root call"), generate one (GUID)
  • trace_id is passed in all calls and responses

Note: this is not (yet) in the GitHub repository:

Tracing: trace collection

(21:42 / 36:22 into the video)

  • If a message (sent or received) has a trace_id, we send out the following things:
    • trace_id
    • call name (or, for return values, OK|ERR+exception)
    • current process name and hostname
    • timestamp

Internal details: the collection is built on top of the standard :mod:`logging` module.

Tracing: trace storages

(22:10 / 36:22 into the video)

  • Traces are sent to a Redis key/value store
    • each trace_id is associated with a list of traces
    • we keep some per=service counters
    • Redis persistence is disabled
    • entries are given a TTL so they expire automatically
    • entries were initially JSON (for easy debugging)
    • ... then "compressed" with msgpack to save space
    • approximately 16 GB of traces per day

Internal details: the logging handler does not talk directly to Redis; it sends traces to a collector (which itself talks to Redis)

The problem with being synchronous

(23:19 / 36:22 into the video)

  • Original implementation was synchronous
  • Long-running calls blocked the server
  • Workaround: multiple workers and a hub
  • Wastes resources
  • Does not work well for very long calls
    • Deployment and provisioning of new cluster nodes
    • Deployment and scaling of user apps

Note: this is not specific to ZeroRPC (Preforking servers, threaded servers, WSGI...)

First shot at asynchronicity

(24:28 / 36:22 into the video)

  • Send asynchronous events & setup callbacks
  • "Please do foo(42) and send the result to this other place once you're done"
  • We tried this. We failed.
    • distributed spaghetti code
    • trees falling in the forest with no one to hear them
  • Might have worked better if we had...
    • better support in the library
    • better naming system
    • something to make sure we don't lose calls (a kind of distributed FSM, maybe?)

Gevent to the rescue!

(26:02 / 36:22 into the video)

  • Gevent --
  • Write synchronous code (a.k.a.: don't rewrite your services)
  • Uses coroutines to achieve concurrency
  • No forks, no threads (no problems? :-))
  • Monkey patch standard library (to replace blocking calls with async versions)
  • Achieve "unlimited" concurrency server-side

The version published on GitHub uses gevent.

Show me the code!

(27:17 / 36:22 into the video)

$ pip install git+git://


Doesn't have:

  • tracing
  • naming
  • helpers for PUB/SUB and PUSH/PULL
  • authentication


(28:25 / 36:22 into the video)