Recursive Message Broker
This is a sketch of a recursive messaging protocol, broker, and client libraries, inspired by AMQP 0-91, PubSubHubBub, STOMP and reversehttp. It's quite different to AMQP 1.0 (but it may be instructive to compare the two approaches).
Currently, the project includes
- a server (written in OCaml), with adapters for
- Hop's own protocol,
- a subset of AMQP 0-9-1, and
- XHR streaming of messages to and from the broker.
- a web-based console for the server
- an OSX GUI for the server
- messaging for the web
Honestly, not meant to be production software... yet.
Messaging à la AMQP 0-91 can be broken down into a few core pieces:
transmission and receipt of messages (publishes, deliveries, and gets)
subscription management (subsuming enrollment, bindings, consumers and relays)
directory (naming of resources in the broker)
object management (creation and destruction of remote resources)
AMQP itself, being a first mover in its space, isn't as orthogonal as it could be. It can be greatly simplified without losing anything of value. This experiment is intended to demonstrate one possible way of paring each of the core pieces of AMQP-style messaging back to their essences.
what recursive means in this context
doing things this way gives you shovels (relays) for free
and effortless interop with legacy messaging networks (including UDP, SMTP, IMAP, HTTP etc)
and effortless federation
and a big step closer to a sensible semantics for transactions
relays (including active client connections!) are just nodes in the network, addressable like any other - so
(post! somerelay (post! someremotenode ...))and so on is the way to cause things to happen remotely.
Compiling the server
The server is written in OCaml. To build and run the server, you will need:
- OCaml itself, version 3.12 or newer
- OCaml Findlib; I have used 1.2.8 and 1.3.1, but older versions may well work
- libev installed somewhere that Findlib can find it
- python to generate parts of the protocol codecs
Make sure you have
ocamlfind etc. on your
path. Then, in the
server subdirectory, run
make. It should first
compile Lwt, which is included as a
third-party library, and then should proceed to compiling the server
ocamlfind can't find
libev, try setting (and exporting) the
LIBRARY_PATH to point to
the include and lib directories containing
To run the server, simply run
./server/hop_server.native, or just
make run from within the
Working with the management and monitoring webpages
If you want to edit and/or recompile the server's built-in webpages, you will need to have installed
Compiling the Java client library
You will need a recent JDK, and Ant v1.6 or newer. Change to the
java subdirectory, and run
ant. You will end up with a file
build/lib/hop.jar, which contains the client library and some test
Open three terminals. Run the server in one of them. You should see output like the following:
hop ALPHA, Copyright (C) 2012 Tony Garnock-Jones. This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public License (version 3 or later) for details. info: ("Node bound" "factory" "factory") info: ("Registered node class" "queue") info: ("Registered node class" "fanout") info: ("Registered node class" "direct") info: ("Node bound" "meta" "direct") info: ("Node create ok" "direct" ("meta") "" "" "meta") info: ("Node bound" "amq.direct" "direct") info: ("Node create ok" "direct" ("amq.direct") "" "" "amq.direct") info: ("Node bound" "amq.fanout" "fanout") info: ("Node create ok" "fanout" ("amq.fanout") "" "" "amq.fanout") info: ("Accepting connections" "AMQP" "5672") info: ("Accepting connections" "HTTP" "5678") info: ("Accepting connections" "Hop" "5671") info: ("Waiting for milestone" "AMQP ready") info: ("Achieved milestone" "AMQP ready") info: ("Waiting for milestone" "HTTP ready") info: ("Achieved milestone" "HTTP ready") info: ("Waiting for milestone" "Hop ready") info: ("Achieved milestone" "Hop ready") info: ("Achieved milestone" "Server initialized")
In the second terminal, run the consuming half of the Java test program pair:
java -cp hop.jar hop.Test1 localhost
In the third, run the producing half:
java -cp hop.jar hop.Test3 localhost
Obviously the wire protocol itself here is the simplest thing that could possibly work, and you'd never use anything like this inefficient in a real system. That said, this is what's there right now:
(post <routing-key> <message> <subscription-token>) - Instructs the
receiving node to route (or process) the given
message according to
routing-key. Different kinds of nodes will do different
things here, and in particular, will interpret the routing key
differently. Queues, for example, will ignore the routing key and will
deliver the message to only one of their active subscribers, whereas
exchanges will generally match the routing key against their active
subscriptions and will deliver the message on to all matches.
(subscribe <routing-key-filter> <target-node> <target-routing-key> <reply-node> <reply-routing-key>) - Instructs the receiving node to
create a new subscription. The new subscription will only route
messages matching the
routing-key-filter, which is interpreted on a
per-node-type basis as above for
routing-key. Matching messages will
be sent to
post!, with a routing key of
reply-node parameter, if nonempty,
instructs the receiving node to send confirmation of subscription
(along with a token that can be used with
unsubscribe below) to the
given address and routing key. If
reply-node is empty, no
confirmation of subscription is sent.
(unsubscribe <token>) - Instructs the receiving node to delete a
previously established subscription. The
token comes from the
subscribe-ok message sent to
reply-node after a successful
(create <class-name> <argument> <reply-node> <reply-routing-key>) -
Instructs the receiving object factory node to construct a new
instance of the given
class-name, with the given
to the constructor. The
reply-routing-key are used
to send confirmation of completion to some waiting node.
Copyright and licensing
Hop is Copyright 2010, 2011, 2012 Tony Garnock-Jones firstname.lastname@example.org.
Hop is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Hop is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with Hop. If not, see http://www.gnu.org/licenses/.