A simple in-memory message broker/queue/switch
This is a simple in-memory message broker (or "queue" or "switch"). A deployment consists of one switch server process and multiple clients. A low-level protocol allows clients to connect to the switch server process and:
- create named message queues
- enqueue messages in named queues
- subscribe to new messages in queues
- a unique "correlation id", allowing replies to be associated to requests
- an optional "reply to" queue name
- a string payload
The client library currently contains a single high-level pattern for synchronous RPC clients and servers. Other patterns (e.g. for pub/sub) could be added.
$ git clone git://github.com/djs55/message-switch $ cd message-switch $ make
Run the server (by default it will bind to 127.0.0.1:8080)
Use the CLI to list the available queues (there will be none):
$ ./main.native list
Use the CLI to execute an RPC against a service with queue name "foo" (this will block):
$ ./main.native call foo --body hello
Use the diagnostics command to see the queued message:
$ ./main.native diagnostics
Use the CLI to start an RPC server servicing the queue name "foo"
$ ./main.native serve foo --program /bin/cat
At this point the original RPC call will unblock and print "hello".
Use the diagnostics command to see the message has been removed:
$ ./main.native diagnostics
(I'd like to improve this, if you have any ideas then let me know)
The low-level protocol is currently using HTTP. Perhaps in future we should use something a bit quicker: a binary protocol using ZeroMQ? Clients make the following requests:
<client identifier>: allows the switch to associate the socket with a single client. When the last socket is closed resources associated with the client are deallocated. I'm currently using a Unix PID as an identifier.
<Some name>: creates a persistent queue with name
<None>: creates a queue with a fresh name which will be deallocated when the last connection opened by the client closes. These queues are created temporarily for RPC replies, the idea being that a client that has closed all connections has given up waiting for the reply and so the queue is not needed.
<string, Message.t>: enqueue a message in a particular queue
<message id>: signal that a message has been processed and should be deleted. A service should presumably only Ack a message once it has committed its effects to disk.
<name>: subscribe to new messages in a named queue
<id, timeout>: blocking poll for new messages in subscribed queues
The RPC pattern
A client will:
<some client id>
- Reply_to <-
Create <None>(used for the RPC reply)
- Create (in case it doesn't exist)
- Send <payload, service name>
- repeatedly poll until a reply turns up
The server will:
<service name>(in case it doesn't exist)
- repeatedly poll
<Transfer>until a request turns up
- process a request, Send
<message id>to mark the message as processed and delete it