Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Fetching contributors…
Cannot retrieve contributors at this time
377 lines (247 sloc) 12.9 KB
Notes for The Guide
+++ A Service-Oriented Queue Device
- how to route to workers based on service names
- kind of like custom pubsub but with answers going back to clients
- collection of patterns
Least Recently Used
Asynchronous Client-Server
Suicidal Snail
Lazy Pirate
Simple Pirate
Paranoid Pirate
- how to make an error logging console
- how to make a TCP-to-0MQ bridge, with examples (email)
- pseudo-recursive job queues
- zqueue class
- worked example
- single point of failure
- presence detection
- router trees
- example of monitoring bus
- cluster broker, send out state
- capture via some visual tool...?
- monitoring and logging
Then we can use this to start building some reusable pieces:
- timer device
- name service
- custom load-balancing
- custom publish-subscribe
- stateful publish-subscribe
- reliable request-reply
- logging device
- reactor (Matt Weinstein's)
+++ Presence Detection
- peers come and go
- if we want to route to them explicitly, we need to know if they are present
- heartbeating, liveness, etc.
- fresh data, failover, etc.
- purging old identities from routing tables
- example of eight robots and console
- robots come and go...
+++ A ZeroMQ Name Service
- name service
translate logical name into connect or bind string
service runs locally, connect via ipc://zns
gets name updates asynchronously from central server
also local zns lookup file
using zpl syntax
pubsub state / refresh example
how to map names?
- XXX -> tcp://lo:5050 if I'm on server 1
- XXX -> tcp://somename:5050
-> does ZMQ do host lookup? Doesn't seem like it...
-> resolve host...
+++ Custom pub/sub
- req/rep socket for subscriptions
- timeout for resubscriptions
- pub/sub socket on dynamic endpoints
+++ Generating Identities
+++ Setting Queue Limits
- setting queue limits prevents nodes from overflowing memory
- by default 0MQ does not set any limits
- example of application that will overflow memory
- publish to subscribe socket but don't read
- now fix this by setting queue limit, ZMQ_HWM
- actual behaviour with automagic LWM
- how this works on different socket types
- 'exception' on each socket type, from man page
- adding capacity for disk offload, ZMQ_SWAP
- creating persistent queues using identity
- example of HWM 1 plus massive SWAP
+++ Reliable Request-Reply
We'll create a reliable request-reply application that uses XREQ and XREP and a simple resend mechanism. When this works between two peers we'll show how it scales across a request-reply broker to effectively create edge-to-edge reliability. We'll also open up the message format for request-reply and explore identities in horrible detail.
+++ Configuration Distribution
We'll look at how to dynamically configure a network of devices using a central configuration broker.
+++ Logging Subsystem
many applications
many subscribers
broker in the middle
persistent logfiles
replay via subscribe socket
+++ Failover and Recovery
We'll look at how to handle crashes in a distributed architecture. For each pattern there is an ideal architecture, and we'll explore each of these with worked examples.
+++ Encrypted Publish-Subscribe
We'll look at how to secure pubsub data against snooping. The actual technique involves out-of-band exchange of keys, symmetric encryption, and a broker that helps the thing along. Hopefully all fairly easy to make, as usual, using 0MQ.
+++ Building a Multicast Bus
We'll now look at how the pgm: and epgm: protocols work. With pgm:, the network switch basically acts as a hardware FORWARDER device.
++++ Customized Publish-Subscribe
- use identity to route message explicitly to A or B
- not using PUBSUB at all but XREP/????
- limitations: no multicast, only TCP
- how to scale with devices...
When a client activates, it chooses a random port that is not in use and creates a SUB socket listening for all traffic on it. The client then sends a message via REQ to the publisher containing the port that it is listening on. The publisher receives this message, acknowledges it, and creates a new pub socket specific to that client. All published events specific to this client go out that socket.
When the client deactivates, it sends a message to the publisher with the port to deactivate and close.
You end up creating a lot more PUB sockets on your server end and doing all of the filtering at the server. This sounds acceptable to you.
I didn't need to do this to avoid network bandwidth bottlenecks; I created this to enforce some security and entitlements.
+++ A Clock Device
We'll look at various ways of building a timer into a network. A clock device sends out a signal (a message) at more or less precise intervals so that other nodes can use these signals for internal timing.
+++ A Logging Device
We'll look at a device that acts as a remote logger.
+++ Building Bridges
zmq_poll(3) lets you mix ZMQ sockets and native system sockets, which is very useful if you have to resort to actual TCP for something, e.g. bridging an existing TCP protocol with 0MQ. We'll build a couple of bridges.
+++ Writing a Reactor
In this section we'll build a simple reactor framework that turns the zmq_poll(3) function inside out into an event-driven framework.
+++ Exotic Devices
Now we'll use the reactor to build some exotic devices. It should be much easier and readable than using zmq_poll.
+++ Serializing Data
Examples of using Protocol Buffers and other options.
- ipc://name
- connects two processes on a single box
- supports all messaging patterns
- typical use case is for multithreading apps
- runs on Unix domain sockets (not available on Windows, OpenVMS)
- permissions issues:
> Since I want to work under /tmp, this all had to be done programatically. My
> server now mkdir -p's a socket subdirectory and chmod 777's it. The server
> creates and binds the socket in that folder, and then chmod 777's it. The
> server must be run as root (which is fine for my project luckily). If it is
> run as a normal user, the client's still timeout.
- tcp://ipaddress:port
- bind to address:port
- bind to *:5555
- localhost
- also bind to interface: lo:port, eth1:port, etc.
- Linux: eth1, eth2, eth3
- Mac OS X: en1, en2, en3
- Solaris: e1000g, etc.
- connect to remote address: host:port
- pgm://address;multicastgroup:port
- address can be interface name
- requires decent hardware support
- means enterprise level switches with IGNP snooping
- some routers also support PGM
- runs over IP, requires root privileges
- more standard
- rate-limited protocol, sender must define bandwidth
- pgm is currently broken
- epgm://address;multicastgroup:port
- encapsulated in UDP packets
- requires decent hardware support
- does not require root access
- non-standard to pgm
- add peer example
- exclusive lock on peer
- for owning other party
- solve reverse connection
- e.g. to cross firewall
- you need to add a bind to allow the client to accept a connection
- could be usecase for EXCLUSIVE socket
XREQ is like PUSH+PULL, XREP is like PUSH+PULL+routing
* Tutorial: chat client/server
* How to handle a REQ/REP socket without it blocking forever
* Related question: how to stop a single thread blocked on recv()
* How to write services in C
- memory leak checking
- security
- etc.
- ZPL config, zfl, etc.
* How do we tunnel 0MQ connections over 0MQ connections?
- e.g. to get 3-4 ports into DMZ via one single port
- two devices, one reads, one writes
- over other protocols: HTTP, SSH, etc...?
- acts as split device, ...
* Cross-language string sending tutorial
I highly recommend that you try out the simpler topology and *verifying* that 0mq cannot keep up with your message rates when publishing all data to all clients. With smart topics the client can reject the data *very* fast. A 1 GB or 10 Gb switched network can also move quite a bit of data without a lot of blocking behavior in the hardware too. You may be able to save yourself a lot of unnecessary work, so just try it.
- explain in
- debugging message flow
- using a queue device that prints out message parts
- debugging versions of devices...
- heartbeating
- set HWM to 1 message only
+++ The Wire-Level Protocol
- writing a thin client in JavaScript
+++ Building a Language Binding
+++ Tuning 0MQ
High performance multiple i/o threads
- for HP applications, several threads
- then, set affinity on sockets
+++ Contributing to 0MQ
+++ What's Missing from 0MQ
* Multicast
- unbreak PGM
- make PGM work on Windows
- exclusive pair reconnect is broken
- location lookup
- abstracted interfaces
- other IPC mechanisms, e.g. shared memory
- IPC on OpenVMS, Windows
- upstreaming of subscriptions/unsubs
- timeouts for receiver missing?
- persistence & reliability
Change proposals
- defaults for identity are to crash the publisher
- handling crashing peers
- non-blocking send/recv
- reliable pub
- pub connects to all subs
- subs send request to pub
- reliable messaging over xreq / xrep
- how to do timer events
- there is no way to disconnect; second connection creates two
- need to destroy the socket
- list known encoding libraries with pros / cons of each
- list each OS
For External Use Only!
More Connections Than Oprah!
Save The Fail Whale!
DEVOPS_BORAT say "Why Not?"
Now With More Vitamins!
Sufficiently Advanced Tech
High Lifestyle, Low Carbon
Must Has Cat to Enter
- how to send multipart job and execute / cancel
* Client-server data distribution (like custom pubsub), with some restrictions.
Either of the two client socket types send requests and replies as a multipart message with two parts. The first part is a unique UUID, that identifies the client. The second is the actual message blob. The ZMQ_XREP socket uses the identity to know what ØMQ connection to send the message to.
What this means... take deep breath... is that you can use ZMQ_XREQ and ZMQ_XREP to build servers that do authentication and arbitrary routing. Don't like the existing topic pubsub? You can build your own. Here's how to build an authenticated pubsub server:
* Client creates ZMQ_XREQ sends its authentication credentials and subscription pattern as a request.
* Service reads credentials, authenticates them, and if they are OK, stores the client identity in a table of "authorized identities". This is in effect treating an identity as an abstracted connection.
* Service builds routing tables using subscription pattern, so that any new message can be routed to a set of identities.
* Service does separate zmq_send() to its ZMQ_XREP socket for each matching subscriber, each time specifying the correct identity.
* Service maintains its table of identities periodically to remove old identities.
* Client periodically resends its credentials ("keep-alive"), which server handles by reading on the socket.
It is not trivial, and should maybe sit in the "Advanced Concepts" section, but this use case is one that so many new users ask about that it's good to know at least that ØMQ handles it.
Here is, I think, how to build an authenticated pubsub service using ØMQ. The design handles authentication, arbitrary routing criteria, and flushing of dead client connections.
You will use ZMQ_XREQ at the client side and ZMQ_XREP at the service side. There is a simple protocol between client and server that is loosely modelled on HTTP:
* Client sends messages containing: identity (ØMQ adds this automatically), authentication credentials, and subscription criteria.
* Server sends messages containing: identity (automatic), error/success code, and content.
The server has two main data structures:
* A connection table that stores authenticated identities and timestamps. Each identity corresponds to a client connection.
* A routing table that maps message keys (e.g. topic value) to identities. There are techniques for doing [ high speed message matching].
The client has a single thread that polls on its ZMQ_XREQ socket. At regular time intervals it (re)sends its authentication and subscription message. This acts as a keep-alive.
The server has a two threads, one that polls on its ZMQ_XREP socket and one that receives outgoing messages and passes them to the polling thread via inproc sockets. The polling thread processes any incoming client requests and updates its data structures. It processes any outgoing messages by looking up the message key in its routing table and then sending the message (one to one) to all matching identities. As it does this it deletes any expired identities.
That's it. Sorry if I got some things wrong here, it's theory and not based on running code.
- setsockopt validation
- identities don't start with zero byte
- integer values are not negative
- fixes to documentation
- REP sockets can't drop data on HWM because they are lock-step
- HWM says n/a for PULL and for SUB, which is wrong
Jump to Line
Something went wrong with that request. Please try again.