Browse files

Updated for 0MQ/3.2

  • Loading branch information...
1 parent 020212c commit febf1c4f8bd377ac981e33002408b645f49799b8 @hintjens hintjens committed Sep 7, 2012
Showing with 7,593 additions and 7,473 deletions.
  1. +57 −42 chapter1.txt
  2. +252 −220 chapter2.txt
  3. +109 −133 chapter3.txt
  4. +50 −41 chapter4.txt
  5. +19 −36 chapter5.txt
  6. +10 −0 chapter6.txt
  7. 0 examples/C#/{rrserver.cs → rrworker.cs}
  8. 0 examples/C++/{rrserver.cpp → rrworker.cpp}
  9. +2 −0 examples/C/.gitignore
  10. +2 −20 examples/C/asyncsrv.c
  11. +89 −0 examples/C/espresso.c
  12. +6 −4 examples/C/msgqueue.c
  13. +2 −2 examples/C/mtserver.c
  14. +1 −1 examples/C/{rrserver.c → rrworker.c}
  15. +2 −2 examples/C/tripping.c
  16. +5 −23 examples/C/wuproxy.c
  17. +4 −6 examples/C/zhelpers.h
  18. 0 examples/CL/{rrserver.asd → rrworker.asd}
  19. 0 examples/CL/{rrserver.lisp → rrworker.lisp}
  20. 0 examples/Clojure/{rrserver.clj → rrworker.clj}
  21. 0 examples/Erlang/{rrserver.es → rrworker.es}
  22. 0 examples/F#/{rrserver.fsx → rrworker.fsx}
  23. 0 examples/Go/{rrserver.go → rrworker.go}
  24. 0 examples/Haskell/{rrserver.hs → rrworker.hs}
  25. 0 examples/Haxe/{rrserver.hx → rrworker.hx}
  26. 0 examples/Java/{rrserver.java → rrworker.java}
  27. 0 examples/Lua/{rrserver.lua → rrworker.lua}
  28. 0 examples/Node.js/{rrserver.js → rrworker.js}
  29. 0 examples/PHP/{rrserver.php → rrworker.php}
  30. 0 examples/Perl/{rrserver.pl → rrworker.pl}
  31. 0 examples/Python/{rrserver.py → rrworker.py}
  32. 0 examples/Racket/{rrserver.rkt → rrworker.rkt}
  33. 0 examples/Ruby/{rrserver.rb → rrworker.rb}
  34. 0 examples/Scala/{rrserver.scala → rrworker.scala}
  35. 0 examples/Tcl/{rrserver.tcl → rrworker.tcl}
  36. +201 −126 images/fig15.eps
  37. +22 −19 images/fig15.html
  38. BIN images/fig15.png
  39. +22 −19 images/fig15.txt
  40. +359 −272 images/fig16.eps
  41. +37 −33 images/fig16.html
  42. BIN images/fig16.png
  43. +37 −33 images/fig16.txt
  44. +230 −345 images/fig17.eps
  45. +23 −40 images/fig17.html
  46. BIN images/fig17.png
  47. +23 −40 images/fig17.txt
  48. +312 −309 images/fig21.eps
  49. +40 −41 images/fig21.html
  50. BIN images/fig21.png
  51. +40 −41 images/fig21.txt
  52. +376 −158 images/fig22.eps
  53. +41 −33 images/fig22.html
  54. BIN images/fig22.png
  55. +41 −33 images/fig22.txt
  56. +157 −122 images/fig23.eps
  57. +33 −21 images/fig23.html
  58. BIN images/fig23.png
  59. +33 −21 images/fig23.txt
  60. +193 −40 images/fig24.eps
  61. +21 −5 images/fig24.html
  62. BIN images/fig24.png
  63. +21 −5 images/fig24.txt
  64. +10 −40 images/fig25.eps
  65. +1 −3 images/fig25.html
  66. BIN images/fig25.png
  67. +1 −3 images/fig25.txt
  68. +54 −70 images/fig26.eps
  69. +7 −7 images/fig26.html
  70. BIN images/fig26.png
  71. +7 −7 images/fig26.txt
  72. +54 −136 images/fig27.eps
  73. +3 −9 images/fig27.html
  74. BIN images/fig27.png
  75. +3 −9 images/fig27.txt
  76. +116 −181 images/fig28.eps
  77. +13 −17 images/fig28.html
  78. BIN images/fig28.png
  79. +13 −17 images/fig28.txt
  80. +25 −21 images/fig29.eps
  81. +5 −5 images/fig29.html
  82. BIN images/fig29.png
  83. +5 −5 images/fig29.txt
  84. +168 −137 images/fig30.eps
  85. +17 −21 images/fig30.html
  86. BIN images/fig30.png
  87. +17 −21 images/fig30.txt
  88. +198 −36 images/fig31.eps
  89. +21 −5 images/fig31.html
  90. BIN images/fig31.png
  91. +21 −5 images/fig31.txt
  92. +36 −208 images/fig32.eps
  93. +5 −22 images/fig32.html
  94. BIN images/fig32.png
  95. +5 −22 images/fig32.txt
  96. +191 −57 images/fig33.eps
  97. +22 −7 images/fig33.html
  98. BIN images/fig33.png
  99. +22 −7 images/fig33.txt
  100. +57 −187 images/fig34.eps
  101. +7 −22 images/fig34.html
  102. BIN images/fig34.png
  103. +7 −22 images/fig34.txt
  104. +182 −64 images/fig35.eps
  105. +22 −7 images/fig35.html
  106. BIN images/fig35.png
  107. +22 −7 images/fig35.txt
  108. +64 −184 images/fig36.eps
  109. +7 −15 images/fig36.html
  110. BIN images/fig36.png
  111. +7 −15 images/fig36.txt
  112. +86 −286 images/fig37.eps
  113. +4 −12 images/fig37.html
  114. BIN images/fig37.png
  115. +4 −12 images/fig37.txt
  116. +7 −13 images/fig38.eps
  117. +4 −4 images/fig38.html
  118. BIN images/fig38.png
  119. +4 −4 images/fig38.txt
  120. +410 −34 images/fig39.eps
  121. +23 −3 images/fig39.html
  122. BIN images/fig39.png
  123. +23 −3 images/fig39.txt
  124. +32 −118 images/fig40.eps
  125. +1 −5 images/fig40.html
  126. BIN images/fig40.png
  127. +1 −5 images/fig40.txt
  128. +20 −106 images/fig41.eps
  129. +2 −6 images/fig41.html
  130. BIN images/fig41.png
  131. +2 −6 images/fig41.txt
  132. +106 −20 images/fig42.eps
  133. +6 −2 images/fig42.html
  134. BIN images/fig42.png
  135. +6 −2 images/fig42.txt
  136. +86 −165 images/fig43.eps
  137. +7 −21 images/fig43.html
  138. BIN images/fig43.png
  139. +7 −21 images/fig43.txt
  140. +132 −483 images/fig44.eps
  141. +21 −39 images/fig44.html
  142. BIN images/fig44.png
  143. +21 −39 images/fig44.txt
  144. +371 −248 images/fig45.eps
  145. +39 −28 images/fig45.html
  146. BIN images/fig45.png
  147. +39 −28 images/fig45.txt
  148. +261 −238 images/fig46.eps
  149. +28 −17 images/fig46.html
  150. BIN images/fig46.png
  151. +28 −17 images/fig46.txt
  152. +299 −168 images/fig47.eps
  153. +17 −18 images/fig47.html
  154. BIN images/fig47.png
  155. +17 −18 images/fig47.txt
  156. +166 −323 images/fig48.eps
  157. +18 −17 images/fig48.html
  158. BIN images/fig48.png
  159. +18 −17 images/fig48.txt
  160. +210 −96 images/fig49.eps
  161. +13 −13 images/fig49.html
  162. BIN images/fig49.png
  163. +13 −13 images/fig49.txt
  164. +186 −606 images/fig50.eps
  165. +17 −37 images/fig50.html
  166. BIN images/fig50.png
  167. +17 −37 images/fig50.txt
  168. +534 −238 images/fig51.eps
Sorry, we could not display the entire diff because it was too big.
View
99 chapter1.txt
@@ -14,7 +14,7 @@ Thanks to Bill Desmarais, Brian Dorsey, Daniel Lin, Eric Desgranges, Gonzalo Die
Thanks to Stathis Sideris for [http://www.ditaa.org Ditaa], which I used for the diagrams.
-Please use the [$(GIT)/issues issue tracker] for all comments and errata. This version covers the latest stable release of 0MQ (2.2.x) and was published on !date("ddd d mmmm, yyyy"). If you are using 0MQ/3.1 some of the examples and explanations won't be accurate.
+Please use the [$(GIT)/issues issue tracker] for all comments and errata. This version covers the latest stable release of 0MQ (3.2) and was published on !date("ddd d mmmm, yyyy"). If you are using older versions of 0MQ then some of the examples and explanations won't be accurate.
The Guide is originally [/page:all in C], but also in [/php:all PHP], [/py:all Python], [/lua:all Lua], and [/hx:all Haxe]. We've also translated most of the examples into C++, C#, CL, Erlang, F#, Felix, Haskell, Java, Objective-C, Ruby, Ada, Basic, Clojure, Go, Haxe, Node.js, ooc, Perl, and Scala.
.reset
@@ -37,23 +37,23 @@ So we live in a world where the wiring is years ahead of our ability to use it.
Brooks missed free and open source software, which solved that crisis, enabling us to share knowledge efficiently. Today we face another software crisis, but it's one we don't talk about much. Only the largest, richest firms can afford to create connected applications. There is a cloud, but it's proprietary. Our data, our knowledge is disappearing from our personal computers into clouds that we cannot access, cannot compete with. Who owns our social networks? It is like the mainframe-PC revolution in reverse.
-We can leave the political philosophy for another book. The point is that while the Internet offers the potential of massively connected code, the reality is that this is out of reach for most of us, and so, large interesting problems (in health, education, economics, transport, and so on) remain unsolved because there is no way to connect the code, and thus no way to connect the brains that could work together to solve these problems.
+We can leave the political philosophy [http://swsi.info for another book]. The point is that while the Internet offers the potential of massively connected code, the reality is that this is out of reach for most of us, and so, large interesting problems (in health, education, economics, transport, and so on) remain unsolved because there is no way to connect the code, and thus no way to connect the brains that could work together to solve these problems.
There have been many attempts to solve the challenge of connected software. There are thousands of IETF specifications, each solving part of the puzzle. For application developers, HTTP is perhaps the one solution to have been simple enough to work, but it arguably makes the problem worse, by encouraging developers and architects to think in terms of big servers and thin, stupid clients.
-So today people are still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, WebSockets. It remains painful, slow, hard to scale, and essentially centralized. Distributed P2P architectures are mostly for play, not work. How many applications use Skype or Bittorrent to exchange data?
+So today people are still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, Websockets. It remains painful, slow, hard to scale, and essentially centralized. Distributed P2P architectures are mostly for play, not work. How many applications use Skype or Bittorrent to exchange data?
Which brings us back to the science of programming. To fix the world, we needed to do two things. One, to solve the general problem of "how to connect any code to any code, anywhere". Two, to wrap that up in the simplest possible building blocks that people could understand and use //easily//.
It sounds ridiculously simple. And maybe it is. That's kind of the whole point.
+++ 0MQ in a Hundred Words
-0MQ (ZeroMQ, 0\MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. 0MQ is from [http://www.imatix.com iMatix] and is LGPL open source.
+0MQ (ZeroMQ, 0\MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. 0MQ is from [http://www.imatix.com iMatix] and is LGPLv3 open source.
+++ Some Assumptions
-We assume you are using the latest stable release of 0MQ. We assume you are using a Linux box or something similar. We assume you can read C code, more or less, that's the default language for the examples. We assume that when we write constants like PUSH or SUBSCRIBE you can imagine they are really called ZMQ_PUSH or ZMQ_SUBSCRIBE if the programming language needs it.
+We assume you are using the latest 3.2 release of 0MQ. We assume you are using a Linux box or something similar. We assume you can read C code, more or less, that's the default language for the examples. We assume that when we write constants like PUSH or SUBSCRIBE you can imagine they are really called ZMQ_PUSH or ZMQ_SUBSCRIBE if the programming language needs it.
+++ Getting the Examples
@@ -63,9 +63,7 @@ The Guide examples live in the Guide's [https://github.com/imatix/zguide git rep
git clone --depth=1 git://github.com/imatix/zguide.git
[[/code]]
-And then browse the examples subdirectory. You'll find examples by language. If there are examples missing in a language you use, you're encouraged to [http://zguide.zeromq.org/main:translate submit a translation]. This is how the Guide became so useful, thanks to the work of many people.
-
-All examples are licensed under MIT/X11, unless otherwise specified in the source code.
+And then browse the examples subdirectory. You'll find examples by language. If there are examples missing in a language you use, you're encouraged to [http://zguide.zeromq.org/main:translate submit a translation]. This is how the Guide became so useful, thanks to the work of many people. All examples are licensed under MIT/X11.
+++ Ask and Ye Shall Receive
@@ -96,7 +94,7 @@ So let's start with some code. We start of course with a Hello World example. We
+------------+
[[/code]]
-The REQ-REP socket pair is lockstep. The client does zmq_send[3] and then zmq_recv[3], in a loop (or once if that's all it needs). Doing any other sequence (e.g. sending two messages in a row) will result in a return code of -1 from the send or recv call. Similarly the service does zmq_recv[3] and then zmq_send[3] in that order, and as often as it needs to.
+The REQ-REP socket pair is lockstep. The client does zmq_msg_send[3] and then zmq_msg_recv[3], in a loop (or once if that's all it needs). Doing any other sequence (e.g. sending two messages in a row) will result in a return code of -1 from the send or recv call. Similarly the service does zmq_msg_recv[3] and then zmq_msg_send[3] in that order, and as often as it needs to.
0MQ uses C as its reference language and this is the main language we'll use for examples. If you're reading this on-line, the link below the example takes you to translations into other programming languages. Let's compare the same server in C++:
@@ -180,8 +178,9 @@ static char *
s_recv (void *socket) {
zmq_msg_t message;
zmq_msg_init (&message);
- zmq_recv (socket, &message, 0);
- int size = zmq_msg_size (&message);
+ int size = zmq_msg_recv (&message, socket, 0);
+ if (size == -1)
+ return NULL;
char *string = malloc (size + 1);
memcpy (string, zmq_msg_data (&message), size);
zmq_msg_close (&message);
@@ -248,9 +247,9 @@ Here is client application, which listens to the stream of updates and grabs any
Note that when you use a SUB socket you **must** set a subscription using zmq_setsockopt[3] and SUBSCRIBE, as in this code. If you don't set any subscription, you won't get any messages. It's a common mistake for beginners. The subscriber can set many subscriptions, which are added together. That is, if a update matches ANY subscription, the subscriber receives it. The subscriber can also unsubscribe specific subscriptions. Subscriptions are length-specified blobs. See zmq_setsockopt[3] for how this works.
-The PUB-SUB socket pair is asynchronous. The client does zmq_recv[3], in a loop (or once if that's all it needs). Trying to send a message to a SUB socket will cause an error. Similarly the service does zmq_send[3] as often as it needs to, but must not do zmq_recv[3] on a PUB socket.
+The PUB-SUB socket pair is asynchronous. The client does zmq_msg_recv[3], in a loop (or once if that's all it needs). Trying to send a message to a SUB socket will cause an error. Similarly the service does zmq_msg_send[3] as often as it needs to, but must not do zmq_msg_recv[3] on a PUB socket.
-In theory with 0MQ sockets, it does not matter which end connects, and which end binds. However with PUB-SUB sockets, if you bind the SUB socket and connect the PUB socket, the SUB socket may receive old messages, i.e. messages sent before the SUB started up. This is an artifact of the way bind/connect works. It's best to bind the PUB and connect the SUB, if you can.
+In theory with 0MQ sockets, it does not matter which end connects, and which end binds. However in practice there are undocumented differences that I'll come to later. For now, bind the PUB and connect the SUB, unless your network design makes that impossible.
There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, **the subscriber will always miss the first messages that the publisher sends**. This is because as the subscriber connects to the publisher (something that takes a small but non-zero time), the publisher may already be sending messages out.
@@ -271,24 +270,24 @@ So the client subscribes to its chosen zip code and collects a thousand updates
Some points about the publish-subscribe pattern:
-* A subscriber can in fact connect to more than one publisher, using one 'connect' call each time. Data will then arrive and be interleaved so that no single publisher drowns out the others.
+* A subscriber can connect to more than one publisher, using one 'connect' call each time. Data will then arrive and be interleaved ("fair-queued") so that no single publisher drowns out the others.
* If a publisher has no connected subscribers, then it will simply drop all messages.
* If you're using TCP, and a subscriber is slow, messages will queue up on the publisher. We'll look at how to protect publishers against this, using the "high-water mark" later.
* In the current versions of 0MQ, filtering happens at the subscriber side, not the publisher side. This means, over TCP, that a publisher will send all messages to all subscribers, which will then drop messages they don't want.
-This is how long it takes to receive and filter 10M messages on my box, which is an Intel 4 core Q8300, fast but nothing special:
+This is how long it takes to receive and filter 10M messages on my laptop, which is an 2011-era Intel I7, fast but nothing special:
[[code]]
-ph@ws200901:~/work/git/0MQGuide/examples/c$ time wuclient
+ph@nb201103:~/work/git/zguide/examples/c$ time wuclient
Collecting updates from weather server...
-Average temperature for zipcode '10001 ' was 18F
+Average temperature for zipcode '10001 ' was 28F
-real 0m5.939s
-user 0m1.590s
-sys 0m2.290s
+real 0m4.470s
+user 0m0.000s
+sys 0m0.008s
[[/code]]
+++ Divide and Conquer
@@ -443,27 +442,27 @@ void* pub_worker(void* arg){
// NOTE: do NOT reuse this example code, It's broken.
// e.g. topic_msg will be invalid the second time through
while (1){
- zmq_send(pubskt, &topic_msg, ZMQ_SNDMORE);
+ zmq_msg_send(pubskt, &topic_msg, ZMQ_SNDMORE);
zmq_msg_init(&cmd_msg);
- zmq_recv(qskt, &cmd_msg, 0);
+ zmq_msg_recv(qskt, &cmd_msg, 0);
memcpy(&cmd, zmq_msg_data(&cmd_msg), sizeof(uint8_t));
- zmq_send(pubskt, &cmd_msg, ZMQ_SNDMORE);
+ zmq_msg_send(pubskt, &cmd_msg, ZMQ_SNDMORE);
zmq_msg_close(&cmd_msg);
fprintf(stdout, "received cmd %u\n", cmd);
zmq_msg_init(&nb_msg);
- zmq_recv(qskt, &nb_msg, 0);
+ zmq_msg_recv(qskt, &nb_msg, 0);
memcpy(&nb, zmq_msg_data(&nb_msg), sizeof(uint32_t));
- zmq_send(pubskt, &nb_msg, 0);
+ zmq_msg_send(pubskt, &nb_msg, 0);
zmq_msg_close(&nb_msg);
fprintf(stdout, "received nb %u\n", nb);
zmq_msg_init_size(&resp_msg, sizeof(uint8_t));
memset(zmq_msg_data(&resp_msg), 0, sizeof(uint8_t));
- zmq_send(qskt, &resp_msg, 0);
+ zmq_msg_send(qskt, &resp_msg, 0);
zmq_msg_close(&resp_msg);
}
@@ -504,33 +503,35 @@ worker_thread (void *arg) {
}
[[/code]]
-In the end, the problem was that the application was passing sockets between threads, which crashed weirdly. It became legal behavior in 0MQ/2.1, but remains dangerous unless you use a "full memory barrier", and it's something we advise against doing.
+In the end, the problem was that the application was passing sockets between threads, which crashes weirdly. Sockets are not threadsafe. It became legal behavior to migrate sockets from one thread to another in 0MQ/2.1, but this remains dangerous unless you use a "full memory barrier". If you don't know what that means, don't attempt socket migration.
+++ Getting the Context Right
-0MQ applications always start by creating a //context//, and then using that for creating sockets. In C, it's the zmq_init[3] call. You should create and use exactly one context in your process. Technically, the context is the container for all sockets in a single process, and acts as the transport for {{inproc}} sockets, which are the fastest way to connect threads in one process. If at runtime a process has two contexts, these are like separate 0MQ instances. If that's explicitly what you want, OK, but otherwise remember:
+0MQ applications always start by creating a //context//, and then using that for creating sockets. In C, it's the zmq_ctx_new[3] call. You should create and use exactly one context in your process. Technically, the context is the container for all sockets in a single process, and acts as the transport for {{inproc}} sockets, which are the fastest way to connect threads in one process. If at runtime a process has two contexts, these are like separate 0MQ instances. If that's explicitly what you want, OK, but otherwise remember:
-**Do one zmq_init[3] at the start of your main line code, and one zmq_term[3] at the end.**
+**Do one zmq_ctx_new[3] at the start of your main line code, and one zmq_ctx_destroy[3] at the end.**
-If you're using the fork() system call, each process needs its own context. If you do zmq_init[3] in the main process before calling fork(), the child processes get their own contexts. In general you want to do the interesting stuff in the child processes, and just manage these from the parent process.
+If you're using the fork() system call, each process needs its own context. If you do zmq_ctx_new[3] in the main process before calling fork(), the child processes get their own contexts. In general you want to do the interesting stuff in the child processes, and just manage these from the parent process.
+++ Making a Clean Exit
Classy programmers share the same motto as classy hit men: always clean-up when you finish the job. When you use 0MQ in a language like Python, stuff gets automatically freed for you. But when using C you have to carefully free objects when you're finished with them, or you get memory leaks, unstable applications, and generally bad karma.
-Memory leaks are one thing, but 0MQ is quite finicky about how you exit an application. The reasons are technical and painful but the upshot is that if you leave any sockets open, the zmq_term[3] function will hang forever. And even if you close all sockets, zmq_term[3] will by default wait forever if there are pending connects or sends. Unless you set the LINGER to zero on those sockets before closing them.
+Memory leaks are one thing, but 0MQ is quite finicky about how you exit an application. The reasons are technical and painful but the upshot is that if you leave any sockets open, the zmq_ctx_destroy[3] function will hang forever. And even if you close all sockets, zmq_ctx_destroy[3] will by default wait forever if there are pending connects or sends. Unless you set the LINGER to zero on those sockets before closing them.
The 0MQ objects we need to worry about are messages, sockets, and contexts. Luckily it's quite simple, at least in simple programs:
* Always close a message the moment you are done with it, using zmq_msg_close[3].
* If you are opening and closing a lot of sockets, that's probably a sign you need to redesign your application.
-* When you exit the program, close your sockets and then call zmq_term[3]. This destroys the context.
+* When you exit the program, close your sockets and then call zmq_ctx_destroy[3]. This destroys the context.
If you're doing multithreaded work, it gets rather more complex than this. We'll get to multithreading in the next chapter, but because some of you will, despite warnings, will try to run before you can safely walk, below is the quick and dirty guide to making a clean exit in a //multithreaded// 0MQ application.
-First, do not try to use the same socket from multiple threads. No, don't explain why you think this would be excellent fun, just please don't do it. Next, relingerfy and close all sockets, and terminate the context in the main thread. Lastly, this'll cause any blocking receives or polls or sends in attached threads (i.e. which share the same context) to return with an error. Catch that, and then relingerize and close sockets in //that// thread, and exit. Do not terminate the same context twice. The zmq_term in the main thread will block until all sockets it knows about are safely closed.
+First, do not try to use the same socket from multiple threads. No, don't explain why you think this would be excellent fun, just please don't do it. Next, you need to shut down each socket that has ongoing requests. The proper way is to set a low LINGER value (1 second), then close the socket. If your language binding doesn't do this for you automatically when you destroy a context, I'd suggest sending a patch.
+
+Finally, destroy the context. This will cause any blocking receives or polls or sends in attached threads (i.e. which share the same context) to return with an error. Catch that error, and then set linger on, and close sockets in //that// thread, and exit. Do not destroy the same context twice. The zmq_ctx_destroy in the main thread will block until all sockets it knows about are safely closed.
Voila! It's complex and painful enough that any language binding author worth his or her salt will do this automatically and make the socket closing dance unnecessary.
@@ -564,7 +565,7 @@ Let's look at the typical problems we face when we start to connect pieces using
* How do we handle network errors? Do we wait and retry, ignore them silently, or abort?
-Take a typical open source project like [http://hadoop.apache.org/zookeeper/ Hadoop Zookeeper] and read the C API code in [http://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c src/c/src/zookeeper.c]. It's 3,200 lines of mystery and in there is an undocumented, client-server network communication protocol. I see it's efficient because it uses poll() instead of select(). But really, Zookeeper should be using a generic messaging layer and an explicitly documented wire level protocol. It is incredibly wasteful for teams to be building this particular wheel over and over.
+Take a typical open source project like [http://hadoop.apache.org/zookeeper/ Hadoop Zookeeper] and read the C API code in [http://github.com/apache/zookeeper/blob/trunk/src/c/src/zookeeper.c src/c/src/zookeeper.c]. As I write this, in 2010, the code is 3,200 lines of mystery and in there is an undocumented, client-server network communication protocol. I see it's efficient because it uses poll() instead of select(). But really, Zookeeper should be using a generic messaging layer and an explicitly documented wire level protocol. It is incredibly wasteful for teams to be building this particular wheel over and over.
[[code type="textdiagram" title="Messaging as it Starts"]]
+------------+
@@ -628,7 +629,7 @@ And this is 0MQ: an efficient, embeddable library that solves most of the proble
Specifically:
-* It handles I/O asynchronously, in background threads. These communicate with application threads using lock-free data structures, so 0MQ applications need no locks, semaphores, or other wait states.
+* It handles I/O asynchronously, in background threads. These communicate with application threads using lock-free data structures, so concurrent 0MQ applications need no locks, semaphores, or other wait states.
* Components can come and go dynamically and 0MQ will automatically reconnect. This means you can start components in any order. You can create "service-oriented architectures" (SOAs) where services can join and leave the network at any time.
@@ -642,7 +643,7 @@ Specifically:
* It lets you route messages using a variety of patterns such as request-reply and publish-subscribe. These patterns are how you create the topology, the structure of your network.
-* It lets you place pattern-extending "devices" (small brokers) in the network when you need to reduce the complexity of interconnecting many pieces.
+* It lets you create proxies to queue, forward, or capture messages with a single call. Proxies can reduce the interconnection complexity of a network.
* It delivers whole messages exactly as they were sent, using a simple framing on the wire. If you write a 10k message, you will receive a 10k message.
@@ -652,7 +653,7 @@ Specifically:
* It reduces your carbon footprint. Doing more with less CPU means your boxes use less power, and you can keep your old boxes in use for longer. Al Gore would love 0MQ.
-Actually 0MQ does rather more than this. It has a subversive effect on how you develop network-capable applications. Superficially it's just a socket API on which you do zmq_recv[3] and zmq_send[3]. But message processing rapidly becomes the central loop, and your application soon breaks down into a set of message processing tasks. It is elegant and natural. And it scales: each of these tasks maps to a node, and the nodes talk to each other across arbitrary transports. Two nodes in one process (node is a thread), two nodes on one box (node is a process), two boxes on one network (node is a box). With no application code changes.
+Actually 0MQ does rather more than this. It has a subversive effect on how you develop network-capable applications. Superficially it's a socket-inspired API on which you do zmq_msg_recv[3] and zmq_msg_send[3]. But message processing rapidly becomes the central loop, and your application soon breaks down into a set of message processing tasks. It is elegant and natural. And it scales: each of these tasks maps to a node, and the nodes talk to each other across arbitrary transports. Two nodes in one process (node is a thread), two nodes on one box (node is a process), or two boxes on one network (node is a box) - it's all the same, with no application code changes.
+++ Socket Scalability
@@ -740,9 +741,9 @@ As you start to program with 0MQ you will come across one problem more than once
+--------------------------+
|
v +-----------------+ +------------------+
-+-----------------+ | Are you calling | | Call zmq_init |
-| Are you using | | zmq_init more +------->| exactly once in |
-| the inproc +------->| than once? | Yes | every process. |
++-----------------+ | Are you calling | | Call zmq_ctx_new |
+| Are you using | | zmq_ctx_new +------->| exactly once in |
+| the inproc +------->| more than once? | Yes | every process. |
| transport? | Yes | {o} | | |
| {o} | +--------+--------+ +------------------+
+--------+--------+ | No
@@ -780,11 +781,25 @@ If you're using 0MQ in a context where failures are expensive, then you want to
In short: if you have not proven an architecture works in realistic conditions, it will most likely break at the worst possible moment.
++++ Upgrading from 0MQ/2.2 to 0MQ/3.2
+
+In early 2012, 0MQ/3.2 became stable enough for live use and by the time you're reading this, it's what you really should be using. If you are still using 2.2, here's a quick summary of the changes, and how to migrate your code.
+
+The main change in 3.x is that PUB-SUB works properly, as in, the publisher only sends subscribers stuff they actually want. In 2.x, publishers send everything and the subscribers filter. Simple, but not ideal for performance on a TCP network.
+
+Most of the API is backwards compatible, except a few blockheaded changes that went into 3.0 with no real regard to the cost of breaking existing code. The syntax of zmq_send[3] and zmq_recv[3] changed, and ZMQ_NOBLOCK got rebaptised to ZMQ_DONTWAIT. So although I'd love to say, //"you just recompile your code with the latest libzmq and everything will work"//, that's not how it is. For what it's worth, we banned such API breakage afterwards.
+
+So the minimal change for C/C++ apps that use the low-level libzmq API is to replace all calls to zmq_send with zmq_msg_send, and zmq_recv with zmq_msg_recv. In other languages, your binding author may have done the work already. Note that these two functions now return -1 in case of error, and zero or more according to how many bytes were sent or received.
+
+Other parts of the libzmq API became more consistent. We deprecated zmq_init[3] and zmq_term[3], replacing them with zmq_ctx_new[3] and zmq_ctx_destroy[3]. We added zmq_ctx_set[3] to let you configure a context before starting to work with it.
+
+Finally, we added context monitoring via the zmq_ctx_set_monitor[3] call, which lets you track connections and disconnections, and other events on sockets.
+
+++ Warning - Unstable Paradigms!
-Traditional network programming is built on the general assumption that one socket talks to one connection, one peer. There are multicast protocols but they are exotic. When we assume "one socket = one connection", we scale our architectures in certain ways. We create threads of logic where each thread work with one socket, one peer. We place intelligence and state in these threads.
+Traditional network programming is built on the general assumption that one socket talks to one connection, one peer. There are multicast protocols but these are exotic. When we assume "one socket = one connection", we scale our architectures in certain ways. We create threads of logic where each thread work with one socket, one peer. We place intelligence and state in these threads.
-In the 0MQ universe, sockets are clever multithreaded applications that manage a whole set of connections automagically for you. You can't see, work with, open, close, or attach state to these connections. Whether you use blocking send or receive, or poll, all you can talk to is the socket, not the connections it manages for you. The connections are private and invisible, and this is the key to 0MQ's scalability.
+In the 0MQ universe, sockets are doorways to fast little background communications engines that manage a whole set of connections automagically for you. You can't see, work with, open, close, or attach state to these connections. Whether you use blocking send or receive, or poll, all you can talk to is the socket, not the connections it manages for you. The connections are private and invisible, and this is the key to 0MQ's scalability.
Because your code, talking to a socket, can then handle any number of connections across whatever network protocols are around, without change. A messaging pattern sitting in 0MQ can scale more cheaply than a messaging pattern sitting in your application code.
View
472 chapter2.txt
@@ -13,7 +13,7 @@ We'll cover:
* How to handle interrupt signals like Ctrl-C.
* How to shutdown a 0MQ application cleanly.
* How to check a 0MQ application for memory leaks.
-* How to send and receive multipart messages.
+* How to send and receive multi-part messages.
* How to forward messages across networks.
* How to build a simple message queuing broker.
* How to write multithreaded applications with 0MQ.
@@ -30,9 +30,9 @@ Originally the zero in 0MQ was meant as "zero broker" and (as close to) "zero la
+++ The Socket API
-To be perfectly honest, 0MQ does a kind of switch-and-bait on you. Which we don't apologize for, it's for your own good and hurts us more than it hurts you. It presents a familiar BSD socket API but that hides a bunch of message-processing machines that will slowly fix your world-view about how to design and write distributed software.
+To be perfectly honest, 0MQ does a kind of switch-and-bait on you. Which we don't apologize for, it's for your own good and hurts us more than it hurts you. It presents a familiar socket-based API but that hides a bunch of message-processing engines that will slowly fix your world-view about how to design and write distributed software.
-Sockets are the de-facto standard API for network programming, as well as being useful for stopping your eyes from falling onto your cheeks. One thing that makes 0MQ especially tasty to developers is that it uses a standard socket API. Kudos to Martin Sustrik for pulling this idea off. It turns "Message Oriented Middleware", a phrase guaranteed to send the whole room off to Catatonia, into "Extra Spicy Sockets!" which leaves us with a strange craving for pizza, and a desire to know more.
+Sockets are the de-facto standard API for network programming, as well as being useful for stopping your eyes from falling onto your cheeks. One thing that makes 0MQ especially tasty to developers is that it uses sockets and messages instead of some other arbitrary set of concepts. Kudos to Martin Sustrik for pulling this off. It turns "Message Oriented Middleware", a phrase guaranteed to send the whole room off to Catatonia, into "Extra Spicy Sockets!" which leaves us with a strange craving for pizza, and a desire to know more.
Like a nice pepperoni pizza, 0MQ sockets are easy to digest. Sockets have a life in four parts, just like BSD sockets:
@@ -42,7 +42,7 @@ Like a nice pepperoni pizza, 0MQ sockets are easy to digest. Sockets have a life
* Plugging sockets onto the network topology by creating 0MQ connections to and from them (see zmq_bind[3], zmq_connect[3]).
-* Using the sockets to carry data by writing and receiving messages on them (see zmq_send[3], zmq_recv[3]).
+* Using the sockets to carry data by writing and receiving messages on them (see zmq_msg_send[3], zmq_msg_recv[3]).
Which looks like this, in C:
@@ -62,15 +62,15 @@ zmq_connect (mousetrap, "tcp://192.168.55.221:5001");
// Wait for juicy mouse to arrive
zmq_msg_t mouse;
zmq_msg_init (&mouse);
-zmq_recv (mousetrap, &mouse, 0);
+zmq_msg_recv (&mouse, mousetrap, 0);
// Destroy the mouse
zmq_msg_close (&mouse);
// Destroy the socket
zmq_close (mousetrap);
[[/code]]
-Note that sockets are always void pointers, and messages (which we'll come to very soon) are structures. So in C you pass sockets as-such, but you pass addresses of messages in all functions that work with messages, like zmq_send[3] and zmq_recv[3]. As a mnemonic, realize that "in 0MQ all your sockets are belong to us", but messages are things you actually own in your code.
+Note that sockets are always void pointers, and messages (which we'll come to very soon) are structures. So in C you pass sockets as-such, but you pass addresses of messages in all functions that work with messages, like zmq_msg_send[3] and zmq_msg_recv[3]. As a mnemonic, realize that "in 0MQ all your sockets are belong to us", but messages are things you actually own in your code.
Creating, destroying, and configuring sockets works as you'd expect for any object. But remember that 0MQ is an asynchronous, elastic fabric. This has some impact on how we plug sockets into the network topology, and how we use the sockets after that.
@@ -94,7 +94,7 @@ To create a connection between two nodes you use zmq_bind[3] in one node, and zm
* Your application code cannot work with these connections directly; they are encapsulated under the socket.
-Many architectures follow some kind of client-server model, where the server is the component that is most stable, and the clients are the components that are most dynamic, i.e. they come and go the most. There are sometimes issues of addressing: servers will be visible to clients, but not necessarily vice-versa. So mostly it's obvious which node should be doing zmq_bind[3] (the server) and which should be doing zmq_connect[3] (the client). It also depends on the kind of sockets you're using, with some exceptions for unusual network architectures. We'll look at socket types later.
+Many architectures follow some kind of client-server model, where the server is the component that is most static, and the clients are the components that are most dynamic, i.e. they come and go the most. There are sometimes issues of addressing: servers will be visible to clients, but not necessarily vice-versa. So mostly it's obvious which node should be doing zmq_bind[3] (the server) and which should be doing zmq_connect[3] (the client). It also depends on the kind of sockets you're using, with some exceptions for unusual network architectures. We'll look at socket types later.
Now, imagine we start the client //before// we start the server. In traditional networking we get a big red Fail flag. But 0MQ lets us start and stop pieces arbitrarily. As soon as the client node does zmq_connect[3] the connection exists and that node can start to write messages to the socket. At some stage (hopefully before messages queue up so much that they start to get discarded, or the client blocks), the server comes alive, does a zmq_bind[3] and 0MQ starts to deliver messages.
@@ -112,15 +112,15 @@ Each time a client node does a zmq_connect[3] to any of these endpoints, the ser
In most cases, which node acts as client, and which as server, is about network topology rather than message flow. However, there //are// cases (resending when connections are broken) where the same socket type will behave differently if it's a server or if it's a client.
-What this means is that you should always think in terms of "servers" as stable parts of your topology, with more-or-less fixed endpoint addresses, and "clients" as dynamic parts that come and go. Then, design your application around this model. The chances that it will "just work" are much better like that.
+What this means is that you should always think in terms of "servers" as static parts of your topology, with more-or-less fixed endpoint addresses, and "clients" as dynamic parts that come and go. Then, design your application around this model. The chances that it will "just work" are much better like that.
Sockets have types. The socket type defines the semantics of the socket, its policies for routing messages inwards and outwards, queuing, etc. You can connect certain types of socket together, e.g. a publisher socket and a subscriber socket. Sockets work together in "messaging patterns". We'll look at this in more detail later.
-It's the ability to connect sockets in these different ways that gives 0MQ its basic power as a message queuing system. There are layers on top of this, such as devices and topic routing, which we'll get to later. But essentially, with 0MQ you define your network architecture by plugging pieces together like a child's construction toy.
+It's the ability to connect sockets in these different ways that gives 0MQ its basic power as a message queuing system. There are layers on top of this, such as proxies, which we'll get to later. But essentially, with 0MQ you define your network architecture by plugging pieces together like a child's construction toy.
+++ Using Sockets to Carry Data
-To send and receive messages you use the zmq_send[3] and zmq_recv[3] methods. The names are conventional but 0MQ's I/O model is different enough from the TCP model that you will need time to get your head around it.
+To send and receive messages you use the zmq_msg_send[3] and zmq_msg_recv[3] methods. The names are conventional but 0MQ's I/O model is different enough from the TCP model that you will need time to get your head around it.
[[code type="textdiagram" title="TCP sockets are 1 to 1"]]
+------------+
@@ -177,9 +177,9 @@ Fan out | |
+------------+ +------------+
[[/code]]
-So writing a message to a socket may send the message to one or many other places at once, and conversely, one socket will collect messages from all connections sending messages to it. The zmq_recv[3] method uses a fair-queuing algorithm so each sender gets an even chance.
+So writing a message to a socket may send the message to one or many other places at once, and conversely, one socket will collect messages from all connections sending messages to it. The zmq_msg_recv[3] method uses a fair-queuing algorithm so each sender gets an even chance.
-The zmq_send[3] method does not actually send the message to the socket connection(s). It queues the message so that the I/O thread can send it asynchronously. It does not block except in some exception cases. So the message is not necessarily sent when zmq_send[3] returns to your application. If you created a message using zmq_msg_init_data[3] you cannot reuse the data or free it, otherwise the I/O thread will rapidly find itself writing overwritten or unallocated garbage. This is a common mistake for beginners. We'll see a little later how to properly work with messages.
+The zmq_msg_send[3] method does not actually send the message to the socket connection(s). It queues the message so that the I/O thread can send it asynchronously. It does not block except in some exception cases. So the message is not necessarily sent when zmq_msg_send[3] returns to your application. If you created a message using zmq_msg_init_data[3] you cannot reuse the data or free it, otherwise the I/O thread will rapidly find itself writing overwritten or unallocated garbage. This is a common mistake for beginners. We'll see a little later how to properly work with messages.
+++ Unicast Transports
@@ -219,19 +219,33 @@ There is however a good answer to the question, "How can I make profitable use o
+++ I/O Threads
-We said that 0MQ does I/O in a background thread. One I/O thread (for all sockets) is sufficient for all but the most extreme applications. This is the magic '1' that we use when creating a context, meaning "use one I/O thread":
+We said that 0MQ does I/O in a background thread. One I/O thread (for all sockets) is sufficient for all but the most extreme applications. When you create a new context it starts with one I/O thread. The general rule of thumb is to allow one I/O thread per gigabyte of data in or out per second. To raise the number of I/O threads, use the zmq_ctx_set[3] call //before// creating any sockets:
[[code language="C"]]
-void *context = zmq_init (1);
+int io_threads = 4;
+void *context = zmq_ctx_new ();
+zmq_ctx_set (context, ZMQ_IO_THREADS, io_threads);
+assert (zmq_ctx_get (context, ZMQ_IO_THREADS) == io_threads);
[[/code]]
There is a major difference between a 0MQ application and a conventional networked application, which is that you don't create one socket per connection. One socket handles all incoming and outgoing connections for a particular point of work. E.g. when you publish to a thousand subscribers, it's via one socket. When you distribute work among twenty services, it's via one socket. When you collect data from a thousand web applications, it's via one socket.
This has a fundamental impact on how you write applications. A traditional networked application has one process or one thread per remote connection, and that process or thread handles one socket. 0MQ lets you collapse this entire structure into a single thread, and then break it up as necessary for scaling.
++++ Limiting Socket Use
+
+By default, a 0MQ socket will continue to accept connections until your operating system runs out of file handles. This isn't always the best policy for public-facing services as it leaves you open to a simple denial-of-service attack. You can set a limit using another zmq_ctx_set[3] call:
+
+[[code language="C"]]
+int max_sockets = 1024;
+void *context = zmq_ctx_new ();
+zmq_ctx_get (context, ZMQ_MAX_SOCKETS, max_sockets);
+assert (zmq_ctx_get (context, ZMQ_MAX_SOCKETS) == max_sockets);
+[[/code]]
+
+++ Core Messaging Patterns
-Underneath the brown paper wrapping of 0MQ's socket API lies the world of messaging patterns. If you have a background in enterprise messaging, these will be vaguely familiar. But to most 0MQ newcomers they are a surprise, we're so used to the TCP paradigm where a socket represents another node.
+Underneath the brown paper wrapping of 0MQ's socket API lies the world of messaging patterns. If you have a background in enterprise messaging, or know UDP well, these will be vaguely familiar. But to most 0MQ newcomers they are a surprise, we're so used to the TCP paradigm where a socket maps one-to-one to another node.
Let's recap briefly what 0MQ does for you. It delivers blobs of data (messages) to nodes, quickly and efficiently. You can map nodes to threads, processes, or boxes. It gives your applications a single socket API to work with, no matter what the actual transport (like in-process, inter-process, TCP, or multicast). It automatically reconnects to peers as they come and go. It queues messages at both sender and receiver, as needed. It manages these queues carefully to ensure processes don't run out of memory, overflowing to disk when appropriate. It handles socket errors. It does all I/O in background threads. It uses lock-free techniques for talking between nodes, so there are never locks, waits, semaphores, or deadlocks.
@@ -249,11 +263,9 @@ The built-in core 0MQ patterns are:
We looked at each of these in the first chapter. There's one more pattern that people tend to try to use when they still think of 0MQ in terms of traditional TCP sockets:
-* **Exclusive pair**, which connects two sockets in an exclusive pair. This is a low-level pattern for specific, advanced use-cases. We'll see an example at the end of this chapter.
+* **Exclusive pair**, which connects two sockets in an exclusive pair. This is a pattern you should use only to connect two threads in a process. We'll see an example at the end of this chapter.
-The zmq_socket[3] man page is fairly clear about the patterns, it's worth reading several times until it starts to make sense. We'll look at each pattern and the use-cases it covers.
-
-These are the socket combinations that are valid for a connect-bind pair (either side can bind):
+The zmq_socket[3] man page is fairly clear about the patterns, it's worth reading several times until it starts to make sense. These are the socket combinations that are valid for a connect-bind pair (either side can bind):
* PUB and SUB
* REQ and REP
@@ -265,27 +277,27 @@ These are the socket combinations that are valid for a connect-bind pair (either
* PUSH and PULL
* PAIR and PAIR
-Any other combination will produce undocumented and unreliable results and future versions of 0MQ will probably return errors if you try them. You can and will of course bridge other socket types //via code//, i.e. read from one socket type and write to another.
+You'll also see references to XPUB and XSUB sockets, which we'll come to later (they're like raw versions of PUB and SUB). Any other combination will produce undocumented and unreliable results and future versions of 0MQ will probably return errors if you try them. You can and will of course bridge other socket types //via code//, i.e. read from one socket type and write to another.
+++ High-level Messaging Patterns
-These four core patterns are cooked-in to 0MQ. They are part of the 0MQ API, implemented in the core C++ library, and guaranteed to be available in all fine retail stores. If one day the Linux kernel includes 0MQ, for example, these patterns would be there.
+These four core patterns are cooked-in to 0MQ. They are part of the 0MQ API, implemented in the core C++ library, and guaranteed to be available in all fine retail stores.
-On top, we add //high-level patterns//. We build these high-level patterns on top of 0MQ and implement them in whatever language we're using for our application. They are not part of the core library, do not come with the 0MQ package, and exist in their own space, as part of the 0MQ community.
+On top, we add //high-level patterns//. We build these high-level patterns on top of 0MQ and implement them in whatever language we're using for our application. They are not part of the core library, do not come with the 0MQ package, and exist in their own space, as part of the 0MQ community. For example the Majordomo pattern, which we explore in Chapter Four, sits in the github Majordomo project in the ZeroMQ organization.
One of the things we aim to provide you with this guide are a set of such high-level patterns, both small (how to handle messages sanely) to large (how to make a reliable publish-subscribe architecture).
+++ Working with Messages
-On the wire, 0MQ messages are blobs of any size from zero upwards, fitting in memory. You do your own serialization using Google Protocol Buffers, XDR, JSON, or whatever else your applications need to speak. It's wise to choose a data representation that is portable and fast, but you can make your own decisions about trade-offs.
+On the wire, 0MQ messages are blobs of any size from zero upwards, fitting in memory. You do your own serialization using protobufs, msgpack, JSON, or whatever else your applications need to speak. It's wise to choose a data representation that is portable and fast, but you can make your own decisions about trade-offs.
In memory, 0MQ messages are zmq_msg_t structures (or classes depending on your language). Here are the basic ground rules for using 0MQ messages in C:
* You create and pass around zmq_msg_t objects, not blocks of data.
-* To read a message you use zmq_msg_init[3] to create an empty message, and then you pass that to zmq_recv[3].
+* To read a message you use zmq_msg_init[3] to create an empty message, and then you pass that to zmq_msg_recv[3].
-* To write a message from new data, you use zmq_msg_init_size[3] to create a message and at the same time allocate a block of data of some size. You then fill that data using memcpy[3], and pass the message to zmq_send[3].
+* To write a message from new data, you use zmq_msg_init_size[3] to create a message and at the same time allocate a block of data of some size. You then fill that data using memcpy[3], and pass the message to zmq_msg_send[3].
* To release (not destroy) a message you call zmq_msg_close[3]. This drops a reference, and eventually 0MQ will destroy the message.
@@ -301,8 +313,9 @@ static char *
s_recv (void *socket) {
zmq_msg_t message;
zmq_msg_init (&message);
- zmq_recv (socket, &message, 0);
- int size = zmq_msg_size (&message);
+ int size = zmq_msg_recv (&message, socket, 0);
+ if (size == -1)
+ return NULL;
char *string = malloc (size + 1);
memcpy (string, zmq_msg_data (&message), size);
zmq_msg_close (&message);
@@ -313,32 +326,30 @@ s_recv (void *socket) {
// Convert C string to 0MQ string and send to socket
static int
s_send (void *socket, char *string) {
- int rc;
zmq_msg_t message;
zmq_msg_init_size (&message, strlen (string));
memcpy (zmq_msg_data (&message), string, strlen (string));
- rc = zmq_send (socket, &message, 0);
- assert (!rc);
+ int size = zmq_msg_send (&message, socket, 0);
zmq_msg_close (&message);
- return (rc);
+ return (size);
}
[[/code]]
You can easily extend this code to send and receive blobs of arbitrary length.
-**Note than when you have passed a message to zmq_send(3), ØMQ will clear the message, i.e. set the size to zero. You cannot send the same message twice, and you cannot access the message data after sending it.**
+**Note than when you have passed a message to zmq_msg_send(3), ØMQ will clear the message, i.e. set the size to zero. You cannot send the same message twice, and you cannot access the message data after sending it.**
If you want to send the same message more than once, create a second message, initialize it using zmq_msg_init[3] and then use zmq_msg_copy[3] to create a copy of the first message. This does not copy the data but the reference. You can then send the message twice (or more, if you create more copies) and the message will only be finally destroyed when the last copy is sent or closed.
-0MQ also supports //multipart// messages, which let you handle a list of blobs as a single message. This is widely used in real applications and we'll look at that later in this chapter and in Chapter Three.
+0MQ also supports //multi-part// messages, which let you send or receive a list of frames as a single on-the-wire message. This is widely used in real applications and we'll look at that later in this chapter and in Chapter Three.
Some other things that are worth knowing about messages:
-* 0MQ sends and receives them atomically, i.e. you get a whole message, or you don't get it at all.
+* 0MQ sends and receives them atomically, i.e. you get a whole message, or you don't get it at all. This is also true for multi-part messages.
* 0MQ does not send a message right away but at some indeterminate later time.
-* You can send zero-length messages, e.g. for sending a signal from one thread to another.
+* You may send zero-length messages, e.g. for sending a signal from one thread to another.
* A message must fit in memory. If you want to send files of arbitrary sizes, you should break them into pieces and send each piece as a separate message.
@@ -354,7 +365,7 @@ In all the examples so far, the main loop of most examples has been:
# process message
# repeat
-What if we want to read from multiple sockets at the same time? The simplest way is to connect one socket to multiple endpoints and get 0MQ to do the fanin for us. This is legal if the remote endpoints are in the same pattern but it would be illegal to e.g. connect a PULL socket to a PUB endpoint. Fun, but illegal. If you start mixing patterns you break future scalability.
+What if we want to read from multiple sockets at the same time? The simplest way is to connect one socket to multiple endpoints and get 0MQ to do the fan-in for us. This is legal if the remote endpoints are in the same pattern but it would be wrong to e.g. connect a PULL socket to a PUB endpoint.
The right way is to use zmq_poll[3]. An even better way might be to wrap zmq_poll[3] in a framework that turns it into a nice event-driven //reactor//, but it's significantly more work than we want to cover here.
@@ -374,41 +385,45 @@ Now let's see the same little senseless application done right, using zmq_poll[3
+++ Handling Errors and ETERM
-0MQ's error handling philosophy is a mix of fail-fast and resilience. Processes, we believe, should be as vulnerable as possible to internal errors, and as robust as possible against external attacks and errors. To give an analogy, a living cell will self-destruct if it detects a single internal error, yet it will resist attack from the outside by all means possible. Assertions, which pepper the 0MQ code, are absolutely vital to robust code, they just have to be on the right side of the cellular wall. And there should be such a wall. If it is unclear whether a fault is internal or external, that is a design flaw that needs to be fixed.
+0MQ's error handling philosophy is a mix of fail-fast and resilience. Processes, we believe, should be as vulnerable as possible to internal errors, and as robust as possible against external attacks and errors. To give an analogy, a living cell will self-destruct if it detects a single internal error, yet it will resist attack from the outside by all means possible.
-In C, assertions stop the application immediately with an error. In other languages you may get exceptions or halts.
+Assertions, which pepper the 0MQ code, are absolutely vital to robust code, they just have to be on the right side of the cellular wall. And there should be such a wall. If it is unclear whether a fault is internal or external, that is a design flaw to be fixed. In C/C++, assertions stop the application immediately with an error. In other languages you may get exceptions or halts.
-When 0MQ detects an external fault it returns an error to the calling code. In some rare cases it drops messages silently, if there is no obvious strategy for recovering from the error. In a few places 0MQ still asserts on external faults, but these are considered bugs.
+When 0MQ detects an external fault it returns an error to the calling code. In some rare cases it drops messages silently, if there is no obvious strategy for recovering from the error.
In most of the C examples we've seen so far there's been no error handling. **Real code should do error handling on every single 0MQ call**. If you're using a language binding other than C, the binding may handle errors for you. In C you do need to do this yourself. There are some simple rules, starting with POSIX conventions:
-* Methods that create objects will return NULL in case they fail.
+* Methods that create objects will return NULL if they fail.
+
+* Methods that process data may return the number of bytes processed, or -1 on an error or failure.
-* Other methods will return 0 on success and other values (mostly -1) on an exceptional condition (usually failure).
+* Other methods will return 0 on success and -1 on an error or failure.
* The error code is provided in {{errno}} or zmq_errno[3].
* A descriptive error text for logging is provided by zmq_strerror[3].
There are two main exceptional conditions that you may want to handle as non-fatal:
-* When a thread calls zmq_recv[3] with the NOBLOCK option and there is no waiting data. 0MQ will return -1 and set errno to EAGAIN.
+* When a thread calls zmq_msg_recv[3] with the ZMQ_DONTWAIT option and there is no waiting data. 0MQ will return -1 and set errno to EAGAIN.
-* When a thread calls zmq_term[3] and other threads are doing blocking work. The zmq_term[3] call closes the context and all blocking calls exit with -1, and errno set to ETERM.
+* When a thread calls zmq_ctx_destroy[3] and other threads are doing blocking work. The zmq_ctx_destroy[3] call closes the context and all blocking calls exit with -1, and errno set to ETERM.
What this boils down to is that in most cases you can use assertions on 0MQ calls, like this, in C:
[[code language="C"]]
-void *context = zmq_init (1);
+void *context = zmq_ctx_new ();
assert (context);
void *socket = zmq_socket (context, ZMQ_REP);
assert (socket);
-int rc;
-rc = zmq_bind (socket, "tcp://*:5555");
-assert (rc == 0);
+int rc = zmq_bind (socket, "tcp://*:5555");
+if (rc != 0) {
+ printf ("E: bind failed: %s\n", strerror (errno));
+ return -1;
+}
[[/code]]
-In the first version of this code I put the assert() call around the function. Not a good idea, since an optimized build will turn all assert() macros to null and happily wallop those functions. Use a return code, and assert the return code.
+In C/C++, asserts can be removed entirely in optimized code, so don't make the mistake of wrapping the whole 0MQ call in an assert(). It looks neat, then the optimizer removes all the asserts and the calls you want to make, and your application breaks in impressive ways.
Let's see how to shut down a process cleanly. We'll take the parallel pipeline example from the previous section. If we've started a whole lot of workers in the background, we now want to kill them when the batch is finished. Let's do this by sending a kill message to the workers. The best place to do this is the sink, since it really knows when the batch is done.
@@ -427,7 +442,7 @@ It doesn't take much new code in the sink:
...
// Send kill signal to workers
zmq_msg_init_data (&message, "KILL", 5);
- zmq_send (control, &message, 0);
+ zmq_msg_send (control, &message, 0);
zmq_msg_close (&message);
[[/code]]
@@ -502,7 +517,7 @@ Here is how we handle a signal in various languages:
The program provides s_catch_signals(), which traps Ctrl-C (SIGINT) and SIGTERM. When either of these signals arrive, the s_catch_signals() handler sets the global variable s_interrupted. Your application will not die automatically, you have to now explicitly check for an interrupt, and handle it properly. Here's how:
* Call s_catch_signals() (copy this from interrupt.c) at the start of your main code. This sets-up the signal handling.
-* If your code is blocking in zmq_recv[3], zmq_poll[3], or zmq_send[3], when a signal arrives, the call will return with EINTR.
+* If your code is blocking in zmq_msg_recv[3], zmq_poll[3], or zmq_msg_send[3], when a signal arrives, the call will return with EINTR.
* Wrappers like s_recv() return NULL if they are interrupted.
* So, your application checks for an EINTR return code, a NULL return, and/or s_interrupted.
@@ -546,7 +561,7 @@ Any long-running application has to manage memory correctly, or eventually it'll
}
[[/code]]
-* Fix your applications to exit cleanly after Ctrl-C. For any application that exits by itself, that's not needed, but for long-running applications (like devices), this is essential, otherwise valgrind will complain about all currently allocated memory.
+* Fix your applications to exit cleanly after Ctrl-C. For any application that exits by itself, that's not needed, but for long-running applications, this is essential, otherwise valgrind will complain about all currently allocated memory.
* Build your application with -DDEBUG, if it's not your default setting. That ensures valgrind can tell you exactly where memory is being leaked.
@@ -562,29 +577,31 @@ And after fixing any errors it reported, you should get the pleasant message:
==30536== ERROR SUMMARY: 0 errors from 0 contexts...
[[/code]]
-+++ Multipart Messages
++++ Multi-part Messages
+
+0MQ lets us compose a message out of several frames, giving us a 'multi-part message'. Realistic applications use multi-part messages heavily, both for wrapping messages with address information, and for simple serialization. We'll look at address envelopes later.
-0MQ lets us compose a message out of several frames, giving us a 'multipart message'. Realistic applications use multipart messages heavily, especially to make "envelopes". We'll look at them later. What we'll learn now is simply how to safely (but blindly) read and write multipart messages because otherwise the devices we write won't work with applications that use multipart messages.
+What we'll learn now is simply how to safely (but blindly) read and write multi-part messages in any application (like a proxy) that needs to forward messages without inspecting them.
-When you work with multipart messages, each part is a zmq_msg item. E.g. if you are sending a message with five parts, you must construct, send, and destroy five zmq_msg items. You can do this in advance (and store the zmq_msg items in an array or structure), or as you send them, one by one.
+When you work with multi-part messages, each part is a zmq_msg item. E.g. if you are sending a message with five parts, you must construct, send, and destroy five zmq_msg items. You can do this in advance (and store the zmq_msg items in an array or structure), or as you send them, one by one.
-Here is how we send the frames in a multipart message (we receive each frame into a message object):
+Here is how we send the frames in a multi-part message (we receive each frame into a message object):
[[code language="C"]]
-zmq_send (socket, &message, ZMQ_SNDMORE);
+zmq_msg_send (socket, &message, ZMQ_SNDMORE);
...
-zmq_send (socket, &message, ZMQ_SNDMORE);
+zmq_msg_send (socket, &message, ZMQ_SNDMORE);
...
-zmq_send (socket, &message, 0);
+zmq_msg_send (socket, &message, 0);
[[/code]]
-Here is how we receive and process all the parts in a message, be it single part or multipart:
+Here is how we receive and process all the parts in a message, be it single part or multi-part:
[[code language="C"]]
while (1) {
zmq_msg_t message;
zmq_msg_init (&message);
- zmq_recv (socket, &message, 0);
+ zmq_msg_recv (socket, &message, 0);
// Process the message frame
zmq_msg_close (&message);
int64_t more;
@@ -595,108 +612,31 @@ while (1) {
}
[[/code]]
-Some things to know about multipart messages:
-
-* When you send a multipart message, the first part (and all following parts) are only sent when you send the final part.
+Some things to know about multi-part messages:
+* When you send a multi-part message, the first part (and all following parts) are only actually sent on the wire when you send the final part.
* If you are using zmq_poll[3], when you receive the first part of a message, all the rest has also arrived.
-
* You will receive all parts of a message, or none at all.
-
* Each part of a message is a separate zmq_msg item.
-
* You will receive all parts of a message whether or not you check the RCVMORE option.
-
* On sending, 0MQ queues message frames in memory until the last is received, then sends them all.
-
* There is no way to cancel a partially sent message, except by closing the socket.
-+++ Intermediates and Devices
-
-Any connected set hits a complexity curve as the number of set members increases. A small number of members can all know about each other but as the set gets larger, the cost to each member of knowing all other interesting members grows linearly, and the overall cost of connecting members is factorial. The solution is to break sets into smaller ones, and use intermediates to connect the sets.
-
-This pattern is extremely common in the real world and is why our societies and economies are filled with intermediaries who have no other real function than to reduce the complexity and scaling costs of larger networks. Intermediaries are typically called wholesalers, distributors, managers, etc.
-
-A 0MQ network like any cannot grow beyond a certain size without needing intermediaries. In 0MQ, we call these "devices". When we use 0MQ we usually start building our applications as a set of nodes on a network with the nodes talking to each other, without intermediaries!figref().
-
-[[code type="textdiagram" title="Small-scale 0MQ Application"]]
- +---------+
- | |
- | Node |
- | |
- +---------+
- | Socket |
- \----+----/
- |
- |
- +------+------+
- | |
- | |
-/----+----\ /----+----\
-| Socket | | Socket |
-+---------+ +---------+
-| | | |
-| Node | | Node |
-| | | |
-+---------+ +---------+
-[[/code]]
++++ Intermediaries and Proxies
-And then we extend the application across a wider network, placing devices in specific places and scaling up the number of nodes!figref().
+0MQ aims for decentralized intelligence but that doesn't mean your network is empty space in the middle. It's filled with message-aware infrastructure and quite often, we build that infrastructure with 0MQ. The 0MQ plumbing can range from tiny pipes to full-blown service-oriented brokers. The messaging industry calls this "intermediation", meaning that the stuff in the middle deals with either side. In 0MQ we call these proxies, queues, forwarders, device, or brokers, depending on the context.
-[[code type="textdiagram" title="Larger-scale 0MQ Application"]]
- +---------+
- | |
- | Node |
- | |
- +---------+
- | Socket |
- \----+----/
- |
- |
- +-------------+-------------+
- | | |
- | | |
-/----+----\ /----+----\ /----+----\
-| Socket | | Socket | | Socket |
-+---------+ +---------+ +---------+
-| | | | | |
-| Node | | Node | | Device |
-| | | | | |
-+---------+ +---------+ +---------+
- | Socket |
- \----+----/
- |
- |
- +------+------+
- | |
- | |
- /----+----\ /----+----\
- | Socket | | Socket |
- +---------+ +---------+
- | | | |
- | Node | | Node |
- | | | |
- +---------+ +---------+
-[[/code]]
+This pattern is extremely common in the real world and is why our societies and economies are filled with intermediaries who have no other real function than to reduce the complexity and scaling costs of larger networks. Real-world intermediaries are typically called wholesalers, distributors, managers, etc.
-0MQ devices generally connect a set of 'frontend' sockets to a set of 'backend' sockets, though there are no strict design rules. They ideally run with no state, so that it becomes possible to stretch applications over as many intermediates as needed. You can run them as threads within a process, or as stand-alone processes. 0MQ provides some very basic devices but you will in practice develop your own.
+++++ The Dynamic Discovery Problem
-0MQ devices can do intermediation of addresses, services, queues, or any other abstraction you care to define above the message and socket layers. Different messaging patterns have different complexity issues and need different kinds of intermediation. For example, request-reply works well with queue and service abstractions, while publish-subscribe works well with streams or topics.
+One of the problems you will hit as you design larger distributed architectures is discovery. That is, how do pieces know about each other? It's especially difficult if pieces come and go, thus we can call this the "dynamic discovery problem".
-What's interesting about 0MQ as compared to traditional centralized brokers is that you can place devices precisely where you need them, and they can do the optimal intermediation.
+There are several solutions to dynamic discovery. The simplest is to entirely avoid it by hard-coding (or configuring) the network architecture so discovery is done by hand. That is, when you add a new piece, you reconfigure the network to know about it.
-++++ A Publish-Subscribe Proxy Server
+In practice this leads to increasingly fragile and hard-to-manage architectures. Let's say you have one publisher and a hundred subscribers. You connect each subscriber to the publisher by configuring a publisher endpoint in each subscriber. That's easy!figref(). Subscribers are dynamic, the publisher is static. Now say you add more publishers. Suddenly it's not so easy any more. If you continue to connect each subscriber to each publisher, the cost of avoiding dynamic discovery gets higher and higher.
-It is a common requirement to extend a publish-subscribe architecture over more than one network segment or transport. Perhaps there are a group of subscribers sitting at a remote location. Perhaps we want to publish to local subscribers via multicast, and to remote subscribers via TCP.
-
-We're going to write a simple proxy server that sits in between a publisher and a set of subscribers, bridging two networks. This is perhaps the simplest case of a useful device. The device has two sockets, a frontend facing the internal network, where the weather server is sitting, and a backend facing subscribers on the external network. It subscribes to the weather service on the frontend socket, and republishes its data on the backend socket:
-
-[[code type="example" title="Weather update proxy" name="wuproxy"]]
-[[/code]]
-
-We call this a //proxy// because it acts as a subscriber to publishers, and acts as a publisher to subscribers!figref(). That means you can slot this device into an existing network without affecting it (of course the new subscribers need to know to speak to the proxy).
-
-[[code type="textdiagram" title="Forwarder Proxy Device"]]
+[[code type="textdiagram" title="Small-scale Pub-Sub Network"]]
+-----------+
| |
| Publisher |
@@ -711,39 +651,91 @@ We call this a //proxy// because it acts as a subscriber to publishers, and acts
+----------------+----------------+
| | |
| | |
- connect connect |
-/------------\ /------------\ connect
-| SUB | | SUB | /------------\
-+------------+ +------------+ | SUB |
-| | | | +------------+
-| Subscriber | | Subscriber | | |
-| | | | | Forwarder |
-+------------+ +------------+ | |
- +------------+
- Internal network | PUB |
- ---------------------------------\------------/--------
- External network bind
- tcp://10.1.1.0:8100
- |
- |
- +--------+--------+
- | |
- | |
- connect connect
- /------------\ /------------\
- | SUB | | SUB |
- +------------+ +------------+
- | | | |
- | Subscriber | | Subscriber |
- | | | |
- +------------+ +------------+
+ connect connect connect
+/------------\ /------------\ /------------\
+| SUB | | SUB | | SUB |
++------------+ +------------+ +------------+
+| | | | | |
+| Subscriber | | Subscriber | | Subscriber |
+| | | | | |
++------------+ +------------+ +------------+
[[/code]]
-Note that this application is multipart safe. It correctly detects multipart messages and sends them as it reads them. If we did not set the SNDMORE option on outgoing multipart data, the final recipient would get a corrupted message. You should always make your devices multipart safe so that there is no risk they will corrupt the data they switch.
+There are quite a few answers to this but the very simplest answer is to add an intermediary, that is, a static point in the network to which all other nodes connect. In classic messaging, this is the job of the "message broker". 0MQ doesn't come with a message broker as such, but it lets us build intermediaries quite easily.
+
+You might wonder, if all networks eventually get large enough to need intermediaries, why don't we simply always design around a message broker? For beginners, it's a fair compromise. Just always use a star topology, forget about performance, and things will usually work. However message brokers are greedy things; in their role as central intermediaries, they become too complex, too stateful, and eventually a problem.
-++++ A Request-Reply Broker
+It's better to think of intermediaries as simple stateless message switches. The best analogy is an HTTP proxy; it's there but doesn't have any special role. Adding a pub-sub proxy solves the dynamic discovery problem in our example. We set the proxy in the "middle" of the network!figref(). The proxy opens an XSUB socket, an XPUB socket, and binds each to well-known IP addresses and ports. Then all other processes connect to the proxy, instead of to each other. It becomes trivial to add more subscribers or publishers.
+
+[[code type="textdiagram" title="Pub-Sub Network with a Proxy"]]
++------------+ +------------+ +------------+
+| | | | | |
+| Publisher | | Publisher | | Publisher |
+| | | | | |
++------------+ +------------+ +------------+
+| PUB | | PUB | | PUB |
+\------------/ \------------/ \------------/
+ connect connect connect
+ | | |
+ | | |
+ +----------------+----------------+
+ |
+ |
+ bind
+ /------------\
+ | XSUB |
+ +------------+
+ | |
+ | Proxy |
+ | |
+ +------------+
+ | XPUB |
+ \------------/
+ bind
+ |
+ |
+ +----------------+----------------+
+ | | |
+ | | |
+ connect connect connect
+/------------\ /------------\ /------------\
+| SUB | | SUB | | SUB |
++------------+ +------------+ +------------+
+| | | | | |
+| Subscriber | | Subscriber | | Subscriber |
+| | | | | |
++------------+ +------------+ +------------+
+[[/code]]
-Let's explore how to solve a problem of scale by writing a little message queuing broker in 0MQ. We'll look at the request-reply pattern for this case.
+We need XPUB and XSUB sockets because 0MQ does subscription forwarding: SUB sockets actually send subscriptions to PUB sockets as special messages. The proxy has to forward these as well, by reading them from the XPUB socket and writing them to the XSUB socket. This is the main use-case for XSUB and XPUB!figref().
+
+[[code type="textdiagram" title="Extended Publish-Subscribe"]]
++---------+ +---------+ +---------+
+| PUB | | PUB | | PUB |
+\----+----/ \----+----/ \----+----/
+ | | |
+ | | |
+ +-------------+-------------+
+ |
+ |
+ /-----+-----\
+ | XSUB |
+ +-----------+
+ | code |
+ +-----------+
+ | XPUB |
+ \-----+-----/
+ |
+ |
+ +-------------+-------------+
+ | | |
+ | | |
+/----+----\ /----+----\ /----+----\
+| SUB | | SUB | | SUB |
++---------+ +---------+ +---------+
+[[/code]]
+
+++++ The Shared Queue Problem
In the Hello World client-server application we have one client that talks to one service. However in real cases we usually need to allow multiple services as well as multiple clients. This lets us scale up the power of the service (many threads or processes or boxes rather than just one). The only constraint is that services must be stateless, all state being in the request or in some shared storage such as a database.
@@ -777,17 +769,15 @@ There are two ways to connect multiple clients to multiple servers. The brute-fo
This design lets you add more clients cheaply. You can also add more services. Each client will load-balance its requests to the services. But each client has to know the service topology. If you have 100 clients and then you decide to add three more services, you need to reconfigure and restart 100 clients in order for the clients to know about the three new services.
-That's clearly not the kind of thing we want to be doing at 3am when our supercomputing cluster has run out of resources and we desperately need to add a couple of hundred new service nodes. Too many stable pieces are like liquid concrete: knowledge is distributed and the more stable pieces you have, the more effort it is to change the topology. What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add and remove services or clients at any time without touching any other part of the topology.
+That's clearly not the kind of thing we want to be doing at 3am when our supercomputing cluster has run out of resources and we desperately need to add a couple of hundred new service nodes. Too many static pieces are like liquid concrete: knowledge is distributed and the more static pieces you have, the more effort it is to change the topology. What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add and remove services or clients at any time without touching any other part of the topology.
So we'll write a little message queuing broker that gives us this flexibility. The broker binds to two endpoints, a frontend for clients and a backend for services. It then uses zmq_poll[3] to monitor these two sockets for activity and when it has some, it shuttles messages between its two sockets. It doesn't actually manage any queues explicitly -- 0MQ does that automatically on each socket.
When you use REQ to talk to REP you get a strictly synchronous request-reply dialog. The client sends a request, the service reads the request and sends a reply. The client then reads the reply. If either the client or the service try to do anything else (e.g. sending two requests in a row without waiting for a response) they will get an error.
But our broker has to be non-blocking. Obviously we can use zmq_poll[3] to wait for activity on either socket, but we can't use REP and REQ.
-Luckily there are two sockets called DEALER and ROUTER that let you do non-blocking request-response. These sockets used to be called XREQ and XREP, and you may see these names in old code. The old names suggested that XREQ was an "extended REQ" and XREP was an "extended REP" but that's inaccurate. You'll see in Chapter Three how DEALER and ROUTER sockets let you build all kinds of asynchronous request-reply flows.
-
-Now, we're just going to see how DEALER and ROUTER let us extend REQ-REP across a device, that is, our little broker.
+Luckily there are two sockets called DEALER and ROUTER that let you do non-blocking request-response. You'll see in Chapter Three how DEALER and ROUTER sockets let you build all kinds of asynchronous request-reply flows. For now, we're just going to see how DEALER and ROUTER let us extend REQ-REP across an intermediary, that is, our little broker.
In this simple stretched request-reply pattern, REQ talks to ROUTER and DEALER talks to REP. In between the DEALER and ROUTER we have to have code (like our broker) that pulls messages off the one socket and shoves them onto the other!figref().
@@ -817,22 +807,22 @@ In this simple stretched request-reply pattern, REQ talks to ROUTER and DEALER t
+---------+ +---------+ +---------+
[[/code]]
-The request-reply broker binds to two endpoints, one for clients to connect to (the frontend socket) and one for services to connect to (the backend). To test this broker, you will want to change your services so they connect to the backend socket. Here are a client and service that show what I mean:
+The request-reply broker binds to two endpoints, one for clients to connect to (the frontend socket) and one for workers to connect to (the backend). To test this broker, you will want to change your workers so they connect to the backend socket. Here are a client and worker that show what I mean:
[[code type="example" title="Request-reply client" name="rrclient"]]
[[/code]]
-Here is the service:
+Here is the worker:
-[[code type="example" title="Request-reply service" name="rrserver"]]
+[[code type="example" title="Request-reply worker" name="rrworker"]]
[[/code]]
-And here is the broker. You will see that it's multipart safe:
+And here is the broker, which properly handles multi-part messages:
[[code type="example" title="Request-reply broker" name="rrbroker"]]
[[/code]]
-Using a request-reply broker makes your client-server architectures easier to scale since clients don't see services, and services don't see clients. The only stable node is the device in the middle!figref().
+Using a request-reply broker makes your client-server architectures easier to scale since clients don't see workers, and workers don't see clients. The only static node is the broker in the middle!figref().
[[code type="textdiagram" title="Request-reply Broker"]]
+---------+ +---------+ +---------+
@@ -882,46 +872,88 @@ Using a request-reply broker makes your client-server architectures easier to sc
+---------+ +---------+ +---------+
[[/code]]
-++++ Built-in Devices
+++++ 0MQ's Built-in Proxy Function
-0MQ provides some built-in devices, though most advanced users write their own devices. The built-in devices are:
-
-* QUEUE, which is like the request-reply broker.
-* FORWARDER, which is like the pub-sub proxy server.
-* STREAMER, which is like FORWARDER but for pipeline flows.
-
-To start a device, you call zmq_device[3] and pass it two sockets, one for the frontend and one for the backend:
+It turns out that that core loop in rrbroker is very useful, and reusable. It lets us build pub-sub forwarders and shared queues and other little intermediaries, with very little effort. 0MQ wraps this up in a single method, zmq_proxy[3]:
[[code language="C"]]
-zmq_device (ZMQ_QUEUE, frontend, backend);
+zmq_proxy (frontend, backend, capture);
[[/code]]
-Which if you start a QUEUE device is exactly like plugging the main body of the request-reply broker into your code at that spot. You need to create the sockets, bind or connect them, and possibly configure them, before calling zmq_device[3]. It is trivial to do. Here is the request-reply broker re-written to call QUEUE and rebadged as an expensive-sounding "message queue" (people have charged houses for code that did less):
+The two (or three sockets, if we want to capture data) must be properly connected, bound, configured. When we call the zmq_proxy method it's exactly like starting the main loop of rrbroker. Let's rewrite the request-reply broker to call zmq_proxy, and re-badge this as an expensive-sounding "message queue" (people have charged houses for code that did less):
[[code type="example" title="Message queue broker" name="msgqueue"]]
[[/code]]
-The built-in devices do proper error handling, whereas the examples we have shown don't. Since you can configure the sockets as you need to, before starting the device, it's worth using the built-in devices when you can.
+If you're like most 0MQ users, at this stage your mind is starting to think, "//what kind of evil stuff can I do if I plug random socket types into the proxy?//" The short answer is: try it and work out what is happening. In practice you would usually stick to ROUTER/DEALER, XSUB/XPUB, or PULL/PUSH.
+
+++++ The Transport Bridging Problem
+
+A frequent request from 0MQ users is "how do I connect my 0MQ network with technology X?" where X is some other networking or messaging technology. The simple answer is to build a "bridge". A bridge is a small application that speaks one protocol at one socket, and converts to/from a second protocol at another socket. A protocol interpreter, if you like. A common bridging problem in 0MQ is to bridge two transports or networks.
-If you're like most 0MQ users, at this stage your mind is starting to think, "//what kind of evil stuff can I do if I plug random socket types into devices?//" The short answer is: don't do it. You can mix socket types but the results are going to be weird. So stick to using ROUTER/DEALER for queue devices, SUB/PUB for forwarders and PULL/PUSH for streamers.
+As example, we're going to write a little proxy that sits in between a publisher and a set of subscribers, bridging two networks. The frontend socket (SUB) faces the internal network, where the weather server is sitting, and the backend (PUB) faces subscribers on the external network. It subscribes to the weather service on the frontend socket, and republishes its data on the backend socket!figref().
-When you start to need other combinations, it's time to write your own devices.
+[[code type="example" title="Weather update proxy" name="wuproxy"]]
+[[/code]]
+
+[[code type="textdiagram" title="Pub-Sub Forwarder Proxy"]]
+ +-----------+
+ | |
+ | Publisher |
+ | |
+ +-----------+
+ | PUB |
+ \-----------/
+ bind
+ tcp://192.168.55.210:5556
+ |
+ |
+ +----------------+----------------+
+ | | |
+ | | |
+ connect connect |
+/------------\ /------------\ connect
+| SUB | | SUB | /------------\
++------------+ +------------+ | XSUB |
+| | | | +------------+
+| Subscriber | | Subscriber | | |
+| | | | | Forwarder |
++------------+ +------------+ | |
+ +------------+
+ Internal network | XPUB |
+ ---------------------------------\------------/--------
+ External network bind
+ tcp://10.1.1.0:8100
+ |
+ |
+ +--------+--------+
+ | |
+ | |
+ connect connect
+ /------------\ /------------\
+ | SUB | | SUB |
+ +------------+ +------------+
+ | | | |
+ | Subscriber | | Subscriber |
+ | | | |
+ +------------+ +------------+
+[[/code]]
+++ Multithreading with 0MQ
0MQ is perhaps the nicest way ever to write multithreaded (MT) applications. Whereas as 0MQ sockets require some readjustment if you are used to traditional sockets, 0MQ multithreading will take everything you know about writing MT applications, throw it into a heap in the garden, pour gasoline over it, and set it alight. It's a rare book that deserves burning, but most books on concurrent programming do.
To make utterly perfect MT programs (and I mean that literally) **we don't need mutexes, locks, or any other form of inter-thread communication except messages sent across 0MQ sockets.**
-By "perfect" MT programs I mean code that's easy to write and understand, that works with one technology in any language and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns.
+By "perfect" MT programs I mean code that's easy to write and understand, that works with one design language in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns.
-If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing. If there's one lesson we've learned from 30+ years of concurrent programming it is: //just don't share state//. It's like two drunkards trying to share a beer. It doesn't matter if they're good buddies. Sooner or later they're going to get into a fight. And the more drunkards you add to the pavement, the more they fight each other over the beer. The tragic majority of MT applications look like drunken bar fights.
+If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing. If there's one lesson we've learned from 30+ years of concurrent programming it is: //just don't share state//. It's like two drunkards trying to share a beer. It doesn't matter if they're good buddies. Sooner or later they're going to get into a fight. And the more drunkards you add to the table, the more they fight each other over the beer. The tragic majority of MT applications look like drunken bar fights.
The list of weird problems that you need to fight as you write classic shared-state MT code would be hilarious if it didn't translate directly into stress and risk, as code that seems to work suddenly fails under pressure. Here is a list of "//11 Likely Problems In Your Multithreaded Code//" from a large firm with world-beating experience in buggy code: forgotten synchronization, incorrect granularity, read and write tearing, lock-free reordering, lock convoys, two-step dance, and priority inversion.
Yeah, we also counted seven, not eleven. That's not the point though. The point is, do you really want that code running the power grid or stock market to start getting two-step lock convoys at 3pm on a busy Thursday? Who cares what the terms actually mean. This is not what turned us on to programming, fighting ever more complex side-effects with ever more complex hacks.
-Some widely used metaphors, despite being the basis for billion-dollar industries, are fundamentally broken, and shared state concurrency is one of them. Code that wants to scale without limit does it like the Internet does, by sending messages and sharing nothing except a common contempt for broken programming metaphors.
+Some widely used models, despite being the basis for entire industries, are fundamentally broken, and shared state concurrency is one of them. Code that wants to scale without limit does it like the Internet does, by sending messages and sharing nothing except a common contempt for broken programming models.
You should follow some rules to write happy multithreaded code with 0MQ:
@@ -933,17 +965,17 @@ You should follow some rules to write happy multithreaded code with 0MQ:
* You MUST NOT share 0MQ sockets between threads. 0MQ sockets are not threadsafe. Technically it's possible to do this, but it demands semaphores, locks, or mutexes. This will make your application slow and fragile. The only place where it's remotely sane to share sockets between threads are in language bindings that need to do magic like garbage collection on sockets.
-If you need to start more than one device in an application, for example, you will want to run each in their own thread. It is easy to make the error of creating the device sockets in one thread, and then passing the sockets to the device in another thread. This may appear to work but will fail randomly. Remember: //Do not use or close sockets except in the thread that created them.//
+If you need to start more than one proxy in an application, for example, you will want to run each in their own thread. It is easy to make the error of creating the proxy frontend and backend sockets in one thread, and then passing the sockets to the proxy in another thread. This may appear to work but will fail randomly. Remember: //Do not use or close sockets except in the thread that created them.//
If you follow these rules, you can quite easily split threads into separate processes, when you need to. Application logic can sit in threads, processes, boxes: whatever your scale needs.
0MQ uses native OS threads rather than virtual "green" threads. The advantage is that you don't need to learn any new threading API, and that 0MQ threads map cleanly to your operating system. You can use standard tools like Intel's ThreadChecker to see what your application is doing. The disadvantages are that your code, when it for instance starts new threads, won't be portable, and that if you have a huge number of threads (thousands), some operating systems will get stressed.
Let's see how this works in practice. We'll turn our old Hello World server into something more capable. The original server was a single thread. If the work per request is low, that's fine: one ØMQ thread can run at full speed on a CPU core, with no waits, doing an awful lot of work. But realistic servers have to do non-trivial work per request. A single core may not be enough when 10,000 clients hit the server all at once. So a realistic server must start multiple worker threads. It then accepts requests as fast as it can, and distributes these to its worker threads. The worker threads grind through the work, and eventually send their replies back.
-You can of course do all this using a queue device and external worker processes, but often it's easier to start one process that gobbles up sixteen cores, than sixteen processes, each gobbling up one core. Further, running workers as threads will cut out a network hop, latency, and network traffic.
+You can of course do all this using a proxy broker and external worker processes, but often it's easier to start one process that gobbles up sixteen cores, than sixteen processes, each gobbling up one core. Further, running workers as threads will cut out a network hop, latency, and network traffic.
-The MT version of the Hello World service basically collapses the queue device and workers into a single process:
+The MT version of the Hello World service basically collapses the broker and workers into a single process:
[[code type="example" title="Multithreaded service" name="mtserver"]]
[[/code]]
@@ -956,7 +988,7 @@ All the code should be recognizable to you by now. How it works:
* The server creates a DEALER socket to talk to the workers and binds this to its internal interface (over {{inproc}}).
-* The server starts a QUEUE device that connects the two sockets. The QUEUE device keeps a single queue for incoming requests, and distributes those out to workers. It also routes replies back to their origin.
+* The server starts a proxy that connects the two sockets. The proxy pulls incoming requests fairly from all clients, and distributes those out to workers. It also routes replies back to their origin.
Note that creating threads is not portable in most programming languages. The POSIX library is {{pthreads}}, but on Windows you have to use a different API. We'll see in Chapter Three how to wrap this in a portable API.
@@ -985,7 +1017,7 @@ Here the 'work' is just a one-second pause. We could do anything in the workers,
| +------------+ |
| | | |
| | Queue | |
-| | device | |
+| | proxy | |
| | | |
| +------------+ |
| | DEALER | |
@@ -1063,15 +1095,15 @@ This is the first time we've shown an example using PAIR sockets. Why use PAIR?
* You can use PUSH for the sender and PULL for the receiver. This looks simple and will work, but remember that PUSH will load-balance messages to all available receivers. If you by accident start two receivers (e.g. you already have one running and you start a second), you'll "lose" half of your signals. PAIR has the advantage of refusing more than one connection, the pair is //exclusive//.
-* You can use DEALER for the sender and ROUTER for the receiver. ROUTER however wraps your message in an "envelope", meaning your zero-size signal turns into a multipart message. If you don't care about the data, and treat anything as a valid signal, and if you don't read more than once from the socket, that won't matter. If however you decide to send real data, you will suddenly find ROUTER providing you with "wrong" messages. DEALER also load-balances, giving the same risk as PUSH.
+* You can use DEALER for the sender and ROUTER for the receiver. ROUTER however wraps your message in an "envelope", meaning your zero-size signal turns into a multi-part message. If you don't care about the data, and treat anything as a valid signal, and if you don't read more than once from the socket, that won't matter. If however you decide to send real data, you will suddenly find ROUTER providing you with "wrong" messages. DEALER also load-balances, giving the same risk as PUSH.
* You can use PUB for the sender and SUB for the receiver. This will correctly deliver your messages exactly as you sent them and PUB does not load-balance as PUSH or DEALER do. However you need to configure the subscriber with an empty subscription, which is annoying. Worse, the reliability of the PUB-SUB link is timing dependent and messages can get lost if the SUB socket is connecting while the PUB socket is sending its messages.
For these reasons, PAIR makes the best choice for coordination between pairs of threads.
+++ Node Coordination
-When you want to coordinate nodes, PAIR sockets won't work well any more. This is one of the few areas where the strategies for threads and nodes are different. Principally nodes come and go whereas threads are stable. PAIR sockets do not automatically reconnect if the remote node goes away and comes back.
+When you want to coordinate nodes, PAIR sockets won't work well any more. This is one of the few areas where the strategies for threads and nodes are different. Principally nodes come and go whereas threads are static. PAIR sockets do not automatically reconnect if the remote node goes away and comes back.
The second significant difference between threads and nodes is that you typically have a fixed number of threads but a more variable number of nodes. Let's take one of our earlier scenarios (the weather server and clients) and use node coordination to ensure that subscribers don't lose data when starting up.
@@ -1155,11 +1187,11 @@ A more robust model could be:
+++ Zero Copy
-We teased you in Chapter One, when you were still a 0MQ newbie, about zero-copy. If you survived this far, you are probably ready to use zero-copy. However, remember that there are many roads to Hell, and premature optimization is not the most enjoyable nor profitable one, by far. In English, trying to do zero-copy properly while your architecture is not perfect is a waste of time and will make things worse, not better.
+We teased you in Chapter One, when you were still a 0MQ newbie, about zero-copy. If you survived this far, you are probably ready to use zero-copy. However, remember that there are many roads to Hell, and premature optimization is not the most enjoyable nor profitable one, by far. To say this in English, trying to do zero-copy properly while your architecture is not perfect is a waste of time and will make things worse, not better.
0MQ's message API lets you can send and receive messages directly from and to application buffers without copying data. Given that 0MQ sends messages in the background, zero-copy needs some extra sauce.
-To do zero-copy you use zmq_msg_init_data[3] to create a message that refers to a block of data already allocated on the heap with malloc(), and then you pass that to zmq_send[3]. When you create the message you also pass a function that 0MQ will call to free the block of data, when it has finished sending the message. This is the simplest example, assuming 'buffer' is a block of 1000 bytes allocated on the heap:
+To do zero-copy you use zmq_msg_init_data[3] to create a message that refers to a block of data already allocated on the heap with malloc(), and then you pass that to zmq_msg_send[3]. When you create the message you also pass a function that 0MQ will call to free the block of data, when it has finished sending the message. This is the simplest example, assuming 'buffer' is a block of 1000 bytes allocated on the heap:
[[code language="C"]]
void my_free (void *data, void *hint) {
@@ -1168,24 +1200,24 @@ void my_free (void *data, void *hint) {
// Send message from buffer, which we allocate and 0MQ will free for us
zmq_msg_t message;
zmq_msg_init_data (&message, buffer, 1000, my_free, NULL);
-zmq_send (socket, &message, 0);
+zmq_msg_send (socket, &message, 0);
[[/code]]
There is no way to do zero-copy on receive: 0MQ delivers you a buffer that you can store as long as you wish but it will not write data directly into application buffers.
-On writing, 0MQ's multipart messages work nicely together with zero-copy. In traditional messaging you need to marshal different buffers together into one buffer that you can send. That means copying data. With 0MQ, you can send multiple buffers coming from different sources as individual message frames. We send each field as a length-delimited frame. To the application it looks like a series of send and recv calls. But internally the multiple parts get written to the network and read back with single system calls, so it's very efficient.
+On writing, 0MQ's multi-part messages work nicely together with zero-copy. In traditional messaging you need to marshal different buffers together into one buffer that you can send. That means copying data. With 0MQ, you can send multiple buffers coming from different sources as individual message frames. We send each field as a length-delimited frame. To the application it looks like a series of send and recv calls. But internally the multiple parts get written to the network and read back with single system calls, so it's very efficient.
-+++ Pub-sub Message Envelopes
++++ Pub-Sub Message Envelopes
-We've looked briefly at multipart messages. Let's now look at their main use-case, which is //message envelopes//. An envelope is a way of safely packaging up data with an address, without touching the data itself.
+We've looked briefly at multi-part messages. Let's now look at their main use-case, which is //message envelopes//. An envelope is a way of safely packaging up data with an address, without touching the data itself.
In the pub-sub pattern, the envelope at least holds the subscription key for filtering but you can also add the sender identity in the envelope.
If you want to use pub-sub envelopes, you make them yourself. It's optional, and in previous pub-sub examples we didn't do this. Using a pub-sub envelope is a little more work for simple cases but it's cleaner especially for real cases, where the key and the data are naturally separate things. It's also faster, if you are writing the data directly from an application buffer.
Here is what a publish-subscribe message with an envelope looks like:
-[[code type="textdiagram" title="Pub-sub Envelope with Separate Key"]]
+[[code type="textdiagram" title="Pub-Sub Envelope with Separate Key"]]
+-------------+
Frame 1 | Key | Subscription key
+-------------+
@@ -1198,12 +1230,12 @@ Recall that pub-sub matches messages based on the prefix. Putting the key into a
Here is a minimalist example of how pub-sub envelopes look in code. This publisher sends messages of two types, A and B. The envelope holds the message type:
-[[code type="example" title="Pub-sub envelope publisher" name="psenvpub"]]
+[[code type="example" title="Pub-Sub envelope publisher" name="psenvpub"]]
[[/code]]
The subscriber only wants messages of type B:
-[[code type="example" title="Pub-sub envelope subscriber" name="psenvsub"]]
+[[code type="example" title="Pub-Sub envelope subscriber" name="psenvsub"]]
[[/code]]
When you run the two programs, the subscriber should show you this:
@@ -1216,11 +1248,11 @@ When you run the two programs, the subscriber should show you this:
...
[[/code]]
-This examples shows that the subscription filter rejects or accepts the entire multipart message (key plus data). You won't get part of a multipart message, ever.
+This examples shows that the subscription filter rejects or accepts the entire multi-part message (key plus data). You won't get part of a multi-part message, ever.
If you subscribe to multiple publishers and you want to know their identity so that you can send them data via another socket (and this is a fairly typical use-case), you create a three-part message:
-[[code type="textdiagram" title="Pub-sub Envelope with Sender Address"]]
+[[code type="textdiagram" title="Pub-Sub Envelope with Sender Address"]]
+-------------+
Frame 1 | Key | Subscription key
+-------------+
@@ -1244,7 +1276,7 @@ The answer for messaging is to set limits on the size of buffers, and then when
0MQ uses the concept of "high water mark" or HWM to define the capacity of its internal pipes. Each connection out of a socket or into a socket has its own pipe, and HWM capacity.
-In 0MQ/2.x the HWM was set to infinite by default. In 0MQ/3.x it's set to 1,000 by default, which is more sensible. If you're using 0MQ/2.x you should always set a HWM on your sockets, be it 1,000 to match 0MQ/3.x or another figure that takes into account your message sizes.
+In 0MQ/2.x the HWM was set to infinite by default. In 0MQ/3.x it's set to 1,000 by default, which is more sensible. If you're still using 0MQ/2.x you should always set a HWM on your sockets, be it 1,000 to match 0MQ/3.x or another figure that takes into account your message sizes.
The high water mark affects both the transmit and receive buffers of a single socket. Some sockets (PUB, PUSH) only have transmit buffers. Some (SUB, PULL, REQ, REP) only have receive buffers. Some (DEALER, ROUTER, PAIR) have both transmit and receive buffers.
View
242 chapter3.txt
@@ -21,17 +21,17 @@ We'll cover:
In the request-reply pattern, the envelope holds the return address for replies. It is how a 0MQ network with no state can create round-trip request-reply dialogs.
-You don't in fact need to understand how request-reply envelopes work to use them for common cases. When you use REQ and REP, your sockets build and use envelopes automatically. When you write a device, and we covered this in the last chapter, you just need to read and write all the parts of a message. 0MQ implements envelopes using multipart data, so if you copy multipart data safely, you implicitly copy envelopes too.
+You don't in fact need to understand how request-reply envelopes work to use them for common cases. When you use REQ and REP, your sockets build and use envelopes automatically. When you write a device, and we covered this in the last chapter, you just need to read and write all the parts of a message. 0MQ implements envelopes using multi-part data, so if you copy multi-part data safely, you implicitly copy envelopes too.
-However, getting under the hood and playing with request-reply envelopes is necessary for advanced request-reply work. It's time to explain how ROUTER works, in terms of envelopes:
+However, getting under the hood and playing with request-reply envelopes is necessary for advanced request-reply work. It's time to explain how the ROUTER socket works, in terms of envelopes:
-* When you receive a message from a ROUTER socket, it shoves a brown paper envelope around the message and scribbles on with indelible ink, "This came from Lucy". Then it gives that to you. That is, the ROUTER socket gives you what came off the wire, wrapped up in an envelope with the reply address on it.
+* When you receive a message from a ROUTER socket, it shoves a brown paper envelope around the message and scribbles on with indelible ink, "This came from Lucy". Then it gives that to you. That is, the ROUTER gives you what came off the wire, wrapped up in an envelope with the reply address on it.
-* When you send a message to a ROUTER socket, it rips off that brown paper envelope, tries to read its own handwriting, and if it knows who "Lucy" is, sends the contents back to Lucy. That is the reverse process of receiving a message.
+* When you send a message to a ROUTER, it rips off that brown paper envelope, tries to read its own handwriting, and if it knows who "Lucy" is, sends the contents back to Lucy. That is the reverse process of receiving a message.
-If you leave the brown envelope alone, and then pass that message to another ROUTER socket (e.g. by sending to a DEALER connected to a ROUTER), the second ROUTER socket will in turn stick another brown envelope on it, and scribble the name of that DEALER on it.
+If you leave the brown envelope alone, and then pass that message to another ROUTER (e.g. by sending to a DEALER connected to a ROUTER), the second ROUTER will in turn stick another brown envelope on it, and scribble the name of that DEALER on it.
-The whole point of this is that each ROUTER knows how to send replies back to the right place. All you need to do, in your application, is respect the brown envelopes. Now the REP socket makes sense. It carefully slices open the brown envelopes, one by one, keeps them safely aside, and gives you (the application code that owns the REP socket) the original message. When you send the reply, it re-wraps the reply in the brown paper envelopes, so it can hand the resulting brown package back to the ROUTER sockets down the chain.
+The whole point of this is that each ROUTER knows how to send replies back to the right place. All you need to do, in your application, is respect the brown envelopes. Now the REP socket makes sense. It carefully slices open the brown envelopes, one by one, keeps them safely aside, and gives you (the application code that owns the REP socket) the original message. When you send the reply, it re-wraps the reply in the brown paper envelopes, so it can hand the resulting brown package back to the ROUTERs down the chain.
Which lets you insert ROUTER-DEALER devices into a request-reply pattern like this:
@@ -42,7 +42,7 @@ Which lets you insert ROUTER-DEALER devices into a request-reply pattern like th
...etc.
[[/code]]
-If you connect a REQ socket to a ROUTER socket, and send one request message, you will get a message that consists of three frames: a reply address, an empty message frame, and the 'real' message!figref().
+If you connect a REQ socket to a ROUTER, and send one request message, you will get a message that consists of three frames: a reply address, an empty message frame, and the 'real' message!figref().
[[code type="textdiagram" title="Single-hop Request-reply Envelope"]]
+---------------+
@@ -58,7 +58,7 @@ Breaking this down:
* The data in frame 3 is what the sending application sends to the REQ socket.
-* The empty message frame in frame 2 is prepended by the REQ socket when it sends the message to the ROUTER socket.
+* The empty message frame in frame 2 is prepended by the REQ socket when it sends the message to the ROUTER.
* The reply address in frame 1 is prepended by the ROUTER before it passes the message to the receiving application.
@@ -82,7 +82,7 @@ Frame 5 | Data |
Here now is a more detailed explanation of the four socket types we use for request-reply patterns:
-* DEALER just load-balances (deals out) the messages you send to all connected peers, and fair-queues (deals in) the messages it receives. It is exactly like a PUSH and PULL socket combined.
+* DEALER just deals out the messages you send to all connected peers (aka "round-robin"), and deals in (aka "fair queuing") the messages it receives. It is exactly like a PUSH and PULL socket combined.
* REQ prepends an empty message frame to every message you send, and removes the empty message frame from each message you receive. It then works like DEALER (and in fact is built on DEALER) except it also imposes a strict send / receive cycle.
@@ -92,7 +92,7 @@ Here now is a more detailed explanation of the four socket types we use for requ
REP requires that the envelopes end with an empty message frame. If you're not using REQ at the other end of the chain then you must add the empty message frame yourself.
-So the obvious question about ROUTER is, where does it get the reply addresses from? And the obvious answer is, it uses the socket's identity. As we already learned, if a socket does not set an identity, the ROUTER socket generates an identity that it can associate with the connection to that socket!figref().
+So the obvious question about ROUTER is, where does it get the reply addresses from? And the obvious answer is, it uses the socket's identity. As we already learned, if a socket does not set an identity, the ROUTER generates an identity that it can associate with the connection to that socket!figref().
[[code type="textdiagram" title="ROUTER Invents a UUID"]]
| Client |
@@ -114,7 +114,7 @@ So the obvious question about ROUTER is, where does it get the reply addresses f
+-----------+ +---------+
[[/code]]
-When we set our own identity on a socket, this gets passed to the ROUTER socket, which passes it to the application as part of the envelope for each message that comes in!figref().
+When we set our own identity on a socket, this gets passed to the ROUTER, which passes it to the application as part of the envelope for each message that comes in!figref().
[[code type="textdiagram" title="ROUTER uses Identity If It knows It"]]
+-----------+
@@ -136,7 +136,7 @@ When we set our own identity on a socket, this gets passed to the ROUTER socket,
+-----------+ +---------+
[[/code]]
-Let's observe the above two cases in practice. This program dumps the contents of the message frames that a ROUTER socket receives from two REP sockets, one not using identities, and one using an identity 'Hello':
+Let's observe the above two cases in practice. This program dumps the contents of the message frames that a ROUTER receives from two REP sockets, one not using identities, and one using an identity 'Hello':
[[code type="example" title="Identity check" name="identity"]]
[[/code]]
@@ -151,65 +151,48 @@ Here is what the dump function prints:
----------------------------------------
[005] Hello
[000]
-[038] ROUTER socket uses REQ's socket identity
+[038] ROUTER uses REQ's socket identity
[[/code]]
+++ Custom Request-Reply Routing
We already saw that ROUTER uses the message envelope to decide which client to route a reply back to. Now let me express that in another way: //ROUTER will route messages asynchronously to any peer connected to it, if you provide the correct routing address via a properly constructed envelope.//
-So ROUTER is really a fully controllable router. We'll dig into this magic in detail.
+So ROUTER is really a fully controllable ROUTER. We'll dig into this magic in detail.
-But first, and because we're going to go off-road into some rough and possibly illegal terrain now, let's look closer at REQ and REP. Few people know this, but despite their kindergarten approach to messaging, REQ and REP are actually colorful characters:
+But first, and because we're going to go off-road into some rough and possibly illegal terrain now, let's look closer at REQ and REP. These provide your kindergarten request-reply socket pattern. It's an easy pattern to learn but quite rapidly gets annoying as it provides, for instance, no way to resend a request if it got lost for some reason.
-* REQ is a **mama** socket, doesn't listen but always expects an answer. Mamas are strictly synchronous and if you use them they are always the 'request' end of a chain.
-* REP is a **papa** socket, always answers, but never starts a conversation. Papas are strictly synchronous and if you use them, they are always the 'reply' end of a chain.
-
-The thing about Mama sockets is, as we all learned as kids, you can't speak until spoken to. Mamas do not have simple open-mindedness of papas, nor the ambiguous "sure, whatever" shrugged-shoulder aloofness of a dealer. So to speak to a mama socket, you have to get the mama socket to talk to you first. The good part is mamas don't care if you reply now, or much later. Just bring a good sob story and a bag of laundry.
-
-Papa sockets on the other hand are strong and silent, and pedantic. They do just one thing, which is to give you an answer to whatever you ask, perfectly framed and precise. Don't expect a papa socket to be chatty, or to pass a message on to someone else, this is just not going to happen.
-
-While we usually think of request-reply as a to-and-fro pattern, in fact it can be fully asynchronous, as long as we understand that any mamas or papas will be at the end of a chain, never in the middle of it, and always synchronous. All we need to know is the address of the peer we want to talk to, and then we can then send it messages asynchronously, via a router. The router is the one and only 0MQ socket type capable of being told "send this message to X" where X is the address of a connected peer.
+While we usually think of request-reply as a to-and-fro pattern, in fact it can be fully asynchronous, as long as we understand that any REQs and REPS will be at the end of a chain, never in the middle of it, and always synchronous. All we need to know is the address of the peer we want to talk to, and then we can then send it messages asynchronously, via a ROUTER. The ROUTER is the one and only 0MQ socket type capable of being told "send this message to X" where X is the address of a connected peer.
These are the ways we can know the address to send a message to, and you'll see most of these used in the examples of custom request-reply routing:
-* By default, a peer has a null identity and the router will generate a UUID and use that to refer to the connection when it delivers you each incoming message from that peer.
+* By default, a peer has a null identity and the ROUTER will generate a UUID and use that to refer to the connection when it delivers you each incoming message from that peer.
-* If the peer socket set an identity, the router will give that identity when it delivers an incoming request envelope from that peer.
+* If the peer socket set an identity, the ROUTER will give that identity when it delivers an incoming request envelope from that peer.
* Peers with explicit identities can send them via some other mechanism, e.g. via some other sockets.
* Peers can have prior knowledge of each others' identities, e.g. via configuration files or some other magic.
-There are at least three routing patterns, one for each of the socket types we can easily connect to a router:
+There are at least three routing patterns, one for each of the socket types we can easily connect to a ROUTER:
-* Router-to-dealer.
-* Router-to-mama (REQ).
-* Router-to-papa (REP).
+* ROUTER-to-DEALER.
+* ROUTER-to-REQ.
+* ROUTER-to-REP.
In each of these cases we have total control over how we route messages, but the different patterns cover different use-cases and message flows. Let's break it down over the next sections with examples of different routing algorithms.
-But first some warnings about custom routing:
-
-* This goes against a fairly solid 0MQ rule: //delegate peer addressing to the socket//. The only reason we do it is because 0MQ lacks a wide range of routing algorithms.
-
-* Future versions of 0MQ will probably do some of the routing we're going to build here. That means the code we design now may break, or become redundant in the future.
-
-* While the built-in routing has certain guarantees of scalability, such as being friendly to devices, custom routing doesn't. You will need to make your own devices.
++++ ROUTER-to-DEALER Routing
-So overall, custom routing is more expensive and more fragile than delegating this to 0MQ. Only do it if you need it. Having said that, let's jump in, the water's great!
+The ROUTER-to-DEALER pattern is the simplest. You connect one ROUTER to many DEALERs, and then distribute messages to the DEALERs using any algorithm you like. The DEALERs can be sinks (process the messages without any response), proxies (send the messages on to other nodes), or services (send back replies).
-+++ Router-to-Dealer Routing
+If you expect the DEALER to reply, there should only be one ROUTER talking to it. DEALERs have no idea how to reply to a specific peer, so if they have multiple peers, they will just round-robin between them, which would be weird. If the DEALER is a sink, any number of ROUTERs can talk to it.
-The router-to-dealer pattern is the simplest. You connect one router to many dealers, and then distribute messages to the dealers using any algorithm you like. The dealers can be sinks (process the messages without any response), proxies (send the messages on to other nodes), or services (send back replies).
+What kind of routing can you do with a ROUTER-to-DEALER pattern? If the DEALERs talk back to the ROUTER, e.g. telling the ROUTER when they finished a task, you can use that knowledge to route depending on how fast a DEALER is. Since both ROUTER and DEALER are asynchronous, it can get a little tricky. You'd need to use zmq_poll[3] at least.
-If you expect the dealer to reply, there should only be one router talking to it. Dealers have no idea how to reply to a specific peer, so if they have multiple peers, they will load-balance between them, which would be weird. If the dealer is a sink, any number of routers can talk to it.
+We'll make an example where the DEALERs don't talk back, they're pure sinks. Our routing algorithm will be a weighted random scatter: we have two DEALERs and we send twice as many messages to one as to the other!figref().
-What kind of routing can you do with a router-to-dealer pattern? If the dealers talk back to the router, e.g. telling the router when they finished a task, you can use that knowledge to route depending on how fast a dealer is. Since both router and dealer are asynchronous, it can get a little tricky. You'd need to use zmq_poll[3] at least.
-
-We'll make an example where the dealers don't talk back, they're pure sinks. Our routing algorithm will be a weighted random scatter: we have two dealers and we send twice as many messages to one as to the other!figref().
-
-[[code type="textdiagram" title="Router-to-Dealer Custom Routing"]]
+[[code type="textdiagram" title="ROUTER-to-DEALER Custom Routing"]]
+-------------+
| |
| Client | Send to "A" or "B"
@@ -235,46 +218,46 @@ We'll make an example where the dealers don't talk back, they're pure sinks. Our
Here's code that shows how this works:
-[[code type="example" title="Router-to-dealer" name="rtdealer"]]
+[[code type="example" title="ROUTER-to-DEALER" name="rtdealer"]]
[[/code]]
Some comments on this code:
-* The router doesn't know when the dealers are ready, and it would be distracting for our example to add in the signaling to do that. So the router just does a "sleep (1)" after starting the dealer threads. Without this sleep, the router will send out messages that can't be routed, and 0MQ will discard them.
+* The ROUTER doesn't know when the DEALERs are ready, and it would be distracting for our example to add in the signaling to do that. So the ROUTER just does a "sleep (1)" after starting the DEALER threads. Without this sleep, the ROUTER will send out messages that can't be routed, and 0MQ will discard them.
-* Note that this behavior is specific to ROUTER sockets. PUB sockets will also discard messages if there are no subscribers, but all other socket types will queue sent messages until there's a peer to receive them.
+* Note that this behavior is specific to ROUTERs. PUB sockets will also discard messages if there are no subscribers, but all other socket types will queue sent messages until there's a peer to receive them.
-To route to a dealer, we create an envelope consisting of just an identity frame (we don't need a null separator)!figref().
+To route to a DEALER, we create an envelope consisting of just an identity frame (we don't need a null separator)!figref().
-[[code type="textdiagram" title="Routing Envelope for Dealer"]]
+[[code type="textdiagram" title="Routing Envelope for DEALER"]]
+-------------+
Frame 1 | Address |
+-------------+-------------------------+
Frame 2 | Data |
+---------------------------------------+
[[/code]]
-The router socket removes the first frame, and sends the second frame, which the dealer gets as-is. When the dealer sends a message to the router, it sends one frame. The router prepends the dealer's address and gives us back a similar envelope in two parts.
+The ROUTER socket removes the first frame, and sends the second frame, which the DEALER gets as-is. When the DEALER sends a message to the ROUTER, it sends one frame. The ROUTER prepends the DEALER's address and gives us back a similar envelope in two parts.
-Something to note: if you use an invalid address, the router discards the message silently. There is not much else it can do usefully. In normal cases this either means the peer has gone away, or that there is a programming error somewhere and you're using a bogus address. In any case you cannot ever assume a message will be routed successfully until and unless you get a reply of some sort from the destination node. We'll come to creating reliable patterns later on.
+Something to note: if you use an invalid address, the ROUTER discards the message silently. There is not much else it can do usefully. In normal cases this either means the peer has gone away, or that there is a programming error somewhere and you're using a bogus address. In any case you cannot ever assume a message will be routed successfully until and unless you get a reply of some sort from the destination node. We'll come to creating reliable patterns later on.
-Dealers in fact work exactly like PUSH and PULL combined. It's however illegal and pointless to connect PULL or PUSH to a request-reply socket.
+DEALERs in fact work exactly like PUSH and PULL combined. Do not however connect PUSH or PULL sockets to DEALERS. That would just be nasty and pointless.
+++ Least-Recently Used Routing (LRU Pattern)
-Like we said, mamas (REQ sockets, if you really insist on it) don't listen to you, and if you try to speak out of turn they'll ignore you. You have to wait for them to say something, //then// you can give a sarcastic answer. This is very useful for routing because it means we can keep a bunch of mamas waiting for answers. In effect, mamas tell us when they're ready.
+REQ sockets don't listen to you, and if you try to speak out of turn they'll ignore you. You have to wait for them to say something, and //then// you can give a sarcastic answer. This is very useful for routing because it means we can keep a bunch of REQs waiting for answers. In effect, a REQ socket will tell us when it's ready.
-You can connect one router to many mamas, and distribute messages as you would to dealers. Mamas will usually want to reply, but they will let you have the last word. However it's one thing at a time:
+You can connect one ROUTER to many REQs, and distribute messages as you would to DEALERs. REQs will usually want to reply, but they will let you have the last word. However it's one thing at a time:
-* Mama speaks to router
-* Router replies to mama
-* Mama speaks to router
-* Router replies to mama
+* REQ speaks to ROUTER
+* ROUTER replies to REQ
+* REQ speaks to ROUTER
+* ROUTER replies to REQ
* etc.
-Like dealers, mamas can only talk to one router and since mamas always start by talking to the router, you should never connect one mama to more than one router unless you are doing sneaky stuff like multi-pathway redundant routing!figref(). I'm not even going to explain that now, and hopefully the jargon is complex enough to stop you trying this until you need it.
+Like DEALERs, REQs can only talk to one ROUTER and since REQs always start by talking to the ROUTER, you should never connect one REQ to more than one ROUTER unless you are doing sneaky stuff like multi-pathway redundant routing!figref(). I'm not even going to explain that now, and hopefully the jargon is complex enough to stop you trying this until you need it.
-[[code type="textdiagram" title="Router to Mama Custom Routing"]]
+[[code type="textdiagram" title="ROUTER to REQ Custom Routing"]]
+-------------+
| |
| Client | Send to "A" or "B"
@@ -283,11 +266,11 @@ Like dealers, mamas can only talk to one router and since mamas always start by
| ROUTER |
\-------------/
^
- | (1) Mama says Hi
+ | (1) REQ says Hi
|
+-------+-------+
| |
- | | (2) Router gives laundry
+ | | (2) ROUTER gives laundry
v v
/-----------\ /-----------\
| REQ | | REQ |
@@ -299,14 +282,14 @@ Like dealers, mamas can only talk to one router and since mamas always start by
+-----------+ +-----------+
[[/code]]
-What kind of routing can you do with a router-to-mama pattern? Probably the most obvious is "least-recently-used" (LRU), where we always route to the mama that's been waiting longest. Here is an example that does LRU routing to a set of mamas:
+What kind of routing can you do with a ROUTER-to-REQ pattern? Probably the most obvious is "least-recently-used" (LRU), where we always route to the REQ that's been waiting longest. Here is an example that does LRU routing to a set of REQs:
-[[code type="example" title="Router-to-mama" name="rtmama"]]
+[[code type="example" title="ROUTER-to-REQ" name="rtmama"]]
[[/code]]
For this example the LRU doesn't need any particular data structures above what 0MQ gives us (message queues) because we don't need to synchronize the workers with anything. A more realistic LRU algorithm would have to collect workers as they become ready, into a queue, and the use this queue when routing client requests. We'll do this in a later example.
-To prove that the LRU is working as expected, the mamas print the total tasks they each did. Since the mamas do random work, and we're not load balancing, we expect each mama to do approximately the same amount but with random variation. And that is indeed what we see:
+To prove that the LRU is working as expected, the REQs print the total tasks they each did. Since the REQs do random work, and we're not load balancing, we expect each REQ to do approximately the same amount but with random variation. And that is indeed what we see:
[[code]]
Processed: 8 tasks
@@ -323,15 +306,15 @@ Processed: 10 tasks
Some comments on this code
-* We don't need any settle time, since the mamas explicitly tell the router when they are ready.
+* We don't need any settle time, since the REQs explicitly tell the ROUTER when they are ready.
-* We're generating our own identities here, as printable strings, using the zhelpers.h s_set_id function. That's just to make our life a little simpler. In a realistic application the mamas would be fully anonymous and then you'd call zmq_recv[3] and zmq_send[3] directly instead of the zhelpers s_recv() and s_send() functions, which can only handle strings.
+* We're generating our own identities here, as printable strings, using the zhelpers.h s_set_id function. That's just to make our life a little simpler. In a realistic application the REQs would be fully anonymous and then you'd call zmq_msg_recv[3] and zmq_msg_send[3] directly instead of the zhelpers s_recv() and s_send() functions, which can only handle strings.
* If you copy and paste example code without understanding it, you deserve what you get. It's like watching Spiderman leap off the roof and then trying that yourself.
-To route to a mama, we must create a mama-friendly envelope consisting of an address plus an empty message frame!figref().
+To route to a REQ, we must create a REQ-friendly envelope consisting of an address plus an empty message frame!figref().
-[[code type="textdiagram" title="Routing Envelope for Mama (REQ)"]]
+[[code type="textdiagram" title="Routing Envelope for REQ"]]
+-------------+
Frame 1 | Address |
+---+---------+
@@ -343,22 +326,18 @@ Frame 3 | Data |
+++ Address-based Routing
-Papas are, if we care about them at all, only there to answer questions. And to pay the bills, fix the car when mama drives it into the garage wall, put up shelves, and walk the dog when it's raining. But apart from that, papas are only there to answer questions.
-
-In a classic request-reply pattern a router wouldn't talk to a papa socket at all, but rather would get a dealer to do the job for it. That's what dealers are for: to pass questions onto random papas and come back with their answers. Routers are generally more comfortable talking to mamas. OK, dear reader, you may stop the psychoanalysis. These are analogies, not life stories.
+In a classic request-reply pattern a ROUTER wouldn't talk to a REP socket at all, but rather would get a DEALER to do the job for it. It's worth remembering with 0MQ that the classic patterns are the ones that work best, that the beaten path is there for a reason, and that when we go off-road we take the risk of falling off cliffs and getting eaten by zombies. Having said that, let's plug a ROUTER into a REP and see what the heck emerges.
-It's worth remembering with 0MQ that the classic patterns are the ones that work best, that the beaten path is there for a reason, and that when we go off-road we take the risk of falling off cliffs and getting eaten by zombies. Having said that, let's plug a router into a papa and see what the heck emerges.
-
-The special thing about papas, all joking aside, is actually two things:
+The special thing about REPs is actually two things:
* One, they are strictly lockstep request-reply.
* Two, they accept an envelope stack of any size and will return that intact.
-In the normal request-reply pattern, papas are anonymous and replaceable (wow, these analogies //are// scary), but we're learning about custom routing. So, in our use-case we have reason to send a request to papa A rather than papa B. This is essential if you want to keep some kind of a conversation going between you, at one end of a large network, and a papa sitting somewhere far away.
+In the normal request-reply pattern, REPs are anonymous and replaceable, but we're learning about custom routing. So, in our use-case we have reason to send a request to REP A rather than REP B. This is essential if you want to keep some kind of a conversation going between you, at one end of a large network, and a REP sitting somewhere far away.
-A core philosophy of 0MQ is that the edges are smart and many, and the middle is vast and dumb. This does mean the edges can address each other, and this also means we want to know how to reach a given papa. Doing routing across multiple hops is something we'll look at later but for now we'll look just at the final step: a router talking to a specific papa!figref().
+A core philosophy of 0MQ is that the edges are smart and many, and the middle is vast and dumb. This does mean the edges can address each other, and this also means we want to know how to reach a given REP. Doing routing across multiple hops is something we'll look at later but for now we'll look just at the final step: a ROUTER talking to a specific REP!figref().
-[[code type="textdiagram" title="Router-to-Papa Custom Routing"]]
+[[code type="textdiagram" title="ROUTER-to-REP Custom Routing"]]
+-------------+
| |
| Client | Send to "A" or "B"
@@ -385,18 +364,18 @@ A core philosophy of 0MQ is that the edges are smart and many, and the middle is
This example shows a very specific chain of events:
-* The client has a message that it expects to route back (via another router) to some node. The message has two addresses (a stack), an empty part, and a body.
-* The client passes that to the router but specifies a papa address first.
-* The router removes the papa address, uses that to decide which papa to send the message to.
-* The papa receives the addresses, empty part, and body.
+* The client has a message that it expects to route back (via another ROUTER) to some node. The message has two addresses (a stack), an empty part, and a body.
+* The client passes that to the ROUTER but specifies a REP address first.
+* The ROUTER removes the REP address, uses that to decide which REP to send the message to.
+* The REP receives the addresses, empty part, and body.
* It removes the addresses, saves them, and passes the body to the worker.
-* The worker sends a reply back to the papa.
-* The papa recreates the envelope stack and sends that back with the worker's reply to the router.
-* The router prepends the papa's address and provides that to the client along with the rest of the address stack, empty part, and the body.
+* The worker sends a reply back to the REP.
+* The REP recreates the envelope stack and sends that back with the worker's reply to the ROUTER.
+* The ROUTER prepends the REP's address and provides that to the client along with the rest of the address stack, empty part, and the body.
-It's complex but worth working through until you understand it. Just remember a papa is garbage in, garbage out.
+It's complex but worth working through until you understand it. Just remember a REP is garbage in, garbage out.
-[[code type="example" title="Router-to-papa" name="rtpapa"]]
+[[code type="example" title="ROUTER-to-REP" name="rtpapa"]]
[[/code]]
Run this program and it should show you this:
@@ -415,17 +394,17 @@ Run this program and it should show you this:
Some comments on this code:
-* In reality we'd have the papa and router in separate nodes. This example does it all in one thread because it makes the sequence of events really clear.
+* In reality we'd have the REP and ROUTER in separate nodes. This example does it all in one thread because it makes the sequence of events really clear.
-* zmq_connect[3] doesn't happen instantly. When the papa socket connects to the router, that takes a certain time and happens in the background. In a realistic application the router wouldn't even know the papa existed until there had been some previous dialog. In our toy example we'll just {{sleep (1);}} to make sure the connection's done. If you remove the sleep, the papa socket won't get the message. (Try it.)
+* zmq_connect[3] doesn't happen instantly. When the REP socket connects to the ROUTER, that takes a certain time and happens in the background. In a realistic application the ROUTER wouldn't even know the REP existed until there had been some previous dialog. In our toy example we'll just {{sleep (1);}} to make sure the connection's done. If you remove the sleep, the REP socket won't get the message. (Try it.)
-* We're routing using the papa's identity. Just to convince yourself this really is happening, try sending to a wrong address, like "B". The papa won't get the message.
+* We're routing using the REP's identity. Just to convince yourself this really is happening, try sending to a wrong address, like "B". The REP won't get the message.
* The s_dump and other utility functions (in the C code) come from the zhelpers.h header file. It becomes clear that we do the same work over and over on sockets, and there are interesting layers we can build on top of the 0MQ API. We'll come back to this later when we make a real application rather than these toy examples.
-To route to a papa, we must create a papa-friendly envelope!figref().
+To route to a REP, we must create a REP-friendly envelope!figref().
-[[code type="textdiagram" title="Routing Envelope for Papa aka REP"]]
+[[code type="textdiagram" title="Routing Envelope for REP"]]
+-------------+
Frame 1 | Address | <--- Zero or more of these
+---+---------+
@@ -437,7 +416,7 @@ Frame 3 | Data |
+++ A Request-Reply Message Broker
-We'll recap the knowledge we have so far about doing weird stuff with 0MQ message envelopes, and build the core of a generic custom routing queue device that we can properly call a //message broker//. Sorry for all the buzzwords. What we'll make is a //queue device// that connects a bunch of //clients// to a bunch of //workers//, and lets you use //any routing algorithm// you want. What we'll do is //least-recently used//, since it's the most obvious use-case apart from load-balancing.
+I'll recap the knowledge we have so far about doing weird stuff with 0MQ message envelopes, and build the core of a generic custom routing queue device that we can properly call a //message broker//. Sorry for all the buzzwords. What we'll make is a //queue device// that connects a bunch of //clients// to a bunch of //workers//, and lets you use //any routing algorithm// you want. The algorith we'll implement is //least-recently used//, since it's the most obvious use-case after simple round-robin distribution.
To start with, let's look back at the classic request-reply pattern and then see how it extends over a larger and larger service-oriented network. The basic pattern just has one client talking to a few workers!figref().
@@ -459,7 +438,7 @@ To start with, let's look back at the classic request-reply pattern and then see
+--------+ +--------+ +--------+
[[/code]]
-This extends to multiple papas, but if we want to handle multiple mamas as well we need a device in the middle, which normally consists of a router and a dealer back to back, connected by a classic ZMQ_QUEUE device that just copies message frames between the two sockets as fast as it can!figref().
+This extends to multiple workers, but if we want to handle multiple clients as well, we need a device in the middle. We'd use a simple ZMQ_QUEUE device connecting a ROUTER and a DEALER back to back. This device just switches message frames between the two sockets as fast as it can!figref().
[[code type="textdiagram" title="Stretched Request-reply"]]
+--------+ +--------+ +--------+
@@ -487,9 +466,9 @@ This extends to multiple papas, but if we want to handle multiple mamas as well
+--------+ +--------+ +--------+
[[/code]]
-The key here is that the router stores the originating mama address in the request envelope, the dealer and papas don't touch that, and so the router knows which mama to send the reply back to. Papas are anonymous and not addressed in this pattern, all papas are assumed to provide the same service.
+The key here is that the ROUTER stores the originating client address in the request envelope, the DEALER and workers don't touch that, and so the ROUTER knows which client to send the reply back to. This pattern assumes all workers provide the exact same service.
-In the above design, we're using the built-in load balancing routing that the dealer socket provides. However we want for our broker to use a least-recently used algorithm, so we take the router-mama pattern we learned, and apply that!figref().
+In the above design, we're using the built-in round-robin routing that DEALER provides. However this means some workers may be idle while others have multiple requests waiting. For better efficiency and proper load-balancing we want to use a least-recently used algorithm, so we take the ROUTER-REQ pattern we learned, and apply that!figref().
[[code type="textdiagram" title="Stretched Request-reply with LRU"]]
+--------+ +--------+ +--------+
@@ -517,24 +496,24 @@ In the above design, we're using the built-in load balancing routing that the de
+--------+ +--------+ +--------+
[[/code]]
-Our broker - a router-to-router LRU queue - can't simply copy message frames blindly. Here is the code, it's fairly complex but the core logic is reusable in any request-reply broker that wants to do LRU routing:
+Our broker - a ROUTER-to-ROUTER LRU queue - can't simply copy message frames blindly. Here is the code, it's a fair chunk of code, but we can reuse the core logic any time we want to do load-balancing:
[[code type="example" title="LRU queue broker" name="lruqueue"]]
[[/code]]
The difficult part of this program is (a) the envelopes that each socket reads and writes, and (b) the LRU algorithm. We'll take these in turn, starting with the message envelope formats.
-First, recall that a mama REQ socket always puts on an empty part (the envelope delimiter) on sending and removes this empty part on reception. The reason for this isn't important, it's just part of the 'normal' request-reply pattern. What we care about here is just keeping mama happy by doing precisely what she needs. Second, the router always adds an envelope with the address of whomever the message came from.
+First, recall that a REQ REQ socket always puts on an empty part (the envelope delimiter) on sending and removes this empty part on reception. The reason for this isn't important, it's just part of the 'normal' request-reply pattern. What we care about here is just keeping REQ happy by doing precisely what she needs. Second, the ROUTER always adds an envelope with the address of whomever the message came from.
-We can now walk through a full request-reply chain from client to worker and back. In the code we set the identity of client and worker sockets to make it easier to print the message frames if we want to. Let's assume the client's identity is "CLIENT" and the worker's identity is "WORKER". The client sends a single frame with the message!figref().
+We can now walk through a full request-reply chain from client to worker and back. In this code we set the identity of client and worker sockets to make it easier to trace the message frames. Most normal applications do not use identities. Let's assume the client's identity is "CLIENT" and the worker's identity is "WORKER". The client sends a single frame with the message!figref().
[[code type="textdiagram" title="Message that Client Sends"]]
+---+-------+
Frame 1 | 5 | HELLO | Data frame
+---+-------+
[[/code]]
-What the queue gets, when reading off the router frontend socket, are three frames consisting of the sender address, empty frame delimiter, and the data part!figref().
+What the queue gets, when reading off the ROUTER frontend socket, are three frames consisting of the sender address, empty frame delimiter, and the data part!figref().
[[code type="textdiagram" title="Message Coming in on Frontend"]]
+---+--------+
@@ -546,7 +525,7 @@ Frame 3 | 5 | HELLO | Data frame
+---+-------+
[[/code]]
-The broker sends this to the worker, prefixed by the address of the worker, taken from the LRU queue, plus an additional empty part to keep the mama at the other end happy!figref().
+The broker sends this to the worker, prefixed by the address of the worker, taken from the LRU queue, plus an additional empty part to keep the REQ at the other end happy!figref().
[[code type="textdiagram" title="Message Sent to Backend"]]
+---+--------+
@@ -562,7 +541,7 @@ Frame 5 | 5 | HELLO | Data frame
+---+-------+
[[/code]]
-This complex envelope stack gets chewed up first by the backend router socket, which removes the first frame. Then the mama socket in the worker removes the empty part, and provides the rest to the worker!figref().
+This complex envelope stack gets chewed up first by the backend ROUTER socket, which removes the first frame. Then the REQ socket in the worker removes the empty part, and provides the rest to the worker application!figref().
[[code type="textdiagram" title="Message Delivered to Worker"]]
+---+--------+
@@ -574,11 +553,11 @@ Frame 3 | 5 | HELLO | Data frame
+---+-------+
[[/code]]
-Which is exactly the same as what the queue received on its frontend router socket. The worker has to save the envelope (which is all the parts up to and including the empty message frame) and then it can do what's needed with the data part.
+Which is exactly the same as what the queue received on its frontend ROUTER socket. The worker has to save the envelope (which is all the parts up to and including the empty message frame) and then it can do what's needed with the data part.
On the return path the messages are the same as when they come in, i.e. the backend socket gives the queue a message in five parts, and the queue sends the frontend socket a message in three parts, and the client gets a message in one part.
-Now let's look at the LRU algorithm. It requires that both clients and workers use mama sockets, and that workers correctly store and replay the envelope on messages they get. The algorithm is:
+Now let's look at the LRU algorithm. It requires that both clients and workers use REQ sockets, and that workers correctly store and replay the envelope on messages they get. The algorithm is:
* Create a pollset which polls the backend always, and the frontend only if there are one or more workers available.
@@ -592,7 +571,7 @@ You should now see that you can reuse and extend the LRU algorithm with variatio
+++ A High-Level API for 0MQ
-Reading and writing multipart messages using the native 0MQ API is like eating a bowl of hot noodle soup, with fried chicken and extra vegetables, using a toothpick. Look at the core of the worker thread from our LRU queue broker:
+Reading and writing multi-part messages using the native 0MQ API is, to be polite, a lot of work. Look at the core of the worker thread from our LRU queue broker:
[[code language="C"]]
while (1) {
@@ -623,16 +602,16 @@ while (1) {
// In this example there is only 1 but it could be more
zmq_msg_t address;
zmq_msg_init (&address);
- zmq_recv (worker, &address, 0);
+ zmq_msg_recv (worker, &address, 0);
zmq_msg_t empty;
zmq_msg_init (&empty);
- zmq_recv (worker, &empty, 0);
+ zmq_msg_recv (worker, &empty, 0);
// Get request, send reply
zmq_msg_t payload;
zmq_msg_init (&payload);
- zmq_recv (worker, &payload, 0);
+ zmq_msg_recv (worker, &payload, 0);
int char_nbr;
printf ("Worker: ");
@@ -643,18 +622,18 @@ while (1) {
zmq_msg_init_size (&payload, 2);
memcpy (zmq_msg_data (&payload), "OK", 2);
- zmq_send (worker, &address, ZMQ_SNDMORE);
+ zmq_msg_send (worker, &address, ZMQ_SNDMORE);
zmq_close (&address);
- zmq_send (worker, &empty, ZMQ_SNDMORE);
+ zmq_msg_send (worker, &empty, ZMQ_SNDMORE);
zmq_close (&empty);
- zmq_send (worker, &payload, 0);
+ zmq_msg_send (worker, &payload, 0);
zmq_close (&payload);
}
[[/code]]
What we want is an API that lets us receive and send an entire message in one shot, including all envelopes. One that lets us do what we want with the absolute least lines of code. The 0MQ core API itself doesn't aim to do this, but nothing prevents us making layers on top, and part of learning to use 0MQ intelligently is to do exactly that.
-Making a good message API is fairly difficult, especially if we want to avoid copying data around too much. We have a problem of terminology: 0MQ uses "message" to describe both multipart messages, and individual parts of a message. We have a problem of semantics: sometimes it's natural to see message content as printable string data, sometimes as binary blobs.
+Making a good message API is fairly difficult, especially if we want to avoid copying data around too much. We have a problem of terminology: 0MQ uses "message" to describe both multi-part messages, and individual parts of a message. We have a problem of semantics: sometimes it's natural to see message content as printable string data, sometimes as binary blobs.
So one solution is to use three concepts: //string// (already the basis for s_send and s_recv), //frame// (a message frame), and //message// (a list of one or more frames). Here is the worker code, rewritten onto an API using these concepts:
@@ -679,7 +658,7 @@ Replacing 22 lines of code with four is a good deal, especially since the result
* //Proper handling of Ctrl-C.// We already saw how to catch an interrupt. It would be useful if this happened in all applications.
-Turning this wishlist into reality gives us [http://zero.mq/c CZMQ], a high-level C API for 0MQ. This high-level binding in fact developed out of earlier versions of the Guide. It combines nicer semantics for working with 0MQ with some portability layers, and (importantly for C but less for other languages) containers like hashes and lists.
+Turning this wishlist into reality gives us [http://zero.mq/c CZMQ], a high-level C API for 0MQ. This high-level binding in fact developed out of earlier versions of the Guide. It combines nicer semantics for working with 0MQ with some portability layers, and (importantly for C but less for other languages) containers like hashes and lists. CZMQ also uses an elegant object model that leads to frankly lovely code.
Here is the LRU queue broker rewritten to use CZMQ:
@@ -713,6 +692,7 @@ The previous example still uses zmq_poll[3]. So how about reactors? The CZMQ {{z
* Set a reader on any socket, i.e. code that is called whenever the socket has input.
* Cancel a reader on a socket.
* Set a timer that goes off once or multiple times at specific intervals.
+* Cancel a timer.
{{zloop}} of course uses zmq_poll[3] internally. It rebuilds its poll set each time you add or remove readers, and it calculates the poll timeout to match the next timer. Then, it calls the reader and timer handlers for each socket and timer that needs attention.
@@ -736,7 +716,7 @@ Getting applications to properly shut-down when you send them Ctrl-C can be tric
+++ Asynchronous Client-Server
-In the router-to-dealer example we saw a 1-to-N use case where one client talks asynchronously to multiple workers. We can turn this upside-down to get a very useful N-to-1 architecture where various clients talk to a single server, and do this asynchronously!figref().