forked from zeromq/rfc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
spec_7.txt
244 lines (153 loc) · 14.9 KB
/
spec_7.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
The Majordomo Protocol (MDP) defines a reliable service-oriented request-reply dialog between a set of client applications, a broker and a set of worker applications. MDP covers presence, heartbeating, and service-oriented request-reply processing. It originated from the Majordomo pattern defined in Chapter 4 of the Guide.
* Name: rfc.zeromq.org/spec:7/MDP
* Editor: Pieter Hintjens <ph@imatix.com>
* Contributors: Chuck Remes <cremes@mac.com>, Guido Goldstein <zmqdev@a-nugget.de>
* State: legacy
++ License
Copyright (c) 2011 iMatix Corporation.
This Specification is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.
This Specification is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, see <http://www.gnu.org/licenses>.
++ Change Process
This Specification is a free and open standard[((bibcite fandos))] and is governed by the Digital Standards Organization's Consensus-Oriented Specification System (COSS)[((bibcite coss))].
++ Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119[((bibcite rfc2119))].
++ Goals
The Majordomo Protocol (MDP) defines a reliable service-oriented request-reply dialog between a set of client applications, a broker and a set of worker applications. MDP covers presence, heartbeating, and service-oriented request-reply processing. It originated from the Majordomo pattern defined in Chapter 4 of the Guide[((bibcite guide))]. The Majordomo pattern has no relation to the open source mailing list software with the same name.
MDP is an evolution of [http://rfc.zeromq.org/spec:6 6/PPP], adding name-based service resolution and more structured protocol commands.
The goals of MDP are to:
* Allow requests to be routed to workers on the basis of abstract service names.
* Allow both peers to detect disconnection of the other peer, through the use of heartbeating.
* Allow the broker to implement a "least recently used" pattern for task distribution to workers for a given service.
* Allow the broker to recover from dead or disconnected workers by resending requests to other workers.
++ Architecture
[[=image figure7-1.png]]
+++ Overall Topology
MDP connects a set of client applications, a single broker device and a pool of workers applications. Clients connect to the broker, as do workers. Clients and workers do not see each other, and both can come and go arbitrarily. The broker MAY open two sockets (ports), one front-end for clients, and one back-end for workers. However MDP is also designed to work over a single broker socket.
We define 'client' applications as those issuing requests, and 'worker' applications as those processing them. MDP makes these assumptions:
* Workers are idempotent, i.e. it is safe to execute the same request more than once.
* Workers will handle at most one request a time, and will issue exactly one reply for each successful request.
The Majordomo broker handles a set of shared request queues, one per service. Each queue has multiple writers (clients) and multiple readers (workers). The broker SHOULD serve clients on a fair basis and MAY deliver requests to workers on any basis, including round robin and least-recently used.
MDP consists of two sub-protocols:
* MDP/Client, which covers how the MDP broker communicates with client applications.
* MDP/Worker, which covers how the MDP broker communicates with workers applications.
The broker may be an intermediary (a device) or it may be a client application that implements MDP/Worker directly. Similarly, the broker may integrate services directly rather than using MDP/Worker.
+++ ROUTER Addressing
The broker MUST use a ROUTER socket to accept requests from clients, and connections from workers. The broker MAY use a separate socket for each sub-protocol, or MAY use a single socket for both sub-protocols.
From the ØMQ Reference Manual[((bibcite zmqsocket))]:
> When receiving messages a ROUTER socket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. When sending messages a ROUTER socket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to.
This extra frame is not shown in the sub-protocol commands explained below.
+++ MDP/Client
MDP/Client is a strictly synchronous dialog initiated by the client (where 'C' represents the client, and 'B' represents the broker):
[[code]]
Repeat:
C: REQUEST
B: REPLY
...
[[/code]]
A **REQUEST** command consists of a multipart message of 4 or more frames, formatted on the wire as follows:
* Frame 0: Empty (zero bytes, invisible to REQ application)
* Frame 1: "MDPC01" (six bytes, representing MDP/Client v0.1)
* Frame 2: Service name (printable string)
* Frames 3+: Request body (opaque binary)
A **REPLY** command consists of a multipart message of 4 or more frames, formatted on the wire as follows:
* Frame 0: Empty (zero bytes, invisible to REQ application)
* Frame 1: "MDPC01" (six bytes, representing MDP/Client v0.1)
* Frame 2: Service name (printable string)
* Frames 3+: Reply body (opaque binary)
Clients SHOULD use a REQ socket when implementing a synchronous request-reply pattern. The REQ socket will silently create frame 0 for outgoing requests, and remove it for replies before passing them to the calling application. Clients MAY use a DEALER (XREQ) socket when implementing an asynchronous pattern. In that case the clients MUST create the empty frame 0 explicitly.
Clients MAY use any suitable strategy for recovering from a non-responsive broker. One recommended strategy is:
* To use polling instead of blocking receives on the request socket.
* If there is no reply within some timeout, to close the request socket and open a new socket, and resend the request on that new socket.
* If there is no reply after several retries, to signal the transaction as failed.
The service name is a 0MQ string that matches the service name specified by a worker in its READY command (see MDP/Worker below). The broker SHOULD queue client requests for which no service has been registered and SHOULD expire these requests after a reasonable and configurable time if no service has been registered.
+++ MDP/Worker
MDP/Worker is a mix of a synchronous request-reply dialog, initiated by the service worker, and an asynchronous heartbeat dialog that operates independently in both directions. This is the synchronous dialog (where 'S' represents the service worker, and 'B' represents the broker):
[[code]]
S: READY
Repeat:
B: REQUEST
S: REPLY
...
[[/code]]
The asynchronous heartbeat dialog operates on the same sockets and works thus:
[[code]]
Repeat: Repeat:
S: HEARTBEAT B: HEARTBEAT
... ...
S: DISCONNECT B: DISCONNECT
[[/code]]
A **READY** command consists of a multipart message of 4 frames, formatted on the wire as follows:
* Frame 0: Empty frame
* Frame 1: "MDPW01" (six bytes, representing MDP/Worker v0.1)
* Frame 2: 0x01 (one byte, representing READY)
* Frame 3: Service name (printable string)
A **REQUEST** command consists of a multipart message of 6 or more frames, formatted on the wire as follows:
* Frame 0: Empty frame
* Frame 1: "MDPW01" (six bytes, representing MDP/Worker v0.1)
* Frame 2: 0x02 (one byte, representing REQUEST)
* Frame 3: Client address (envelope stack)
* Frame 4: Empty (zero bytes, envelope delimiter)
* Frames 5+: Request body (opaque binary)
A **REPLY** command consists of a multipart message of 6 or more frames, formatted on the wire as follows:
* Frame 0: Empty frame
* Frame 1: "MDPW01" (six bytes, representing MDP/Worker v0.1)
* Frame 2: 0x03 (one byte, representing REPLY)
* Frame 3: Client address (envelope stack)
* Frame 4: Empty (zero bytes, envelope delimiter)
* Frames 5+: Reply body (opaque binary)
A **HEARTBEAT** command consists of a multipart message of 3 frames, formatted on the wire as follows:
* Frame 0: Empty frame
* Frame 1: "MDPW01" (six bytes, representing MDP/Worker v0.1)
* Frame 2: 0x04 (one byte, representing HEARTBEAT)
A **DISCONNECT** command consists of a multipart message of 3 frames, formatted on the wire as follows:
* Frame 0: Empty frame
* Frame 1: "MDPW01" (six bytes, representing MDP/Worker v0.1)
* Frame 2: 0x05 (one byte, representing DISCONNECT)
MD/Worker commands all start with an empty frame to allow consistent processing of client and worker frames in a broker, over a single socket. The empty frame has no other significance.
++++ Opening and Closing a Connection
* The worker is responsible for opening and closing a logical connection. One worker MUST connect to exactly one broker using a single ØMQ DEALER (XREQ) socket.
* Since ØMQ automatically reconnects peers after a failure, every MDP command includes the protocol header to allow proper validation of all messages that a peer receives.
* The worker opens the connection to the broker by creating a new socket, connecting it, and then sending a READY command to register a service. One worker handles precisely one service, and many workers MAY handle the same service. The worker MUST NOT send a further READY.
* There is no response to a READY. The worker SHOULD assume the registration succeeded until or unless it receives a DISCONNECT, or it detects a broker failure through heartbeating.
* The worker MAY send DISCONNECT at any time, including before READY. When the broker receives DISCONNECT from a worker it MUST send no further commands to that worker.
* The broker MAY send DISCONNECT at any time, by definition after it has received at least one command from the worker.
* The broker MUST respond to any valid but unexpected command by sending DISCONNECT and then no further commands to that worker. The broker SHOULD respond to invalid messages by dropping them and treating that peer as invalid.
* When the worker receives DISCONNECT it must send no further commands to the broker; it MUST close its socket, and reconnect to the broker on a new socket. This mechanism allows workers to re-register after a broker failure and recovery.
++++ Request and Reply Processing
* The REQUEST and the REPLY commands MUST contain precisely one client address frame. This frame MUST be followed by an empty (zero sized) frame.
* The address of each directly connected client is prepended by the ROUTER socket to all request messages coming from clients. That ROUTER socket also expects a client address to be prepended to each reply message sent to a client.
++++ Heartbeating
* HEARTBEAT commands are valid at any time, after a READY command.
* Any received command except DISCONNECT acts as a heartbeat. Peers SHOULD NOT send HEARTBEAT commands while also sending other commands.
* Both broker and worker MUST send heartbeats at regular and agreed-upon intervals. A peer MUST consider the other peer "disconnected" if no heartbeat arrives within some multiple of that interval (usually 3-5).
* If the worker detects that the broker has disconnected, it SHOULD restart a new conversation.
* If the broker detects that the worked has disconnected, it SHOULD stop sending it messages of any type.
+++ Reliability
The Majordomo pattern is designed to extend the basic ØMQ request-reply pattern with the ability to detect and recover from a specific set of failures:
* Worker applications which crash, run too slowly, or freeze.
* Worker applications that are disconnected from the network (temporarily or permanently).
* Client applications that are temporarily disconnected from the network.
* A queue broker that crashes and is restarted.
* A queue broker that suffers a permanent failure.
* Requests or replies that are lost due to any of these failures.
The general approach is to retry and reconnect, using heartbeating when needed. Majordomo supports live-live broker failover indirectly, by keeping no significant state in a broker. Actual failover to alternate brokers is handled by the clients and workers, not the protocol. For a fuller discussion of these failures and the strategies that Majordomo uses to detect and recover, see Chapter 4 of the Guide[((bibcite guide))].
+++ Scalability and Performance
Majordomo is designed to be scalable to large numbers (thousands) of workers and clients, limited only by system resources on the broker. Partitioning of workers by service allows for multiple applications to share the same broker infrastructure.
Throughput performance for a single client application will be limited to tens of thousands, not millions, of request-reply transactions per second due to round-trip costs and the extra latency of a broker-based approach. The larger the request and reply messages, the more efficient Majordomo will become. Majordomo may be complemented by high-speed data delivery architectures.
System requirements for the broker are moderate: no more than one outstanding request per client will be queued, and message contents can be switched between clients and workers without copying or processing. A single broker thread can therefore switch several million messages per second, and multithreaded implementations (offering multiple virtual brokers, each on its own port) can scale to as many cores as required.
+++ Security
MDP does not implement any authentication, access control, or encryption mechanisms and should not be used in any deployment where these are required. However, all these can be layered on top of MDP, and may be the subject of future RFCs.
+++ Reference Implementations
The C99 Majordomo examples from Chapter 4 of the Guide[((bibcite guide))] act as the prime reference implementation for MDP. There is a [http://guidog.github.com/pyzmq-mdp/ Python implementation] of MDP by Guido Goldstein.
+++ Known Weaknesses
* The heartbeat rate must be set to similar values in broker and worker, or false disconnections will occur. A better heartbeat design will be developed later.
* The use of multiple frames for command formatting has a performance impact. Future versions of MDP may place commands into single frames.
++ References
[[bibliography]]
: rfc2119 : "Key words for use in RFCs to Indicate Requirement Levels" - [http://tools.ietf.org/html/rfc2119 ietf.org]
: fandos : "Definition of a Free and Open Standard" - [http://www.digistan.org/open-standard:definition digistan.org]
: coss : "Consensus Oriented Specification System" - [http://www.digistan.org/spec:1/COSS digistan.org]
: guide : "ØMQ - The Guide" - [http://zguide.zeromq.org zeromq.org]
: zmqsocket : "zmq_socket(3)" [http://api.zeromq.org/master:zmq-socket zeromq.org]
[[/bibliography]]