Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 593 lines (399 sloc) 24.059 kB
37db4e0 @nicotaing [doc] add protocol.md
nicotaing authored
1 # Beanstalkd
2
3 ## Protocol
4
5 ### Description
6
7 The beanstalk protocol runs over TCP using ASCII encoding. Clients connect, send commands and data, wait for responses, and close the connection. For each connection, the server processes commands serially in the order in which they were received and sends responses in the same order. All integers in the protocol are formatted in decimal and (unless otherwise indicated) nonnegative.
8
9 ### Name convention
10
11 Names only supports ASCII strings.
12
13 #### Characters Allowed
14
15 * **letters** (A-Z and a-z)
16 * **numerals** (0-9)
17 * **hyphen** ("-")
18 * **plus** ("+")
19 * **slash** ("/")
20 * **semicolon** (";")
21 * **dot** (".")
22 * **dollar-sign** ("$")
23 * **underscore** ("_")
24 * **parentheses** ("*(*" and "*)*")
25
26 **Notice:** They may not begin with a hyphen and they are terminated by white space (either a space char or end of line). Each name must be at least one character long.
27
28 ### Errors
29
30 | Errors | Description |
31 | --------------------| ------------- |
32 | `OUT_OF_MEMORY\r\n` | The server cannot allocate enough memory for the job. The client should try again later.|
33 | `INTERNAL_ERROR\r\n` | This indicates a bug in the server. It should never happen. If it does happen, please report it at http://groups.google.com/group/beanstalk-talk. |
34 | `BAD_FORMAT\r\n` | The client sent a command line that was not well-formed. This can happen if the line does not end with \r\n, if non-numeric characters occur where an integer is expected, if the wrong number of arguments are present, or if the command line is mal-formed in any other way. |
35 | `UNKNOWN_COMMAND\r\n` | The client sent a command that the server does not know. |
36
37
38 ### Job Lifecycle
39
40 A job in beanstalk gets created by a client with the `put` command. During its life it can be in one of four states:
41
42 | Status | Description |
43 | --------------------| ------------- |
44 | `ready` | it waits in the ready queue until a worker comes along and runs the "reserve" command |
45 | `reserved` | if this job is next in the queue, it will be reserved for the worker. The worker will execute the job |
46 | `delayed` | when it's waiting "x" seconds before to be `ready` |
47 | `buried` | when it is finished the worker will send a "delete" ; when it is finished the worker will send a "delete" |
48
49
50
51 Here is a picture of the typical job lifecycle:
52
53 ```
54 put reserve delete
55 -----> [READY] ---------> [RESERVED] --------> *poof*
56 ```
57
58
59
60 Here is a picture with more possibilities:
61
62 ```
63 put with delay release with delay
64 ----------------> [DELAYED] <------------.
65 | |
66 | (time passes) |
67 | |
68 put v reserve | delete
69 -----------------> [READY] ---------> [RESERVED] --------> *poof*
70 ^ ^ | |
71 | \ release | |
72 | `-------------' |
73 | |
74 | kick |
75 | |
76 | bury |
77 [BURIED] <---------------'
78 |
79 | delete
80 `--------> *poof*
81 ```
82
83 ### Tubes
84
85 The system has one or more tubes. Each tube consists of a ready queue and a delay queue. Each job spends its entire life in one tube. Consumers can show interest in tubes by sending the `watch` command; they can show disinterest by sending the `ignore` command. This set of interesting tubes is said to be a consumer's `watch list`. When a client reserves a job, it may come from any of the tubes in its watch list.
86
87 When a client connects, its watch list is initially just the tube named `default`. If it submits jobs without having sent a `use` command, they will live in the tube named `default`.
88
89 Tubes are created on demand whenever they are referenced. If a tube is empty (that is, it contains no `ready`, `delayed`, or `buried` jobs) and no client refers to it, it will be deleted.
90
91 ## Commands
92
93 ### Producer Commands
94
95 #### `put` command
96
97 The `put` command is for any process that wants to insert a job into the queue. It comprises a command line followed by the job body:
98
99 ```
100 put <pri> <delay> <ttr> <bytes>\r\n
101 <data>\r\n
102 ```
103
104 #####`put` options
105
106 It inserts a job into the client's currently used tube (see the `use` command below).
107
108 * `<pri>` is an integer < 2**32. Jobs with smaller priority values will be scheduled before jobs with larger priorities. The most urgent priority is 0;the least urgent priority is 4,294,967,295.
109 * `<delay>` is an integer number of seconds to wait before putting the job in the ready queue. The job will be in the "delayed" state during this time.
110 * `<ttr>` -- time to run -- is an integer number of seconds to allow a worker to run this job. This time is counted from the moment a worker reserves this job. If the worker does not delete, release, or bury the job within `<ttr>` seconds, the job will time out and the server will release the job. The minimum ttr is 1. If the client sends 0, the server will silently increase the ttr to 1.
111 * `<bytes>` is an integer indicating the size of the job body, not including the trailing "\r\n". This value must be less than max-job-size (default: 2**16).
112 * `<data>` is the job body -- a sequence of bytes of length <bytes> from the previous line.
113
114
115 ##### `put` responses
116 After sending the command line and body, the client waits for a reply, which
117 may be:
118
119 * `INSERTED <id>\r\n` to indicate success. `<id>` is the integer id of the new job
120 * `BURIED <id>\r\n` if the server ran out of memory trying to grow the priority queue data structure. `<id>` is the integer id of the new job
121 * `EXPECTED_CRLF\r\n` The job body must be followed by a CR-LF pair, that is, `"\r\n"`. These two bytes are not counted in the job size given by the client in the put command line.
122 * `JOB_TOO_BIG\r\n` The client has requested to put a job with a body larger than max-job-size bytes.
123 * `DRAINING\r\n` This means that the server has been put into "drain mode" and is no longer accepting new jobs. The client should try another server or disconnect and try again later.
124
125 #### `use` command
126
127 The `use` command is for producers. Subsequent put commands will put jobs into the tube specified by this command. If no use command has been issued, jobs will be put into the tube named `default`.
128
129 ```
130 use <tube>\r\n
131 ```
132
133 ##### `use` options
134
135 * `<tube>` is a name at most 200 bytes. It specifies the tube to use. If the tube does not exist, it will be created.
136
137 ##### `use` responses
138
139 * `USING <tube>\r\n` -- `<tube>` is the name of the tube now being used.
140
bf4dd7a @nicotaing [protocol.md] add bury description
nicotaing authored
141 ### Worker Commands
37db4e0 @nicotaing [doc] add protocol.md
nicotaing authored
142
143 A process that wants to consume jobs from the queue uses those commands:
144
145 * `reserve`
146 * `delete`
147 * `release`
148 * `bury`
149
150 #### `reserve` command
151
152 ```
153 reserve\r\n
154 ```
155
156 Alternatively, you can specify a timeout as follows:
157
158 ```
159 reserve-with-timeout <seconds>\r\n
160 ```
161
162 This will return a newly-reserved job. If no job is available to be reserved, beanstalkd will wait to send a response until one becomes available. Once a job is reserved for the client, the client has limited time to run (TTR) the job before the job times out. When the job times out, the server will put the job back into the ready queue. Both the TTR and the actual time left can be found in response to the `stats-job` command.
163
164 A timeout value of `0` will cause the server to immediately return either a response or `TIMED_OUT`. A positive value of timeout will limit the amount of time the client will block on the reserve request until a job becomes available.
165
166 ##### `reserve` responses
167
168 ###### Non-succesful responses
169
170 * `DEADLINE_SOON\r\n` During the TTR of a reserved job, the last second is kept by the server as a safety margin, during which the client will not be made to wait for another job. If the client issues a reserve command during the safety margin, or if the safety margin arrives while the client is waiting on a reserve command.
171 * `TIMED_OUT\r\n` If a non-negative timeout was specified and the timeout exceeded before a job became available, the server will respond with TIMED_OUT.
172
173 Otherwise, the only other response to this command is a successful reservation
174 in the form of a text line followed by the job body:
175
176 ####### Succesful response
177
178
179 ```
180 RESERVED <id> <bytes>\r\n
181 <data>\r\n
182 ```
183
184 * `<id>` is the job id -- an integer unique to this job in this instance of beanstalkd.
185 * `<bytes>` is an integer indicating the size of the job body, not including the trailing `\r\n"`.
186 * `<data>` is the job body -- a sequence of bytes of length <bytes> from the previous line. This is a verbatim copy of the bytes that were originally sent to the server in the put command for this job.
187
188 #### `delete` command
189
190 The delete command removes a job from the server entirely. It is normally used by the client when the job has successfully run to completion. A client can delete jobs that it has `reserved`, `ready` jobs, `delayed` jobs, and jobs that are
191 `buried`. The delete command looks like this:
192
193 ```
194 delete <id>\r\n
195 ```
196
197 ##### `delete` options
198
199 * `<id>` is the job id to delete.
200
201 ##### `delete` responses
202
203 The client then waits for one line of response, which may be:
204
205 * `DELETED\r\n` to indicate success.
206 * `NOT_FOUND\r\n` if the job does not exist or is not either reserved by the client, ready, or buried. This could happen if the job timed out before the client sent the delete command.
207
208 #### `release` command
209
210 The release command puts a `reserved` job back into the ready queue (and marks its state as `ready`) to be run by any client. It is normally used when the job fails because of a transitory error. It looks like this:
211
212 ```
213 release <id> <pri> <delay>\r\n
214 ```
215
216 ##### `release` options
217
218 * `<id>` is the job id to release.
219 * `<pri>` is a new priority to assign to the job.
220 * `<delay>` is an integer number of seconds to wait before putting the job in the ready queue. The job will be in the "delayed" state during this time.
221
222 ##### `release` responses
223
224 The client expects one line of response, which may be:
225
226 * `RELEASED\r\n` to indicate success.
227 * `BURIED\r\n` if the server ran out of memory trying to grow the priority queue data structure.
228 * `NOT_FOUND\r\n` if the job does not exist or is not reserved by the client.
229
bf4dd7a @nicotaing [protocol.md] add bury description
nicotaing authored
230 #### `bury` command
231 The bury command puts a job into the "buried" state. Buried jobs are put into a FIFO linked list and will not be touched by the server again until a client kicks them with the `kick`" command.
232
233 The bury command looks like this:
234
235 ```
236 bury <id> <pri>\r\n
237 ```
238
239 ##### `bury` options
240
241 * `<id>` is the job id to release.
242 * `<pri>` is a new priority to assign to the job.
243
244 ##### `bury` responses
245
246 There are two possible responses:
247
248 * `BURIED\r\n` to indicate success.
249 * `NOT_FOUND\r\n` if the job does not exist or is not reserved by the client.
250
251
37db4e0 @nicotaing [doc] add protocol.md
nicotaing authored
252 #### `touch` command
253
254 The `touch` command allows a worker to request more time to work on a job. This is useful for jobs that potentially take a long time, but you still want the benefits of a TTR pulling a job away from an unresponsive worker. A worker may periodically tell the server that it's still alive and processing a job (e.g. it may do this on `DEADLINE_SOON`).
255
256 The touch command looks like this:
257
258 ```
259 touch <id>\r\n
260 ```
261
262 ##### `touch` options
263
264 * `<id>` is the ID of a job reserved by the current connection.
265
266 ##### `touch` responses
267
268 There are two possible responses:
269
270 * `TOUCHED\r\n` to indicate success.
271 * `NOT_FOUND\r\n` if the job does not exist or is not reserved by the client.
272
273 #### `watch` command
274
275 The `watch` command adds the named tube to the watch list for the current connection. A reserve command will take a job from any of the tubes in the watch list. For each new connection, the watch list initially consists of one tube, named `default`.
276
277 ```
278 watch <tube>\r\n
279 ```
280
281 ##### `watch` options
282
283 * `<tube>` is a name at most 200 bytes. It specifies a tube to add to the watch list. If the tube doesn't exist, it will be created.
284
285 ##### `watch` responses
286
287 The reply is:
288
289 * `WATCHING <count>\r\n` `<count>` is the integer number of tubes currently in the watch list.
290
291 ##### `ignore` command
292
293 The `ignore` command is for consumers. It removes the named tube from the watch list for the current connection.
294
295 ```
296 ignore <tube>\r\n
297 ```
298
299 ##### `ignore` options
300
301 * `<tube>` is a name at most 200 bytes. It specifies a tube to add to the watch list. If the tube doesn't exist, it will be created.
302
303 ##### `ignore` command
304
305 The reply is one of:
306
307 * `WATCHING <count>\r\n` to indicate success. `<count>` is the integer number of tubes currently in the watch list.
308 * `NOT_IGNORED\r\n` if the client attempts to ignore the only tube in its watch list.
309
310 ### Other Commands
311
312 #### `peek` command
313
314 The peek commands let the client inspect a job in the system. There are four variations. All but the first operate only on the currently used tube.
315
316 * `peek <id>\r\n` - return `job <id>`.
317 * `peek-ready\r\n` - return the next ready job.
318 * `peek-delayed\r\n` - return the delayed job with the shortest delay left.
319 * `peek-buried\r\n` - return the next job in the list of buried jobs.
320
321 ##### `peek` responses
322
323 There are two possible responses, either a single line:
324
325 * `NOT_FOUND\r\n` if the requested job doesn't exist or there are no jobs in the requested state.
326 * `FOUND <id> <bytes>\r\n <data>\r\n`
327 * `<id>` is the job id.
328 * `<bytes>` is an integer indicating the size of the job body, not including the trailing `\r\n`.
329 * `<data>` is the job body -- a sequence of bytes of length <bytes> from the previous line.
330
331 #### `kick` command
332
333 The kick command applies only to the currently used tube. It moves jobs into the ready queue. If there are any buried jobs, it will only kick buried jobs. Otherwise it will kick delayed jobs. It looks like:
334
335 ```
336 kick <bound>\r\n
337 ```
338
339 ##### `kick` options
340
341 * `<bound>` is an integer upper bound on the number of jobs to kick. The server will kick no more than <bound> jobs.
342
343 ##### `kick` responses
344
345 The response is of the form:
346
347 * `KICKED <count>\r\n`
348 * <count> is an integer indicating the number of jobs actually kicked.
349
350 #### `kick-job` command
351
352 The kick-job command is a variant of kick that operates with a single job identified by its job id. If the given job id exists and is in a buried or delayed state, it will be moved to the ready queue of the the same tube where it currently belongs. The syntax is:
353
354 ```
355 kick-job <id>\r\n
356 ```
357
358 ##### `kick-job` options
359
360 * <id> is the job id to kick.
361
362 ##### `kick-job` responses
363
364 The response is one of:
365
366 * `NOT_FOUND\r\n` if the job does not exist or is not in a kickable state. This can also happen upon internal errors.
367 * `KICKED\r\n` when the operation succeeded.
368
369 #### `stats-jobs` command
370
371 The stats-job command gives statistical information about the specified job if it exists. Its form is:
372
373 ```
374 stats-job <id>\r\n
375 ```
376
377 ##### `stats-jobs` options
378
379 * `<id>` is a job id.
380
381 ##### `stats-jobs` responses
382
383 The response is one of:
384
385 * `NOT_FOUND\r\n` if the job does not exist.
386 * `OK <bytes>\r\n<data>\r\n`
387 * `<bytes>` is the size of the following data section in bytes.
388 * `<data>` is a sequence of bytes of length `<bytes>` from the previous line. It is a YAML file with statistical information represented a dictionary.
389
390 The `stats-job` data is a YAML file representing a single dictionary of strings to scalars. It contains these keys:
391
392 * `id` is the job id
393 * `tube` is the name of the tube that contains this job
394 * `state` is `ready` or `delayed` or `reserved` or `buried`
395 * `pri` is the priority value set by the put, release, or bury commands.
396 * `age` is the time in seconds since the put command that created this job.
397 * `time-left` is the number of seconds left until the server puts this job into the ready queue. This number is only meaningful if the job is reserved or delayed. If the job is reserved and this amount of time elapses before its state changes, it is considered to have timed out.
398 * `file` is the number of the earliest binlog file containing this job. If -b wasn't used, this will be 0.
399 * `reserves` is the number of times this job has been reserved.
400 * `timeouts` is the number of times this job has timed out during a reservation.
401 * `releases` is the number of times a client has released this job from a reservation.
402 * `buries` is the number of times this job has been buried.
403 * `kicks` is the number of times this job has been kicked.
404
405 #### `stats-tube` command
406
407 The stats-tube command gives statistical information about the specified tube if it exists. Its form is:
408
409 ```
410 stats-tube <tube>\r\n
411 ```
412
413 ##### `stats-tube` options
414
415 * `<tube>` is a name at most 200 bytes. Stats will be returned for this tube.
416
417 ##### `stats-tube` responses
418
419 The response is one of:
420
421 * `NOT_FOUND\r\n` if the tube does not exist.
422 * `OK <bytes>\r\n<data>\r\n`
423 * `<bytes>` is the size of the following data section in bytes.
424 * `<data>` is a sequence of bytes of length `<bytes>` from the previous line. It is a YAML file with statistical information represented a dictionary.
425
426 The stats-tube data is a YAML file representing a single dictionary of strings to scalars. It contains these keys:
427
428 * `name` is the tube's name.
429 * `current-jobs-urgent` is the number of ready jobs with priority < 1024 in this tube.
430 * `current-jobs-ready` is the number of jobs in the ready queue in this tube.
431 * `current-jobs-reserved` is the number of jobs reserved by all clients in this tube.
432 * `current-jobs-delayed` is the number of delayed jobs in this tube.
433 * `current-jobs-buried` is the number of buried jobs in this tube.
434 * `total-jobs` is the cumulative count of jobs created in this tube in the current beanstalkd process.
435 * `current-using` is the number of open connections that are currently using this tube.
436 * `current-waiting` is the number of open connections that have issued a reserve command while watching this tube but not yet received a response.
437 * `current-watching` is the number of open connections that are currently watching this tube.
438 * `pause` is the number of seconds the tube has been paused for.
439 * `cmd-delete` is the cumulative number of delete commands for this tube
440 * `cmd-pause-tube` is the cumulative number of pause-tube commands for this tube.
441 * `pause-time-left` is the number of seconds until the tube is un-paused.
442
443 #### `stats` command
444
445 The stats command gives statistical information about the system as a whole. Its form is:
446
447 ```
448 stats\r\n
449 ```
450
451 ##### `stats` responses
452
453 The server will respond:
454
455 ```
456 OK <bytes>\r\n
457 <data>\r\n
458 ```
459
460 * `<bytes>` is the size of the following data section in bytes.
461 * `<data>` is a sequence of bytes of length <bytes> from the previous line. It is a YAML file with statistical information represented a dictionary.
462
463 The stats data for the system is a YAML file representing a single dictionary of strings to scalars. Entries described as "cumulative" are reset when the beanstalkd process starts; they are not stored on disk with the -b flag.
464
465 * `current-jobs-urgent` is the number of ready jobs with priority < 1024.
466 * `current-jobs-ready` is the number of jobs in the ready queue.
467 * `current-jobs-reserved` is the number of jobs reserved by all clients.
468 * `current-jobs-delayed` is the number of delayed jobs.
469 * `current-jobs-buried` is the number of buried jobs.
470 * `cmd-put` is the cumulative number of put commands.
471 * `cmd-peek` is the cumulative number of peek commands.
472 * `cmd-peek-ready` is the cumulative number of peek-ready commands.
473 * `cmd-peek-delayed` is the cumulative number of peek-delayed commands.
474 * `cmd-peek-buried` is the cumulative number of peek-buried commands.
475 * `cmd-reserve` is the cumulative number of reserve commands.
476 * `cmd-use` is the cumulative number of use commands.
477 * `cmd-watch` is the cumulative number of watch commands.
478 * `cmd-ignore` is the cumulative number of ignore commands.
479 * `cmd-delete` is the cumulative number of delete commands.
480 * `cmd-release` is the cumulative number of release commands.
481 * `cmd-bury` is the cumulative number of bury commands.
482 * `cmd-kick` is the cumulative number of kick commands.
483 * `cmd-stats` is the cumulative number of stats commands.
484 * `cmd-stats-job` is the cumulative number of stats-job commands.
485 * `cmd-stats-tube` is the cumulative number of stats-tube commands.
486 * `cmd-list-tubes` is the cumulative number of list-tubes commands.
487 * `cmd-list-tube-used` is the cumulative number of list-tube-used commands.
488 * `cmd-list-tubes-watched` is the cumulative number of list-tubes-watched commands.
489 * `cmd-pause-tube` is the cumulative number of pause-tube commands
490 * `job-timeouts` is the cumulative count of times a job has timed out.
491 * `total-jobs` is the cumulative count of jobs created.
492 * `max-job-size` is the maximum number of bytes in a job.
493 * `current-tubes` is the number of currently-existing tubes.
494 * `current-connections` is the number of currently open connections.
495 * `current-producers` is the number of open connections that have each issued at least one put command.
496 * `current-workers` is the number of open connections that have each issued at least one reserve command.
497 * `current-waiting` is the number of open connections that have issued a reserve command but not yet received a response.
498 * `total-connections` is the cumulative count of connections.
499 * `pid` is the process id of the server.
500 * `version` is the version string of the server.
501 * `rusage-utime` is the cumulative user CPU time of this process in seconds and microseconds.
502 * `rusage-stime` is the cumulative system CPU time of this process in seconds and microseconds.
503 * `uptime` is the number of seconds since this server process started running.
504 * `binlog-oldest-index` is the index of the oldest binlog file needed to store the current jobs
505 * `binlog-current-index` is the index of the current binlog file being written to. If binlog is not active this value will be 0
506 * `binlog-max-size` is the maximum size in bytes a binlog file is allowed to get before a new binlog file is opened
507 * `binlog-records-written` is the cumulative number of records written to the binlog
508 * `binlog-records-migrated` is the cumulative number of records written as part of compaction
509
510 #### `list-tubes` command
511
512 The list-tubes command returns a list of all existing tubes. Its form is:
513
514 ```
515 list-tubes\r\n
516 ```
517
518 ##### `list-tubes` responses
519
520 The response is:
521
522 ```
523 OK <bytes>\r\n
524 <data>\r\n
525 ```
526
527 * `<bytes>` is the size of the following data section in bytes.
528 * `<data>` is a sequence of bytes of length <bytes> from the previous line. It is a YAML file containing all tube names as a list of strings.
529
530 #### `list-tube-used` command
531
532 The list-tube-used command returns the tube currently being used by the client. Its form is:
533
534 ```
535 list-tube-used\r\n
536 ```
537
538 ##### `list-tube-used` responses
539 The response is:
540
541 ```
542 USING <tube>\r\n
543 ```
544
545 * `<tube>` is the name of the tube being used.
546
547 #### `list-tubes-watched` command
548
549 The list-tubes-watched command returns a list tubes currently being watched by the client. Its form is:
550
551 ```
552 list-tubes-watched\r\n
553 ```
554
555 ##### `list-tubes-watched` responses
556
557 The response is:
558
559 ```
560 OK <bytes>\r\n
561 <data>\r\n
562 ```
563
564 * `<bytes>` is the size of the following data section in bytes.
565 * `<data>` is a sequence of bytes of length <bytes> from the previous line. It is a YAML file containing watched tube names as a list of strings.
566
567 #### `quit` command
568
569 The quit command simply closes the connection. Its form is:
570
571 ```
572 quit\r\n
573 ```
574
575 #### `pause-tube` command
576
577 The pause-tube command can delay any new job being reserved for a given time. Its form is:
578
579 ```
580 pause-tube <tube-name> <delay>\r\n
581 ```
582
583 ##### `pause-tube` options
584
585 * `<tube>` is the tube to pause
586 * `<delay>` is an integer number of seconds to wait before reserving any more jobs from the queue
587
588 ##### `pause-tube` responses
589
590 There are two possible responses:
591
592 * `PAUSED\r\n` to indicate success.
593 * `NOT_FOUND\r\n` if the tube does not exist.
Something went wrong with that request. Please try again.