Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100755 228 lines (159 sloc) 6.826 kB
bc8dfdd @agentzh more README tweaks
authored
1 This is a queue-size-ware version of memcacheq
2 (http://memcachedb.org/memcacheq/) from xunxin and agentzh
3 originally developed for Yahoo! China's webpage information
fe1779c @agentzh README tunes
authored
4 extraction cluster based on WebKit.
5
6 It's opensourced here meant to be useful for others and for
7 the original author of memcacheq's potential consideration of
8 merging back to the mainstream.
3c1d358 added more docs to README
agentzh authored
9
10 This fork adds the ability to limit and query the length of queues
11 and to specify a length limit when creating a queue.
12
13 Here's the usage of this fork.
14
15 * Server side
16
17 $ memcacheq -u my_user -p 5000 -B 4064 -A 4096 -r -c 1024 -m 64 -N \
18 -H /path/to/my/bdbdata/base
19
20 where -u indicates the user who runs this deamon, -p specifies a port
21 number that this daemon listens to, -B specifies the message body length
22 in bytes (but actually you can only store messages of size tens of bytes
23 smaller that that), -A specifies the underlying BDB page size, -m
24 specifies the in-memory cache size, -N uses BDB's no-sync feature to
25 gain more speed at the cost of consistency, and -H specifies the on-disk
26 storage location. See memcacheq's command-line usage for more information.
27
28 To start the memcacheq daemon:
29
30 If there's corruption in your memcacheq's underlying BDB database, try
31 the following command:
32
48f2770 @agentzh fixed a bad path in README
authored
33 $ /path/to/your/bdb/bin/db_recover -h /path/to/your/bdbdata/base
3c1d358 added more docs to README
agentzh authored
34
35 * Client side
36
37 We use pseudo libmemcached C code to demonstrate the client-side usage.
38
39 1. To create a queue:
40
41 memcached_add(memc, queue_name, size_limit);
42
43 where "size_limit" is the maximum number of elements in that queue named
44 "queue_name".
45
a14001c documented some of the exceptional error code
agentzh authored
46 Note that the number of queues is limited by the underlying BDB queue
47 storage.
48
3c1d358 added more docs to README
agentzh authored
49 2. To remove a queue:
50
51 memcached_delete(memc, queue_name);
52
53 3. To insert an element to a queue:
54
55 memcached_set(memc, queue_name, element_content);
56
a14001c documented some of the exceptional error code
agentzh authored
57 Once the queue has already reached the length limit, it returns
58 the standard memcached "NOT STORED" exception code.
59
3c1d358 added more docs to README
agentzh authored
60 4. To read an element from a queue:
61
62 element_content = memcached_get(memc, queue_name);
63
a14001c documented some of the exceptional error code
agentzh authored
64 Once the queue is already empty, it returns a standard memcached
65 "NOT FOUND" exception code.
66
3c1d358 added more docs to README
agentzh authored
67 5. To monitor the queues' state in a certain memcacheq server
68 (here we use a shell command to illustrate):
69
70 $ echo stats queue | nc 10.62.100.35:11211
71
72 where the memcacheq daemon listens the 11211 port at 10.62.100.35. A
73 typical instance of the output might be:
74
75 STAT bbsdetails 0 2000000
76 STAT bbslists 72 2000000
77 STAT comment 3 1234567
78 STAT done 0 1234567
79 STAT initial 25 1234567
80 STAT pagecat 5006 1234567
81 STAT preprocessed 10 500
82 END
83
84 where the second column in the listing specify the names of the queues,
85 the third column the current size of the queues, and the forth column
86 the size limit of the queues.
87
a14001c documented some of the exceptional error code
agentzh authored
88 Note that the queue size is stored separately in contrast to real-time
89 counting, so it might be kinda out of sync if the daemon exits abnormally
90 and the -N option is specified when starting the daemon.
91
15814f5 @agentzh emphasize Perl 5 in README
authored
92 We usually use the command-line utilities as well as the high-level Perl 5
3c1d358 added more docs to README
agentzh authored
93 library provided by the Queue::Memcached::Buffered module opensourced here:
94
95 http://github.com/agentzh/queue-memcached-buffered
96
97 Below is the original memcacheq documentation in README:
98
3888b56 add hacked memcacheq
xunxin authored
99 ===============================================
100 MemcacheQ - Simple Queue Service over Memcache
101 ===============================================
102
103 Features
104 =========
105 * damn simple
106 * very fast
107 * multiple queue
108 * concurrent well
109 * memcache protocol compatible
110
111 Getting Started
112 ===============
113
114 Download
115 ---------
116 See: <http://code.google.com/p/memcacheq/downloads/list>
117
118 Installation
119 -------------
120 See: <http://memcachedb.org/memcacheq/INSTALL.html>
121
122 Please take a look at 'ChangLog' file in the distribution, see what's new.
123
124 Commands
125 ---------
126
127 Only two commands are used to operate the queue:
128
129 **Append a message to the tail of queue**::
130
131 set <queue name> <flags> 0 <message_len>\r\n
132 <put your message body here>\r\n
133 STORED\r\n
134
135 **Note:** MQ will create a new queue automatically if your queue is not existed. The original 'expire time' field is ignored by server.
136
137 **Consume a message from the head of queue**::
138
139 get <queue name>\r\n
140 VALUE <queue name> <flags> <message_len>\r\n
141 <your message body will come here>\r\n
142 END\r\n
143
144
145 Examples
146 ---------
147
148 Assuming you are using PHP memcache<http://www.php.net/memcache>::
149
150 <?php
151 /* connect to memcached server */
152 $memcache_obj = memcache_connect('memcacheq_host', 21201);
153
154 /* append a message to queue */
155 memcache_set($memcache_obj, 'demoqueue1', 'message body here', 0, 0);
156
157 /* consume a message from 'demoqueue1' */
158 memcache_get($memcache_obj, 'demoqueue1');
159
160 memcache_close($memcache_obj);
161 ?>
162
163 Limitation
164 ===========
165 The message body is stored in Berkeley DB with fixed length. Any message that is shorter than the declared length will automatically be padded with space character (0x20 in the ASCII character set).
166
167 In Berkeley DB, as the official document refers,
168
169 "For the Queue access method, the record length must be enough smaller than the database's page size that at least one record plus the database page's metadata information can fit on each database page."
170
171 "The minimum page size is 512 bytes, the maximum page size is 64K bytes, and the page size must be a power-of-two."
172
173 So we have a limit on the message body size with a max of a bit less than *64K*.
174
175 Other tips
176 ===========
177 use 'stats queue' to see your current queues::
178
179 $ telnet 127.0.0.1 22201
180 Trying 127.0.0.1...
181 Connected to localhost.
182 Escape character is '^]'.
183 stats queue
184 STAT test1
185 STAT test2
186 STAT test3
187 STAT test4
188 END
189
190 delete a queue::
191
192 $ telnet 127.0.0.1 22201
193 Trying 127.0.0.1...
194 Connected to localhost.
195 Escape character is '^]'.
196 delete test1
197 DELETED
198
199
200 'db_stat' a queue to see how many records now in::
201
202 $ cd <your queue dir>
203 $ /usr/local/BerkeleyDB.4.7/bin/db_stat -d test1
204 Mon Sep 22 20:25:56 2008 Local time
205 42253 Queue magic number
206 4 Queue version number
207 1024 Fixed-length record size
208 0x20 Fixed-length record pad
209 4096 Underlying database page size
210 131072 Underlying database extent size
211 100000 Number of records in the database
212 33334 Number of database pages
213 2048 Number of bytes free in database pages (99% ff)
214 1 First undeleted record
215 100001 Next available record number
216
217 Feedback
218 =========
219 MemcacheDB mailing list now hosts on Google Group: http://groups.google.com/group/memcachedb
220
221 * To subscribe to maillist, send email to memcachedb-subscribe@googlegroups.com
222 * To post to maillist, send email to memcachedb@googlegroups.com
223 * To unsubscribe from maillist, send email to memcachedb-unsubscribe@googlegroups.com
224
225 Please report your bugs and issues to the Maillist.
226
227 Last updated by Steve Chu<http://stvchu.org>: 09/22/2008
Something went wrong with that request. Please try again.