Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
executable file 228 lines (159 sloc) 6.67 KB
This is a queue-size-ware version of memcacheq
( from xunxin and agentzh
originally developed for Yahoo! China's webpage information
extraction cluster based on WebKit.
It's opensourced here meant to be useful for others and for
the original author of memcacheq's potential consideration of
merging back to the mainstream.
This fork adds the ability to limit and query the length of queues
and to specify a length limit when creating a queue.
Here's the usage of this fork.
* Server side
$ memcacheq -u my_user -p 5000 -B 4064 -A 4096 -r -c 1024 -m 64 -N \
-H /path/to/my/bdbdata/base
where -u indicates the user who runs this deamon, -p specifies a port
number that this daemon listens to, -B specifies the message body length
in bytes (but actually you can only store messages of size tens of bytes
smaller that that), -A specifies the underlying BDB page size, -m
specifies the in-memory cache size, -N uses BDB's no-sync feature to
gain more speed at the cost of consistency, and -H specifies the on-disk
storage location. See memcacheq's command-line usage for more information.
To start the memcacheq daemon:
If there's corruption in your memcacheq's underlying BDB database, try
the following command:
$ /path/to/your/bdb/bin/db_recover -h /path/to/your/bdbdata/base
* Client side
We use pseudo libmemcached C code to demonstrate the client-side usage.
1. To create a queue:
memcached_add(memc, queue_name, size_limit);
where "size_limit" is the maximum number of elements in that queue named
Note that the number of queues is limited by the underlying BDB queue
2. To remove a queue:
memcached_delete(memc, queue_name);
3. To insert an element to a queue:
memcached_set(memc, queue_name, element_content);
Once the queue has already reached the length limit, it returns
the standard memcached "NOT STORED" exception code.
4. To read an element from a queue:
element_content = memcached_get(memc, queue_name);
Once the queue is already empty, it returns a standard memcached
"NOT FOUND" exception code.
5. To monitor the queues' state in a certain memcacheq server
(here we use a shell command to illustrate):
$ echo stats queue | nc
where the memcacheq daemon listens the 11211 port at A
typical instance of the output might be:
STAT bbsdetails 0 2000000
STAT bbslists 72 2000000
STAT comment 3 1234567
STAT done 0 1234567
STAT initial 25 1234567
STAT pagecat 5006 1234567
STAT preprocessed 10 500
where the second column in the listing specify the names of the queues,
the third column the current size of the queues, and the forth column
the size limit of the queues.
Note that the queue size is stored separately in contrast to real-time
counting, so it might be kinda out of sync if the daemon exits abnormally
and the -N option is specified when starting the daemon.
We usually use the command-line utilities as well as the high-level Perl 5
library provided by the Queue::Memcached::Buffered module opensourced here:
Below is the original memcacheq documentation in README:
MemcacheQ - Simple Queue Service over Memcache
* damn simple
* very fast
* multiple queue
* concurrent well
* memcache protocol compatible
Getting Started
See: <>
See: <>
Please take a look at 'ChangLog' file in the distribution, see what's new.
Only two commands are used to operate the queue:
**Append a message to the tail of queue**::
set <queue name> <flags> 0 <message_len>\r\n
<put your message body here>\r\n
**Note:** MQ will create a new queue automatically if your queue is not existed. The original 'expire time' field is ignored by server.
**Consume a message from the head of queue**::
get <queue name>\r\n
VALUE <queue name> <flags> <message_len>\r\n
<your message body will come here>\r\n
Assuming you are using PHP memcache<>::
/* connect to memcached server */
$memcache_obj = memcache_connect('memcacheq_host', 21201);
/* append a message to queue */
memcache_set($memcache_obj, 'demoqueue1', 'message body here', 0, 0);
/* consume a message from 'demoqueue1' */
memcache_get($memcache_obj, 'demoqueue1');
The message body is stored in Berkeley DB with fixed length. Any message that is shorter than the declared length will automatically be padded with space character (0x20 in the ASCII character set).
In Berkeley DB, as the official document refers,
"For the Queue access method, the record length must be enough smaller than the database's page size that at least one record plus the database page's metadata information can fit on each database page."
"The minimum page size is 512 bytes, the maximum page size is 64K bytes, and the page size must be a power-of-two."
So we have a limit on the message body size with a max of a bit less than *64K*.
Other tips
use 'stats queue' to see your current queues::
$ telnet 22201
Connected to localhost.
Escape character is '^]'.
stats queue
STAT test1
STAT test2
STAT test3
STAT test4
delete a queue::
$ telnet 22201
Connected to localhost.
Escape character is '^]'.
delete test1
'db_stat' a queue to see how many records now in::
$ cd <your queue dir>
$ /usr/local/BerkeleyDB.4.7/bin/db_stat -d test1
Mon Sep 22 20:25:56 2008 Local time
42253 Queue magic number
4 Queue version number
1024 Fixed-length record size
0x20 Fixed-length record pad
4096 Underlying database page size
131072 Underlying database extent size
100000 Number of records in the database
33334 Number of database pages
2048 Number of bytes free in database pages (99% ff)
1 First undeleted record
100001 Next available record number
MemcacheDB mailing list now hosts on Google Group:
* To subscribe to maillist, send email to
* To post to maillist, send email to
* To unsubscribe from maillist, send email to
Please report your bugs and issues to the Maillist.
Last updated by Steve Chu<>: 09/22/2008