Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Reliable and Highly Available Message Queue based on ZooKeeper (Prototype)
Python Shell
branch: master

Merge pull request #4 from forsberg/ctime_is_milliseconds

Fix bug where removal of empty items would not happen due to assumption ...
latest commit 010c26ed90
@andreisavu authored
Failed to load latest commit information.
experiments added experimental code for a queue that does retry on connection loss
lib.linux-i686-2.6
lib.linux-x86_64-2.6 added precompiled libraries for python
test_5_trunk fixed a critical error and improved the faul injector. the test works…
.gitignore basic producer and basic consumer ready
LICENSE.txt updated readme and license
README Edited README via GitHub
check_all.py
priority.py
put_all.py
test_sequential.py get configuration from current environment
zkmq.py Merge pull request #4 from forsberg/ctime_is_milliseconds

README

Message Queue based on ZooKeeper
================================

NOTICE: This is work in progress and it's not production ready (more like a proof of concept)

NOTICE: It should be easy to rewrite the code in Java by following the same concepts.

Objectives
----------

Build a highly available & reliable message queue. The main focus is not on perfomance! 

This queue should be able to survive hardware and software upgrades and failures 
with no downtime for years :) 

The queue should be accesible using both the default zookeeper binary protocol
and the zookeeper rest interface. 

ZooKeeper Data Tree Structure
-----------------------------

<chroot-prefix OR />/
  /queue
    /items/
      /item-000-1
      /item-050-2
      ...
    /partial        - items reserved but not marked as done (consumer failure)
      /item-1
      /item-2
      ...
    /consumers
      /consumer-1
        /active     - ephemeral znode
        /item
      /consumer-2
        /active     - ephemeral
        /item

ZooKeeper Availability
----------------------

A ZooKeeper cluster with N servers will still be available if CEIL(N/2)-1 nodes fail!

E.g. if you run a cluster with five nodes you can take one offline for maintenance
anytime and you will still be able to handle an unexpected failure.

Requirements
------------

Tested with Python 2.6
and zkpython 3.4.0--1, built on 02/19/2011 14:52 GMT

You also need simplejson for the rest interface

How to install zkpython?
------------------------

Fetch the latest sources from the trunk and run:

$ apt-get install sun-java6-jdk libtool libcppunit-dev automake build-essential ant

$ cd zookeeper
$ ant compile compile_jute
$ cd src/c/
$ autoreconf -if
$ ./configure
$ make && sudo make install
$ cd ../contrib/zkpython
$ ant compile
$ sudo ant install

After all this you should be able to do:

$ python -c "import zookeeper;print zookeeper.__version__"
3.4.0

Fault Injection Testing Scenarios
---------------------------------

I'm not going to test the producer and the consumer code. I'm trying to
to test the availability and reliability of the ZooKeeper quorum when used
for storing the queue data.

Start a ZooKeeper quorum and the fault injector:

$ cd test_5_trunk
$ ./clean.sh
$ ./start.sh
$ python fail.py leader

Push 500 elements:

$ ./put_all.py 

or 

$ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./put_all.py

Check 500 elements (also checks that the sequence is strictly increasing):

$ ./check_all.py 

or 

$ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./check_all.py

It should work like a charm! :) Keep in mind that you are not going to experience 
this amount of critical failures in production.

License
-------

Apache License Version 2.0

Resources
---------

Inspired by this article wrote by Henry Robinson on the Cloudera blog:

http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/

Something went wrong with that request. Please try again.