Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
include
server
site_scons
.gitignore
LICENSE
NOTICE
README
SConstruct

README

Cloudkick QMQ
=====

Quick Message Queue (QMQ) is a light weight queueing system designed to be 
robust and quick.  It likely isn't ideal if you want to track financial 
transactions, but since most Queuing systems were first developed for financial
systems, and later brought to the wider communities, their priorities are 
different than QMQ.


Feature Goals
=====
  - Bounded Queues, so if you have a problem client, kill them, rather than the
    the whole queue.  When a client subscribes, they describe how they want 
    the queue to be buffered. (x bytes network buffer or N messages maximum)
  - Multi-threaded, with clients sticky to a thread. Similar in a concurrency 
    model to Apache Traffic Server.
  - Be able to implement AMQP using the primitives provided, but not a short
    term goal.
  - Multiple wire formats:
    - Pure JSON over HTTP
    - STOMP over WebSockets
    - STOMP over HTTP/long polling etc.
    - Avro over HTTP
    - AMQP
    - Support for SSL/TLS layers in all protocols.
  - Multi-node replication and fault tolerance.
    - Optional disk bound, commit log style queues, 
  - Equivalents to Orbited for usage from browsers (ideally direct integration 
    with the Google Closure library)
  - Internal to QMQ, Write most things in a C-style, but do use a few basics from STL:
      - std::string
      - std::vector
      - other basic data structures when needed.
  - All external facing libraries and APIs will only provide a C interface and 
    data structure.

Replication Thoughts
=====

The primary goal is for a masterless system, in which the queue can always
flow.  To accomplish this, each QMQ node treats any of its peers who need data
as just another client. 

A concrete example:
  - QMQ A has:
      Client 1, who publishes messages to "foo".
      Client 2, who subscribes to "foo"
  - QMQ B has:
      Client 3 who publishes messages to "foo"
      Client 4 who subscribes to "foo"
  - QMQ C has:
      Client 5 who subscribes to "foo".

"foo" was created as a fanout queue.

QMQ A will send subscribe messages to B and C.
QMQ B will send subscribe messages to C and A.
QMQ C will send subscribe messages to A and B.

When Client 1 Publishes a message to "foo":
  - QMQ A receives it.
  - A forwards it on to:
    - Client 2
    - B
    - C
  - B forwards it to client 4.
  - C forwards it to client 5.

In the long run, you would want to redesign this to create a hierarchy, but 
the mesh will work for <8 node clusters, which is all you will ever need (tm).

For early versions, I believe we will just use a hard coded list of peers,
as doing peer discovery and gossip adds complication.  Ideally in the long
run, I think we would implement it by having a "system.peers" queue, which
everyone subscribes to, and peers would periodically send messages to.
Something went wrong with that request. Please try again.