No description, website, or topics provided.
JavaScript Lua Makefile
Latest commit 12bbf2e Dec 21, 2016 Gabriel Andrés Eisbruch Rollback body parser
Permalink
Failed to load latest commit information.
bin Fix on launcher Dec 15, 2016
examples move example to examples folder Mar 26, 2012
ext Fix health to not alert if local api is not returning node state Dec 15, 2016
lib
scripts Fixes to tasks replace Oct 14, 2015
test Increase timeout for slow porcessing machines Dec 16, 2016
.gitignore remove node modules Mar 27, 2012
.npmignore Initial Commit Mar 8, 2012
Makefile Change to use newrelic by default Dec 7, 2016
README.asciidoc Readmin Feb 28, 2013
create_db.sql
package.json Rollback body parser Dec 21, 2016

README.asciidoc

Big Queue

Big queue (BQ) is a multi-consumer topic system designed to be very simple and very scalable, the main idea of is that each topic can be consumed asynchronously by many consumer groups, we’ll call consumer group to a pool of process which will receive the same bunch of messages, using this method we can send a group of messages to a topic and each consumer group will receive this messages (like a kestrel fanout queue) then we can read it in a paralelized fashion using many process reading from the same consumer group

BQ is based on top of redis http://redis.io/ as their main storage (in memory) but could be implemented over any persistent system

One of the very atractive things of BQ is the idea of be asynchronous this enable us to create an http api (could be any other faster protocol if we want) that implements a similar Amazon SQS comunication pattern being very easy integrate and platform/languaje agnostic

Running it in your computer

Preparing the environment

    sudo make prepare_development

It could take many time

Starting bigqueue

   make run_development

Creating a topic

   curl -d'{"name":"test_topic"}' 127.0.0.1:8081/topics -H"Content-Type: application/json"

Creating a consumer

   curl -d'{"name":"test_consumer"}' 127.0.0.1:8081/topics/test_topic/consumers -H"Content-Type: application/json"

Reset a consumer

   curl -XPUT 127.0.0.1:8081/topics/test_topic/consumers/test_consumer -H"Content-Type: application/json"

Posting message

   curl -d'{"msg":{"test":"message"},"topics":["test_topic"]}' 127.0.0.1:8081/messages -H"Content-Type: application/json"

Reading message

   curl 127.0.0.1:8081/topics/test_topic/consumers/test_consumer/messages

Acking the message

Warning
the id used to delete the message is the recipientCallback
   curl 127.0.0.1:8081/topics/test_topic/consumers/test_consumer/messages/1 -XDELETE

Design

TODO

BQ Cluster

TODO

Rest Interface

The rest interface can be runned using the bigqueue-http command installed at the module install time, this command will run in default mode (using a pre instaled redis at default port), if you want to configure it to run in cluster mode you could use the config example at example/clusterconfig.js

If you want to run with the cluster configuration you should run

    ./node_modules/.bin/bigqueue-http ./node_modules/BigQueue/examples/clusterconfig.js

Monitoring

TODO

Testing

To run the test you should run:

    make test