Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
A realtime distributed messaging platform
Go Other

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
docs
examples
nsq
nsqadmin
nsqd
nsqlookupd
pynsq
util
.gitignore
.travis.yml
CONTRIBUTING.md
ChangeLog.md
INSTALLING.md
LICENSE
Makefile
README.md
dist.sh
fmt.sh
test.sh

README.md

NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day.

It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and Python libraries are available out of the box and, if you're interested in building your own client, there's a protocol spec (see client libraries).

The latest stable release is 0.2.15. We publish binary releases for linux and darwin.

NOTE: master is our development branch and may not be stable at all times.

Build Status

Why?

NSQ was built as a successor to simplequeue (part of simplehttp) and as such was designed to (in no particular order):

  • provide easy topology solutions that enable high-availability and eliminate SPOFs
  • address the need for stronger message delivery guarantees
  • bound the memory footprint of a single process (by persisting some messages to disk)
  • greatly simplify configuration requirements for producers and consumers
  • provide a straightforward upgrade path
  • improve efficiency

If you're interested in more of the design, history, and evolution please read our design doc or blog post.

Client Libraries

Additional Documentation

NSQ is composed of the following individual components:

  • nsqd is the daemon that receives, buffers, and delivers messages to clients.
  • nsqlookupd is the daemon that manages topology information
  • nsqadmin is the web UI to view message statistics and perform administrative tasks
  • nsq is a go package for writing nsqd clients

There is also a protocol spec.

Getting Started

The following steps will run NSQ on your local machine and walk through publishing, consuming, and archiving messages to disk.

  1. follow the instructions in the INSTALLING doc (or download a binary release).
  2. in one shell, start nsqlookupd:

    $ nsqlookupd
    
  3. in another shell, start nsqd:

    $ nsqd --lookupd-tcp-address=127.0.0.1:4160
    
  4. in another shell, start nsqadmin:

    $ nsqadmin --lookupd-http-address=127.0.0.1:4161
    
  5. finally, in another shell, start nsq_to_file:

    $ nsq_to_file --topic=test --output-dir=/tmp --lookupd-http-address=127.0.0.1:4161
    
  6. publish some messages to nsqd:

    $ curl -d 'hello world 1' 'http://127.0.0.1:4151/put?topic=test'
    $ curl -d 'hello world 2' 'http://127.0.0.1:4151/put?topic=test'
    $ curl -d 'hello world 3' 'http://127.0.0.1:4151/put?topic=test'
    
  7. to verify things worked as expected, in a web browser open http://127.0.0.1:4171/ to view the nsqadmin UI and see statistics. Also, check the contents of the log files (test.*.log) written to /tmp.

The important lesson here is that nsq_to_file (the client) is not explicitly told where the test topic is produced, it retrieves this information from nsqlookupd.

Authors

NSQ was designed and developed by Matt Reiferson (@imsnakes) and Jehiah Czebotar (@jehiah) but wouldn't have been possible without the support of bitly:

Contributors

Something went wrong with that request. Please try again.