ooooo ooo .oooooo..o .oooooo. `888b. `8' d8P' `Y8 d8P' `Y8b 8 `88b. 8 Y88bo. 888 888 8 `88b. 8 `"Y8888o. 888 888 8 `88b.8 `"Y88b 888 888 8 `888 oo .d8P `88b d88b o8o `8 8""88888P' `Y8bood8P'Ybd'
NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day.
Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and Python libraries are available out of the box and, if you're interested in building your own client, there's a protocol spec (see client libraries).
NOTE: master is our development branch and may not be stable at all times.
- provide easy topology solutions that enable high-availability and eliminate SPOFs
- address the need for stronger message delivery guarantees
- bound the memory footprint of a single process (by persisting some messages to disk)
- greatly simplify configuration requirements for producers and consumers
- provide a straightforward upgrade path
- improve efficiency
- nsq Go (official)
- pynsq Python (official) pypi
- libnsq C
- nsq-java Java
- nodensq Node.js npm
- nsqphp PHP
- ruby_nsq Ruby rubygems
NSQ is composed of the following individual components, each with their own README:
- nsqd is the daemon that receives, buffers, and delivers messages to clients.
- nsqlookupd is the daemon that manages topology information
- nsqadmin is the web UI to view message statistics and perform administrative tasks
- nsq is a go package for writing
As well as the following documents in the docs directory:
- Design - in-depth overview of how and why
- Patterns - implementation solutions for various use cases
- Protocol Spec - technical details for the
DISCLAIMER: Please keep in mind that NSQ is designed to be used in a distributed fashion. Single node performance is important, but not the end-all-be-all of what we're looking to achieve. Also, benchmarks are stupid, but here's a few anyway to ignite the flame:
On a 2012 MacBook Air i7 2ghz (
go tip 8bbc0bdf832e) single publisher, single consumer:
$ ./nsqd --mem-queue-size=100000 $ ./bench_writer --num=100000 2013/01/06 17:24:38 duration: 310.575793ms - 61.413mb/s - 321982.596ops/s - 3.106us/op $ ./bench_reader --num=100000 2013/01/06 17:25:47 duration: 672.059087ms - 28.381mb/s - 148796.441ops/s - 6.721us/op
The following steps will run NSQ on your local machine and walk through publishing, consuming, and archiving messages to disk.
- follow the instructions in the INSTALLING doc (or download a binary release).
in one shell, start
in another shell, start
$ nsqd --lookupd-tcp-address=127.0.0.1:4160
in another shell, start
$ nsqadmin --lookupd-http-address=127.0.0.1:4161
publish an initial message (creates the topic in the cluster, too):
$ curl -d 'hello world 1' 'http://127.0.0.1:4151/put?topic=test'
finally, in another shell, start
$ nsq_to_file --topic=test --output-dir=/tmp --lookupd-http-address=127.0.0.1:4161
publish more messages to
$ curl -d 'hello world 2' 'http://127.0.0.1:4151/put?topic=test' $ curl -d 'hello world 3' 'http://127.0.0.1:4151/put?topic=test'
to verify things worked as expected, in a web browser open
http://127.0.0.1:4171/to view the
nsqadminUI and see statistics. Also, check the contents of the log files (
test.*.log) written to
The important lesson here is that
nsq_to_file (the client) is not explicitly told where the
topic is produced, it retrieves this information from
nsqlookupd and, despite the timing of the
connection, no messages are lost.
- Dan Frank (@danielhfrank)
- Pierce Lopez (@ploxiln)
- Will McCutchen (@mccutchen)
- Micha Gorelick (@mynameisfiber)
- Jay Ridgeway (@jayridge)