Skip to content
RabbitIO is used to move data in and out of RabbitMQ
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github
cmd Improving error handling through more testing (#12) Apr 2, 2018
file
rmq
.gitignore
.travis.yml
Gopkg.lock
Gopkg.toml
LICENSE
Makefile Improving error handling through more testing (#12) Apr 2, 2018
README.md
docker-compose.yml
main.go

README.md

RabbitIO

Build Status

RabbitIO is a tool to backup and restore RabbitMQ messages.

To learn more about common problems with queue management and our reasoning and design principles when building RabbitIO, please read this blog post.

Typical use case

A typical use case is to consume all messages from a RabbitMQ deadletter queue to your local dev machine, do some analysis of the messages, and possibly push (some of) them back to an exchange for reprocessing.

Installing

Download binary

Pick your binary from Releases and download, in addition you'll need to set the binary to be executable.

Example with linux-amd64 and version v0.5.4:

wget https://github.com/meltwater/rabbitio/releases/download/v0.5.4/rabbitio-v0.5.4-linux-amd64 -O rabbitio
chmod 755 rabbitio

Using go get

go get -u github.com/meltwater/rabbitio

How to use RabbitIO

After installing rabbitio, you can quickly test out rabbitio by using docker-compose.

Included is a docker-compose file to set up local rabbitmq.

cd $GOPATH/src/github.com/meltwater
docker-compose up -d

Go to your now running local rabbit and create example exchange rabbitio-exchange and queue rabbitio-queue. Then bind the queue to the exchange.

Publish your first message

echo "My first message" > message # write a message into a file
tar cfz message.tgz message # create a tarball containing this message
rabbitio in -e rabbitio-exchange -q rabbitio-queue -f message.tgz

This will publish your first message into rabbitio-exchange and you'll see your message in the queue rabbitio-queue

Consume your first message

$ mkdir data
$ rabbitio out -e rabbitio-exchange -q rabbitio-queue -d data/
2018/03/15 15:37:35 RabbitMQ connected: amqp://guest:guest@localhost:5672/
2018/03/15 15:37:35 Bind to Exchange: "rabbitio-exchange" and Queue: "rabbitio-queue", Messaging waiting: 1
^C Interruption, saving last memory bits..
2018/03/15 15:37:45 Wrote 203 bytes to data/1_messages_1.tgz
2018/03/15 15:37:45 tarball writer closing

We interrupt when the queue is empty by directly using a combination of CTRL + C once. This will save the last bits and ack the message.

Detailed Usage

$ rabbitio
Rabbit IO will help backup and restore your messages in RabbitMQ

Usage:
  rabbitio [command]

Available Commands:
  help        Help about any command
  in          Publishes documents from tarballs into RabbitMQ exchange
  out         Consumes data out from RabbitMQ and stores to tarballs
  version     Prints the version of Rabbit IO

Flags:
  -e, --exchange string     Exchange to connect to
  -h, --help                help for rabbitio
  -p, --prefetch int        Prefetch for batches (default 100)
  -q, --queue string        Queue to connect to
  -r, --routingkey string   Routing Key, if specified will override tarball routing key configuration (default "#")
  -t, --tag string          AMQP Client Tag (default "Rabbit IO Connector ")
  -u, --uri string          AMQP URI, uri to for instance RabbitMQ (default "amqp://guest:guest@localhost:5672/")

Use "rabbitio [command] --help" for more information about a command.

AMQP Headers and Routing Key

When you read messages from a queue, the headers as well as the routing key will be saved as metadata in the tarballs, utilizing what in tar is called PAX Records. This is helpful if you one day want to replay the data back into the original queue, while keeping the attributes that belong to the message.

Header Format AMQP Headers Tar PAX Records
Format Translation map[String] Bool, Integer, String, Float map[String] String
Body Type Bytes Bytes

The tar metadata can be accessed using pax:

pax -r -zf 1_message_100.tgz

This will output the messages and in addition a PaxHeaders.0 directory containing identical filenames as the messages, enabling access of the metadata. Right now there is no way to selectively publish only certain messages containing a header value, this is a planned feature.

Contributing

If you plan to work on rabbitio you will need Golang. PRs are welcome as well as implementation discussions.

Clone rabbitio

mkdir -p $GOPATH/src/github.com/meltwater
cd $GOPATH/src/github.com/meltwater
git clone git@github.com:meltwater/rabbitio.git

Building

cd rabbitio
make && make build

Maintainers

For any bug reports or change requests, please create a GitHub issue or submit a PR.

Also feel free to drop a line to the maintainers:

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.