storm-amqp-spout: AMQP input source for Storm
storm-amqp-spout allows a Storm topology to consume an AMQP exchange as an input source. It currently provides one class:
an implementation of
backtype.storm.topology.IRichSpoutthat connects to an AMQP broker and consumes the messages published to a specified AMQP exchange.
The Javadocs can be found at http://code.rapportive.com/storm-amqp-spout.
To produce a jar:
$ mvn package
To install in your local Maven repository:
$ mvn install
To use in your
<project> <!-- ... --> <dependencies> <!-- ... --> <dependency> <groupId>com.rapportive</groupId> <artifactId>storm-amqp-spout</artifactId> <version>0.0.1</version> </dependency> <!-- ... --> </dependencies> <!-- ... --> </project>
This is very early software. It may break and the API is liable to change completely between releases. Pull requests, patches and bug reports are very welcome.
This should not currently be used where guaranteed message processing is required, because of two limitations:
Uses a temporary queue to bind to the specified exchange when the topology calls
open()on the spout, so it will only receive messages published to the exchange after the call to
open(), and if the spout worker restarts or the topology is killed, it will not receive any messages published while the worker or topology is down.
Currently auto-acks all consumed messages with the AMQP broker, and does not implement Storm's reliability API, so if processing a message fails it will simply be discarded.
Limitation 1 also means this spout cannot currently be distributed among multiple workers (each worker gets its own exclusive queue, so multiple workers would each receive their own copy of every message).
Improvements are planned to overcome both these limitations and support guaranteed message processing, distributed across any number of workers. These improvements may require API changes (e.g. to specify the name of an existing queue to consume, rather than an exchange to bind to).
AMQPSpout has been tested with RabbitMQ. It should probably work with other