Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Allows a Storm topology to consume an AMQP exchange as an input source.

tree: efc72b59d3

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 .settings boilerplate (Eclipse and Maven) September 24, 2011
Octocat-spinner-32 src
Octocat-spinner-32 .classpath
Octocat-spinner-32 .gitignore gitignore target/ September 25, 2011
Octocat-spinner-32 .project boilerplate (Eclipse and Maven) September 24, 2011
Octocat-spinner-32 LICENSE.MIT Add MIT license September 29, 2011
Octocat-spinner-32 README.markdown
Octocat-spinner-32 pom.xml
README.markdown

storm-amqp-spout: AMQP input source for Storm

storm-amqp-spout allows a Storm topology to consume an AMQP exchange as an input source. It currently provides one class:

You'll need to provide a Scheme to tell AMQPSpout how to interpret the messages and turn them into Storm tuples. See e.g. storm-json if your messages are JSON.

Documentation

The Javadocs can be found at http://code.rapportive.com/storm-amqp-spout.

Usage

To produce a jar:

$ mvn package

To install in your local Maven repository:

$ mvn install

To use in your pom.xml:

<project>
  <!-- ... -->
  <dependencies>
    <!-- ... -->
    <dependency>
      <groupId>com.rapportive</groupId>
      <artifactId>storm-amqp-spout</artifactId>
      <version>0.0.2-SNAPSHOT</version>
    </dependency>
    <!-- ... -->
  </dependencies>
  <!-- ... -->
</project>

Caveats

This is very early software. It may break and the API is liable to change completely between releases. Pull requests, patches and bug reports are very welcome.

This should not currently be used where guaranteed message processing is required, because of two limitations:

  1. Uses a temporary queue to bind to the specified exchange when the topology calls open() on the spout, so it will only receive messages published to the exchange after the call to open(), and if the spout worker restarts or the topology is killed, it will not receive any messages published while the worker or topology is down.

  2. Currently auto-acks all consumed messages with the AMQP broker, and does not implement Storm's reliability API, so if processing a message fails it will simply be discarded.

Limitation 1 also means this spout cannot currently be distributed among multiple workers (each worker gets its own exclusive queue, so multiple workers would each receive their own copy of every message).

Improvements are planned to overcome both these limitations and support guaranteed message processing, distributed across any number of workers. These improvements may require API changes (e.g. to specify the name of an existing queue to consume, rather than an exchange to bind to).

AMQPSpout has been tested with RabbitMQ 2.3.1 and 2.6.1. It should probably work with other versions and other AMQP brokers.

Something went wrong with that request. Please try again.