Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 114 lines (87 sloc) 4.509 kB
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
1 # storm-amqp-spout: AMQP input source for Storm #
2
b369353 @samstokes Link to maintained forks
samstokes authored
3 ## This is out of date!
4
5 This is a rather old version of the spout that is not API-compatible with newer
6 versions of Storm. Various people have forked it to bring it up to date and add
7 features:
8
9 * https://github.com/Xorlev/storm-amqp-spout
10 * https://github.com/dkincaid/storm-amqp-spout
11
12 If you want your fork added to this list, or to contribute a description of
13 your fork, please email sam@rapportive.com.
14
15 ## Description
16
c9a5544 @samstokes Make queue declaration configurable; now supports reliable processing.
samstokes authored
17 storm-amqp-spout allows a [Storm][] topology to consume an AMQP queue as an
18 input source. It currently provides:
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
19
fa9dc6a @samstokes Switch docs to Markdown reference-style links for readability
samstokes authored
20 * <tt>[AMQPSpout][]</tt>: an implementation of
21 [`backtype.storm.topology.IRichSpout`][IRichSpout] that connects to an AMQP
c9a5544 @samstokes Make queue declaration configurable; now supports reliable processing.
samstokes authored
22 broker, consumes the messages routed to a specified AMQP queue and emits them
23 as Storm tuples.
24 * <tt>[QueueDeclaration][]</tt>: an interface that encapsulates declaring an
25 AMQP queue and setting up any exchange bindings it requires, used by
26 AMQPSpout to set up the queue to consume.
27 * <tt>[ExclusiveQueueWithBinding][]</tt>: a QueueDeclaration suitable for
28 prototyping and one-off analytics use cases.
29 * <tt>[SharedQueueWithBinding][]</tt>: a QueueDeclaration suitable for
30 production use cases needing guaranteed processing.
fa9dc6a @samstokes Switch docs to Markdown reference-style links for readability
samstokes authored
31
32 You'll need to provide a [Scheme][] to tell AMQPSpout how to interpret the
33 messages and turn them into Storm tuples. See e.g. [storm-json][] if your
2bf479a @samstokes Consistent formatting in docs
samstokes authored
34 messages are JSON.
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
35
36 ## Documentation ##
37
38 The Javadocs can be found at [http://code.rapportive.com/storm-amqp-spout]().
39
40 ## Usage ##
41
42 To produce a jar:
43
44 $ mvn package
45
46 To install in your local Maven repository:
47
48 $ mvn install
49
50 To use in your `pom.xml`:
51
52 ```xml
53 <project>
54 <!-- ... -->
55 <dependencies>
56 <!-- ... -->
57 <dependency>
58 <groupId>com.rapportive</groupId>
59 <artifactId>storm-amqp-spout</artifactId>
552e570 @samstokes Readme tweaks
samstokes authored
60 <version>0.1.1</version>
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
61 </dependency>
62 <!-- ... -->
63 </dependencies>
64 <!-- ... -->
65 </project>
66 ```
67
68 ## Caveats ##
69
cecc796 @samstokes Be less cautious in README
samstokes authored
70 This is early software. It has been used at production volumes, but not yet
71 for business-critical use cases. It may break and the API is liable to change
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
72 completely between releases. Pull requests, patches and bug reports are very
73 welcome.
74
c9a5544 @samstokes Make queue declaration configurable; now supports reliable processing.
samstokes authored
75 **N.B.** if you need to guarantee all messages are reliably processed, you
76 should have AMQPSpout consume from a queue that is *not* set as 'exclusive' or
77 'auto-delete': otherwise if the spout task crashes or is restarted, the queue
78 will be deleted and any messages in it lost, as will any messages published
79 while the task remains down. See [SharedQueueWithBinding][] to declare a
80 shared queue that allows for guaranteed processing. (For prototyping, an
81 [ExclusiveQueueWithBinding][] may be simpler to manage.)
d3661c4 @samstokes Readme and bump to 0.0.1
samstokes authored
82
2838727 @samstokes Discuss caveats when a message fails to be processed.
samstokes authored
83 This does not currently handle malformed messages very well: the spout worker
84 will crash if the provided [Scheme][] fails to deserialise a message.
85
86 This does not currently support retrying messages in the event of transient
87 failure to process: any message which the topology fails to process will simply
88 be dropped. This is to prevent infinite redelivery in the event of
89 non-transient failures (e.g. malformed messages, though see previous caveat!).
90 This will probably be made configurable in a future release.
91
2852852 @samstokes Readability tweak to README
samstokes authored
92 ## Compatibility ##
93
552e570 @samstokes Readme tweaks
samstokes authored
94 AMQPSpout has been tested with RabbitMQ 2.3.1, 2.6.1 and 2.7.0. It should probably work with other
cb55bb4 @samstokes More detail on which AMQP brokers I've tested it with.
samstokes authored
95 versions and other AMQP brokers.
fa9dc6a @samstokes Switch docs to Markdown reference-style links for readability
samstokes authored
96
97
98 [Storm]: <https://github.com/nathanmarz/storm>
99 "Storm project homepage"
100 [IRichSpout]: <http://nathanmarz.github.com/storm/doc/backtype/storm/topology/IRichSpout.html>
101 "Javadoc for backtype.storm.topology.IRichSpout"
102 [Scheme]: <http://nathanmarz.github.com/storm/doc/backtype/storm/spout/Scheme.html>
103 "Javadoc for backtype.storm.spout.Scheme"
104 [AMQPSpout]: <http://code.rapportive.com/storm-amqp-spout/doc/com/rapportive/storm/spout/AMQPSpout.html>
105 "Javadoc for AMQPSpout"
c9a5544 @samstokes Make queue declaration configurable; now supports reliable processing.
samstokes authored
106 [QueueDeclaration]: <http://code.rapportive.com/storm-amqp-spout/doc/com/rapportive/storm/amqp/QueueDeclaration.html>
107 "Javadoc for QueueDeclaration"
108 [ExclusiveQueueWithBinding]: <http://code.rapportive.com/storm-amqp-spout/doc/com/rapportive/storm/amqp/ExclusiveQueueWithBinding.html>
109 "Javadoc for ExclusiveQueueWithBinding"
110 [SharedQueueWithBinding]: <http://code.rapportive.com/storm-amqp-spout/doc/com/rapportive/storm/amqp/SharedQueueWithBinding.html>
111 "Javadoc for SharedQueueWithBinding"
fa9dc6a @samstokes Switch docs to Markdown reference-style links for readability
samstokes authored
112 [storm-json]: <https://github.com/rapportive-oss/storm-json>
113 "JSON {,de}serialisation support for Storm"
Something went wrong with that request. Please try again.