EZQ provides a simple configuration-style interface to processing messages
in an Amazon SQS queue. The application you use to process messages (your
*process_command*) doesn't have to know anything about AWS. If you set up
an AMI containing your *process_command*, and set that AMI to run EZQ on
startup, you have instant scalability for a message processing type
application.
EZQ wraps up all the logic of:
-
polling and retrieving messages from a queue
-
optionally performing a set of pre-processing operations that can be encoded into the messages, such as
- fetching one or more files from Amazon S3
- fetching the contents of one or more uris
- decompressing the message body with zlib
-
executing the
process_commandto process the message (or downloaded file) -
optionally storing job results in Amazon S3, or posting them to a result queue
-
optionally posting a "Job complete" message to a result queue
When run on an Amazon EC2 instance, EQZ can also be setup to stop or terminate the instance when no new messages have arrived in certain period of time.
The full list of configurable options is detailed in {file:Processor_Config_Details.md}.
{file:Processor_Message_Details.md} explains in more depth the structure of messages and how to embed pre-processing directives.
{file:Processor_Example_Use.md} shows an example application using EZQ::Processor. This shows how "it all fits together."
Once you've setup an appropriate configuration file for EZQ, running a processor is as simple as
$ Processor.rb -c my_config_file.yml
It's just a matter of getting your configuration file setup correctly....