By setting the Reduce parameter to true, Watchbot will be capable of helping track the
progress of distributed map-reduce operations. This is useful if your stack performs
a bunch of individual jobs that need to be "rolled up" into a final output of some sort.
Generally, a reduce-enabled Watchbot stack should be built in order to process three types of messages: one type that kicks of a map-reduce operation, one type that processes individual parts, and another type that performs the reduce or "roll up" operation.
Your code should include all the logic required to interpret these different types of messages. The flow of messages will generally be as follows:
- You (or some AWS resource) sends the initial message to your Watchbot stack's SNS topic. When your code receives this message, it should:
- determine how the work will be split across multiple parts
- generate an identifier for the entire map-reduce operation
- report to Watchbot the identifier, and the number of parts
- send SNS messages to Watchbot's SNS topic for each part, providing the identifier
for the operation, and the part number. Part numbers start a
1and increase up to the total number of parts.
- Watchbot will receive the "work" jobs that were sent by your initial message processor. When your code receives these messages, it should:
- perform appropriate processing
- once processing is complete, report the identifier and the part number of the job to Watchbot. In response, Watchbot will inform your code as to whether or not all the parts in the map-reduce operation are completed.
- if the worker receives the notification that all parts are complete, the worker should send a single message to Watchbot's SNS topic to trigger the reduce step
- Upon receiving the reduce message, your code should take any appropriate roll-up action.
🚨 These docs are for legacy watchbot. There is currently no docs for this starting Watchbot 10.
watchbot-progress is a CLI command that is available to use on a reduce-enabled
stack. This is one mechanism by which you can report progress to Watchbot as part
ofthe above messaging flow.
For usage examples and additional documentation, see watchbot-progress.
Install Watchbot globally as part of your worker's Dockerfile to gain access to the CLI command on your workers at runtime:
RUN npm install -g watchbot
$ watchbot-progress <command> <job-id> [options]
Note that by default, workers in reduce-enabled Watchbot stacks will have the $ProgressTable
environment variable set automatically. For more information on this command, see
🚨 These docs are for legacy watchbot. There is currently no docs for this starting Watchbot 10.
A JavaScript module is also available as a mechanism for progress reporting.
var progress = require('@mapbox/watchbot').progress();