Skip to content

Commit

Permalink
Fixes based on feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
David Clark committed Aug 27, 2018
1 parent 29d4471 commit 2846686
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 2 deletions.
1 change: 1 addition & 0 deletions docs/building-a-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ When creating your watchbot stacks with the `watchbot.template()` method, you no
**alarmPeriods** | Use this parameter to control the duration that the SQS queue must be over the message threshold before triggering an alarm. You specify the number of 5-minute periods before an alarm is triggered. The default is 24 periods, or 2 hours. This parameter can be provided as either a number or a reference, i.e. `{"Ref": "..."}`. | String/Ref | No | 24
**alarmThreshold** | Watchbot creates a CloudWatch alarm that will go off when there have been too many messages in SQS for a certain period of time. Use this parameter to adjust the Threshold number of messages to trigger the alarm. This parameter can be provided as either a number or a reference, i.e. `{"Ref": "..."}`. | Number/Ref | No | 40
**errorThreshold** | Watchbot creates a CloudWatch alarm that will fire if there have been more than this number of failed worker invocations in a 60 second period. This parameter can be provided as either a number or a reference, i.e. `{"Ref": "..."}`. | Number/Ref | No | 10
**deadletterThreshold** | Use this parameter to control the number of times a message is delivered to the source queue before being moved to the dead-letter queue. This parameter can be provided as either a number or a reference, i.e. `{"Ref": "..."}`. | Number/Ref | No | 10
**dashboard** | Watchbot creates a Cloudwatch Dashboard called `<cloudformation-stack>-<region>`. If running in cn-north-1, this may need to be disabled | Boolean | No | `true`
**fifo** | Whether you want Watchbot's SQS queue to be first-in-first-out (FIFO). By default, Watchbot creates a standard SQS queue, in which the order of jobs is not guaranteed to match the order of messages. If your program requires more precise ordering and the limitations of a FIFO queue will be acceptable, set this option to `true`. Learn more in ["Using a FIFO queue"](./using-a-fifo-queue.md) | Boolean | No | `false`

Expand Down
14 changes: 12 additions & 2 deletions docs/using-a-fifo-queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,24 @@ By default, Watchbot creates a standard SQS queue, in which the order of jobs is

To use a FIFO queue, set the template option `fifo: true`.

A FIFO queue behaves differently and has some limitations, so you should read the AWS documentation and the details below if you are considering the switch.
A FIFO queue behaves differently and has some limitations, so you should read [the AWS documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html) and the details below if you are considering the switch.

### Triggering FIFO workers

With a standard queue, you trigger a Watchbot worker by sending an SNS message to the watcher's topic. With a FIFO queue, you will send a message directly to the watcher's FIFO queue, instead. Make sure you have read about the AWS documentation about the expected for these messages (for example, you need to include a `MessageGroupId`).
With a standard queue, you trigger a Watchbot worker by sending an SNS message to the watcher's topic. With a FIFO queue, you send a message directly to the watcher's FIFO queue, instead. Make sure you have read [the AWS documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html) about the expected format of FIFO queue messages (for example, you need to include a `MessageGroupId`).

### Scaling FIFO workers

[Scaling behavior](./scaling-in-watchbot.md) in Watchbot is based on the total number of messages in the queue. If there are 50 messages, concurrent processing scales up to 50 (or the configured `maxSize`, if lower). In a FIFO queue, this scaling will not always make sense. For example, if your queue includes 50 messages but they only span 2 `MessageGroupId`s, then you will only be able to process 2 messages at a time and 48 worker machines will sit idle.

For this reason, it's important that you plan for the number of `MessageGroupId`s you will use and set the template option `maxSize` accordingly. If you are only going to use 2 `MessageGroupId`s, you should set `maxSize: 2`.

### Dead-letter queues and FIFO

[The AWS documentation for dead-letter queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) suggests the following:

> Don’t use a dead-letter queue with a FIFO queue if you don’t want to break the exact order of messages or operations.
If exact ordering is absolutely necessary for your program, you may prefer that the whole system stall if one message fails, instead of dropping the failed message into a dead-letter queue and moving on to the next one.

Currently there is no way to turn off Watchbot's dead-letter queue. If you need this feature, please submit an issue or pull request.

0 comments on commit 2846686

Please sign in to comment.