Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create bounce collection and alarms #131

Merged
merged 53 commits into from
Aug 18, 2020
Merged

feat: create bounce collection and alarms #131

merged 53 commits into from
Aug 18, 2020

Conversation

mantariksh
Copy link
Contributor

@mantariksh mantariksh commented Aug 13, 2020

Problem

We do not have a reliable way of being alerted to critical bounces, i.e. email-mode forms where emails to all recipients are bouncing, resulting in lost responses. The current system (CloudWatch alarm if admin responses cross a certain threshold) does not discriminate by critical/non-critical bounces, and also allows critical bounces for low-volume forms to slip under the radar.

Closes #5

Solution

Create a Bounce collection to keep track of bounces per form, and activate CloudWatch alarms only for critical bounces. The collection was necessary because we need the data to persist across bounce notifications, since bounce notifications can arrive piecemeal (e.g. if there are 2 email recipients, SES might tell us that each recipient bounced in 2 separate notifications). The collection also helps us keep track of whether alarms have already been sent so that we can control the rate of alarms.

Documents in this collection have a 30min TTL so that we receive alarms every 30min as long as critical bounces are still happening for a given form.

Tests

Before release

  • Submit a form with all valid email recipients. Check that in the Bounces collection, all the email addresses are present for that form ID and hasBounced is false for all of them.
  • Submit the same form as the previous test again. Check that the Bounces collection still only has 1 document for that form ID, i.e. a duplicate document is not created.
  • Add an invalid email recipient to that form and submit it again. Check that the Bounces collection still only has 1 document for that form ID, and hasBounced is set to true only for the invalid recipient.
  • For the same form, edit the recipient list such that it only has invalid emails. Check that the corresponding document in the Bounces collection is updated with the new recipient list, and hasBounced is set to true for all of them. Check CloudWatch alarms and ensure that the alarm called "Critical Bounces (Staging)" has triggered.
  • Bounce document expires 30min after initial creation.
  • BOUNCE_LIFE_SPAN env var has been added in production and set to 1800000 (number of ms in 30min).
  • Alarm has been created for critical bounces in production.

After release

  • Create index on expireAt in Bounce collection in production.
  • Update CloudWatch dashboard widgets based on new logging format.

docker-compose.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@liangyuanruo liangyuanruo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i made some suggestions so that the APIs can better reflect a domain-driven design.

@mantariksh mantariksh merged commit bb852aa into develop Aug 18, 2020
@karrui karrui mentioned this pull request Aug 25, 2020
@liangyuanruo liangyuanruo deleted the bounce-coll branch September 2, 2020 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Intelligent email bounce alarm
2 participants