Heimdall-Burstable-SES 📊📧

About the Project

Main Goals

• Send emails without exceeding the AWS SES send rate limit. 📧
• Low cost 💲✂️
  • Pay as you use
  • Scales to zero

Build With

• AWS SAM
• AWS SES, AWS Lambda, AWS SQS
• Node.js, TypeScript, esbuild
• prettier, eslint, jest

Problem description

AWS SES has a soft send rate limit, which is the maximum number of emails that Amazon SES can accept from your account each second. You can exceed this quota for short bursts, but not for sustained periods of time (source). If you send more than that, you will get a Throttling error. This is a problem if you want to send a lot of emails at once, for example, if you want to send a newsletter to all your users.

Typical send rates:

• 1 email per second for AWS SES sandbox accounts (source)
• ~50 emails per second for AWS SES production accounts (source)
• SendGrid has a default limit of 100 emails per second (600 emails/minute) (source).

There are multiple ways to solve this problem, but the most common one is to use a queue. You can send the emails to a queue and then process them one by one. See:

• How to Automatically Prevent Email Throttling when Reaching Concurrency Limit
• Sending emails by SES as fast as possible without exceeding the rate limit

Leaky bucket principle. Source: How to Automatically Prevent Email Throttling when Reaching Concurrency Limit

These solutions are great in general, but doesn't seem to be optimal for our use case and might be a little bit outdated and therefore overly complex..

Our Solution

AWS recently (12 JAN 2023) introduced the maximum concurrency of AWS Lambda functions when using Amazon SQS as an event source feature, which seems to be a perfect fit for our use case. This feature allows us to set a maximum concurrency for a Lambda function that is triggered by a SQS queue. This means that we can set a maximum concurrency for our Lambda function, which will process the emails one by one. This will allow us to send a lot of emails without exceeding the SES send rate limit.

Source: Introducing maximum concurrency of AWS Lambda functions when using Amazon SQS as an event source

By setting the maximum concurrency equal to our SES send rate limit and making sure that each lambda function runs at least 1s, we are able to send emails without exceeding the SES send rate limit! 🙌🎉

Cost Estimation

Cost estimation for emails per month

• 1k: ~0.10$
• 10k: ~1.03$
• 100k: ~10.27$

Getting Started 🚀

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

For this project you will need:

• Node.js and yarn installed.

Optional:

• Visual Studio Code with ESLint and Prettier extensions installed.
• SAM CLI installed
  • Docker installed?
  • AWS CLI installed?

Usage

• Clone the repo

git clone

• Install NPM packages

yarn install

• Run the unit tests

yarn test

Deploying to AWS

The application will be automatically deployed by the cd pipeline for each PR and on every push to the main branch.

Contributing 🤝

To get code to production:

1. Create a new branch or fork from latest main.
2. Make your changes.
3. Push your branch to GitHub and open a pull request.
4. The pipeline will automatically run the CI checks (typecheck, prettier, linting & unit tests) and will create a new feature deployment once the deploy label is added to the PR by the code owner.
5. Test the feature deployment.
6. The PR will be reviewed and merged to main and the pipeline will automatically create a new production deployment.

Forking

If you want to fork this project and run the same pipeline, you will need to adjust the OpenID Connect role.

see: Configuring OpenID Connect in Amazon Web Services