sqs-worker

Boilerplate for a Fargate container that processes an SQS queue.

Includes Terraform and Python code.

This repo uses asdf to manage the terraform CLI and the various other tools it depends upon. Completely optional.

Usage

To initialize the project:

make init

This will install the required tools and register a pre-commit hook.

To run the pre-commit checks:

make checks

To see a summary of the Terraform resource changes:

make summary

Autoscaling

This project includes a Lambda-based autoscaler that scales the ECS service based on the number of messages in the SQS queue. It runs on a 1-minute schedule and checks the queue attributes to determine if the service needs to be scaled up or down.

Monitoring

The project also includes a CloudWatch dashboard that provides visibility into the following metrics:

• Messages Queued
• Messages Processing
• Workers Desired
• Workers Running
• CPU Utilization
• Memory Utilization

Python App

The Python app that runs in the Fargate container is located in the app/ directory. It uses the boto3 library to receive messages from the SQS queue, process them, and delete them from the queue.

Deployment

The Python app can be built and deployed using the deploy.sh script in the app/ directory. This script logs in to the ECR registry, builds and pushes the Docker image, and then updates the ECS service to use the new image.

Terraform

Requirements

Name	Version
terraform	>= 1.0
archive	~> 2.2
aws	~> 5.0

Providers

Name	Version
archive	~> 2.2
aws	~> 5.0

Modules

Name	Source	Version
ecs_cluster	terraform-aws-modules/ecs/aws//modules/cluster	~> 5.6
ecs_service	terraform-aws-modules/ecs/aws//modules/service	~> 5.6

Resources

Name	Type
aws_cloudwatch_dashboard.main	resource
aws_cloudwatch_event_rule.queue_autoscaler	resource
aws_cloudwatch_event_target.queue_autoscaler	resource
aws_ecr_repository.main	resource
aws_iam_role.queue_autoscaler	resource
aws_iam_role_policy.queue_autoscaler_logs	resource
aws_lambda_alias.queue_autoscaler_alias	resource
aws_lambda_function.queue_autoscaler	resource
aws_lambda_permission.allow_cloudwatch_to_call_queue_autoscaler	resource
aws_sqs_queue.dlq	resource
aws_sqs_queue.main	resource
archive_file.queue_autoscaler	data source
aws_s3_bucket.main	data source
aws_subnets.main	data source

Inputs

Name	Description	Type	Default	Required
name	The name of the app	string	n/a	yes
region	AWS region	string	"us-east-1"	no
s3_bucket	The name of the S3 bucket to use	string	n/a	yes
subnet_tag_name	The subnet tag name to use to deploy the ECS service	string	n/a	yes
tags	tags	map(string)	n/a	yes

Outputs

Name	Description
dead_letter_queue_arn	dead letter sqs queue arn
dead_letter_queue_url	dead letter sqs queue url
main_queue_arn	main sqs queue arn
main_queue_url	main sqs queue url