-
Notifications
You must be signed in to change notification settings - Fork 0
LambdaDoomLoop
title: Lambda Doom Loop Prevention type: technique created: 2026-05-26 last_updated: 2026-05-26 related: ["radar/techniques/ContinuousIntegration"] sources: ["https://aaronstuyvenberg.com/posts/lambda-timeout-doom-loop"] radar_quadrant: Techniques radar_ring: Assess radar_position: inner
A defensive configuration pattern for AWS Lambda functions triggered by SQS queues that prevents timeout-induced infinite requeue loops from exhausting concurrency quota and running up unbounded costs.
When a Lambda function triggered by SQS times out before completing, AWS returns the message to the queue. If the message consistently causes timeouts — due to a poison payload, an external dependency that never responds, or a logic bug — the cycle repeats indefinitely: Lambda starts, times out, message requeues, Lambda starts again. Each iteration consumes a concurrency slot for the full timeout duration, and the loop continues until manually interrupted.
Three configuration parameters bound the damage:
-
Lambda timeout < SQS visibility timeout — the function must time out before the visibility timeout expires, or SQS will never reclaim the message. Set the Lambda timeout to roughly half the SQS visibility timeout as a safe margin.
-
Dead-letter queue (DLQ) — configure a DLQ on the SQS queue. Messages that fail beyond the retry threshold are moved to the DLQ instead of requeuing indefinitely. DLQ messages can be inspected, replayed, or discarded manually.
-
maxReceiveCount — set to 3–5. After this many failed delivery attempts, the message moves to the DLQ. This hard-caps the number of times a poison message can trigger the function.
- Alert on DLQ depth — a non-empty DLQ indicates a recurring failure that needs investigation
- Set a Lambda reserved concurrency limit to cap blast radius during a loop
- Use AWS X-Ray or CloudWatch Logs Insights to identify the poison payload once it lands in the DLQ
Lambda Doom Loop Prevention sits in the Assess ring of the Techniques quadrant, at inner position. First studied via Aaron Stuyvenberg's blog (2024-10-14). The failure mode is a well-known Lambda+SQS pitfall that teams typically discover in production rather than during design. The mitigation is entirely configuration — no code changes required. Inner position reflects zero implementation cost and direct applicability to any Lambda+SQS architecture. Remaining gate before Trial is confirmed configuration of all three parameters (timeout ratio, DLQ, maxReceiveCount) in an active Lambda+SQS deployment with DLQ alerting verified.