Lambda invocation loop rework

[joe4dev]

Motivation

• Fixes the infinite Lambda re-spawn loop: Whenever a Lambda environment fails to start (e.g., even due to common user errors), the Lambda container enters an infinite retry loop instead of reporting an appropriate error message and stopping retries (depending on the error cause). This behavior is user-unfriendly and affects many GitHub issues and support tickets.
• Architecture mismatch between LocalStack and real AWS Lambda: The architectural mismatch causes parity issues, impedes debugging, and complicates adding new features.

Goals

• Core loop: The default invocation path should be synchronous instead of asynchronous. This enables improved error handling and better AWS parity.
• Environment management: Allow for more explicit control over execution environments. This unblocks custom workers.
• Asynchronous event invocation: Separate asynchronous event invocations and handle retries, DLQs, and event destinations properly. This fixes bugs and increases AWS parity.
• Encapsulate concurrency logic: The logic related to concurrency management should be better encapsulated rather than scattered throughout an implicit and complex control flow through an asynchronous invocation loop. This enables better AWS parity and makes improving and adding features easier.

Changes

This PR basically re-writes the data plane (i.e., everything related to environment assignment and invocation scheduling) of our new Lambda provider. The re-architected implementation is inspired based on the real architecture of AWS Lambda explained at AWS re:Invent 2022 - A closer look at AWS Lambda (SVS404-R).

Invocation model

Images from AWS: https://youtu.be/0_jfH6qijVY

Synchronous invocation

Asynchronous Invocation

More details to follow ...

Internal Documentation

• Collection of Meeting Notes (Notion)
• Initial goals and brainstorming notes (Notion)

Testing

This PR adds more AWS-validated tests and fixes some existing tests (e.g., fixes timing assumptions, and makes async tests faster). One example: We previously triggered the DLQ destination upon every failed invocation rather than only after the last retry.

TODO

See internal documentation Collection of Meeting Notes (Notion)

[github-actions]

LocalStack Community integration with Pro

       2 files         2 suites   1h 22m 58s ⏱️
2 193 tests 1 706 ✔️ 487 💤 0 ❌
2 194 runs  1 706 ✔️ 488 💤 0 ❌

Results for commit 1ec503f.

♻️ This comment has been updated with latest results.

[coveralls]

coverage: 79.839% (+0.2%) from 79.684% when pulling 1ec503f on lambda-invocation-loop-rework into a9a0668 on master.

[dominikschubert]

Love it 🚀