Asynchronous Event Processing with API Gateway and Lambda

Sample architecture to process events asynchronously using API Gateway and Lambda.

Architecture

A typical workflow looks as following:

1. The user authenticates against AWS Identity and Access Management (IAM) and obtains security credentials
2. The user does an HTTP POST request to the /jobs jobs API endpoint, specifying in the request body the job parameters
3. The jobs API returns to the user an HTTP response containing the job identifier
4. The jobs API invokes asynchronously the event processing AWS Lambda function
5. The event processing function processes the event, then puts the job results in the jobs Amazon DynamoDB table
6. The user does an HTTP GET request to the /jobs/{jobId} jobs API endpoint, with the job identifier from step 3. as {jobId}
7. The jobs API queries the jobs table to retrieve the job results
8. The jobs API returns to the user an HTTP response containing the job results

If the event processing fails:

1. The event processing function sends the event to the error handling function
2. The error handling function puts the job parameters in the jobs Amazon DynamoDB table
3. The user can retrieve the job parameters by doing an HTTP GET request to the /jobs/{jobId} jobs API endpoint

If the error handling fails:

1. The error handling function sends the event to an Amazon EventBridge archive
2. The user can replay the archived events by using the related Amazon EventBridge feature

Best Practices

• This sample architecture doesn't include monitoring of the deployed infrastructure. If your use case requires monitoring, evaluate to add it (for example, using CDK Monitoring Constructs)
• This sample architecture uses IAM Permissions to control the access to the jobs API. Anyone authorized to assume the JobsAPIInvokeRole will be able to invoke the jobs API: as such, the access control mechanism is binary. If your use case requires a more complex authorization model, evaluate to use a different access control mechanism
• When a user does an HTTP POST request to the /jobs jobs API endpoint, the input data is validated at two different levels: Amazon API Gateway is in charge of the first request validation while the event processing function executes the second one. No validation is performed when the user does an HTTP GET request to the /jobs/{jobId} jobs API endpoint. If your use case requires additional input validation and an increased level of security, evaluate to use AWS WAF to protect your API

Prerequisites

Install on your workstation the following tools:

• AWS Cloud Development Kit (CDK) Toolkit version 2.61.0
• Docker version 20.10.21
• Node.js version 18.13.0
• Projen version 0.67.3
• Python version 3.9.16

You should also own or have access to an AWS account.

Setup

Execute the following steps on your workstation:

• Clone this repository
• Change directory to the repository root:

cd asynchronous-event-processing-api-gateway-api-gateway-lambda-cdk

• Setup the project using Projen:

npx projen

• Activate the Python virtual environment:

source .env/bin/activate

• Install the pre-commit hooks:

pre-commit install
pre-commit install --hook-type commit-msg

Test

To run the project tests execute:

npx projen test

Lint

To lint the project code execute:

npx projen synth
npx projen lint

Scan

To perform a security scan of the project code execute:

npx projen synth
npx projen scan

Release

Execute the following steps on your workstation:

• Change the value of __version__ in .projenrc.py (follow PEP440 specification):

...
__version__ = "0.1.0"
...

• Update Projen task definitions:

npx projen

• Release a new version using Projen:

npx projen release

• Push to remote, including tags:

git push
git push --tags

Bootstrap

To bootstrap the AWS CDK in your AWS account execute:

AWS_PROFILE=$YOUR_AWS_PROFILE npx projen bootstrap

Deployment

To deploy your application in your AWS account execute:

AWS_PROFILE=$YOUR_AWS_PROFILE npx projen deploy

Cleanup

To cleanup your application in your AWS account execute:

AWS_PROFILE=$YOUR_AWS_PROFILE npx projen destroy

Try it out

To try the sample architecture included in this repository proceed to:

• Perform a deployment in your AWS account
• Install on your workstation the AWS Command Line Interface (CLI)
• Assume the JobsAPIInvokeRole printed as output from the deploy command
• Install Postman on your workstation
• Import the Postman collection included in this repository
• Set the JobsAPI variables as following:
  • accessKey: value of the Credentials.AccessKeyId attribute from the assume-role command
  • baseUrl: value of the JobsApiJobsAPIEndpoint output from the deploy command without trailing slash
  • region: value of the AWS Region where you deployed the sample architecture
  • seconds: the value of the input parameter for the sample job. It must be a positive integer
  • secretKey: value of the Credentials.SecretAccessKey attribute from the assume-role command
  • sessionToken: value of the Credentials.SessionToken attribute from the assume-role command
• Test the sample architecture by sending requests to the jobs API

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.