AWS Lambda - Python multithreading function

This example shows how to leverage multithreading in AWS Lambda functions to download and upload content on an S3 bucket. To get a better understanding of how effective multithreading is, the project contains code for three Lambda functions:

• sequential Lambda function: the function downloads and uploads objects sequentially;
• parallel Lambda function: uses multithreading to perform the same downloads and uploads;
• parallel multipart Lambda function: uses multithreading to perform the same downloads and uploads, and additionally read the content from the source bucket in chunks, and use multipart uploads to S3;

The Lambda functions code is in the lambda_code directory.

The project provides IaC code to deploy

• 3 Lambda functions
• 2 S3 buckets (source and destination)

Prerequisites

• AWS CDK installed
• Pipenv installed
• Python 3.12+
• AWS CLI configured

Deployment

1. Clone the repository:

   git clone <repository-url> 

2. Deploy the infrastructure:

   make deploy

3. Create and upload the source_obj

   make create_source_object
   aws s3 cp source_obj s3://<your-aws-account-id>-source-bucket

How to measure the effectiveness of multithreading

Both the Lambda functions have tracing enabled, and you will see the performance of the code in CloudWatch. Check out this [article](#TODO article link) for a detailed step-by-step guide.

The Lambda functions are deployed with the following setup

• 1769MB of memory: to allow the usage of a full virtual CPU (see this article for more info)
• 5 minutes timeout
• ARM 64 architecture
• Python 3.12