Goal: Deploy event-driven workflow where object upload to s3 bucket sends event into SQS queue to be processed by Lambda function to convert image file into jpg format using Pillow library via Lambda Layer.
- S3
- SQS
- Lambda
- DynamoDB
- Python, Pillow library
Create an SQS queue with default settings.
Created Access policy using Amazon policy generator to allow S3 bucket to access the queue.
Create a bucket with the default settings.
Created event notification for S3 bucket on all object create events and for S3 to publish messages to the SQS queue.
Wrote Lambda function for Python 3.9, added Lambda Layer using Klayer ARN for Pillow library that will perform the image file format conversion to jpg (https://api.klayers.cloud/api/v2/p3.9/layers/latest/ap-southeast-2/html). Set up SQS as a trigger for Lambda to perform conversion function on passed event, then upload converted jpg image back into s3 bucket.
Created an IAM role for Lambda to be able to access S3 and SQS.
Ran into key error message when attempting to load json formatted message event passed from SQS. I had initially thought SQS passes the same json format as S3, but upon setting up test events, noticed that SQS wraps the message. Solved the issue by extracting s3 event as string from SQS JSON message body then converted the string into dictionary. Followed tutorial https://docs.aws.amazon.com/lambda/latest/dg/with-s3-tutorial.html in order to be able to upload file generated by Pillow library functions from Lambda temporary memory/files.