AWS Challenge by A Cloud Guru
Automate an ETL processing pipeline for COVID-19 data using Python and AWS Cloud Services
The following image displays the entire architecture developed using THE SERVERLESS FRAMEWORK
The following tech stacks were used:
- Python
- AWS
- The Serverless Framework
- Github
The following AWS Services are used for this project:
- S3 Bucket
- Glue Crawler
- Glue Database
- IAM Roles
- DynamoDB Table
- Cloudwatch Event Rule
- Lambda Function
- Cloudwatch Logs
- SNS Subscription
- SNS Topic
Used Python's built-in module, Unittest, to perform some simple tests, to ensure that the code can handle unexpected situations and subsitute invalid data.
Depending on the result of the process, a notification will be sent out to all the subscribers through AWS SNS
Dashboard developed using Quicksight and Data retrived from Athena
https://mohdsaquib.hashnode.dev/event-driven-covid-19-data-etl-process-acloudguruchallenge
Feel free to email me at nsaquib96@gmail.com