Project | Creating real-time Tweets Extraction and Analysis Pipelines

Introduction

Social media - Twitter - is becoming an important tool in developing a community around a product

Business case

1. Create an engaged Twitter community around our product
2. Detect and minimize damage from negative publicity (Tweets)
3. Identifying potential customers
4. Getting customer feedback from the confort of their Twitter accounts
5. Keep tabs on our competitors

Main Objective

1. Provide cleaned and tagged tweets to the Social Media, Marketing and Analytics teams for Analysis and Visualization

Other objectives of the project

1. Extract tweets with a keyword (this case "tesla") in real-time
2. Create batches of tweets
3. Analyze sentiments in the tweets every 30 minutes
4. Insert the clean and analyzed data into an AWS S3 bucket and Redshift table ready for querrying and visualization
5. Visualize the tweets using AWS QuickSight

Medium Article

https://medium.com/@jkimera5/creating-real-time-tweets-processing-pipeline-using-aws-ec2-lambda-s3-glue-and-athena-c0004000029d