Realtime-Stock-market-Analysis-using-Kafka

Introduction

This project focuses on implementing an end-to-end data engineering solution for real-time stock market data using Kafka.

Various technologies will be utilized, including Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Technologies Used

• Programming Language: Python
• Amazon Web Services (AWS)

1. S3 (Simple Storage Service)
2. Athena
3. Glue Crawler
4. Glue Catalog
5. EC2

• Apache Kafka

Dataset

You are free to choose any dataset for this project. The main objective is to showcase the data engineering aspects, particularly the development of a data pipeline.