Real Time Data Engineering on Stock Market Data using Apache Kafka

Introduction

In this project, I execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Dataset Used

You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline)

Here is the dataset used by me - https://github.com/Ishaan-Rawat/stock-market-kafka/blob/master/data/indexProcessed.csv

For further information, visit the blog - https://medium.com/@ishaan.rawat611real-time-data-pipeline-using-apache-kafka-glue-and-athena-on-aws-cloud-fb29eecb4788