Data Analytics Dashboard

This is to create the data analytics dashboard using elasticsearch, performing EDA thorugh pyspark, hadoop, hive and AWS glue ETL

Getting started

Let me clearly define the yardsticks, for me not to deviate much;

Create the data analystics project with either web or the mobile interface
Big data, have to satisfy at least one V our of 5 Vs (Volume, velocity, veracity, value & variety)
tech stack should have at least one of these mentioned tools
- Map reduce, spark, storm, hive, pig, flink etc.

Current status : web interface, using fastAPI (yet to be done) for more info on the steps, please follow steps.pdf

Cloud service provider : AWS, GCP
1. AWS for EMR, ETL operations
2. GCP hosted elasticsearch cloud, final data visulisations
Pre-processing : AWS Athena
Processing & ETL : pyspark, hive, glue
File system : S3, GCP bucket, HDFS
Data storage : hive for big data storage and elsticsearch index for final dashbaord
Visualisation : Kibana Elastic Cloud Dashboard
BE API : python

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
cloud		cloud
elasticsearch_kibana		elasticsearch_kibana
hive		hive
python_spark		python_spark
README.md		README.md
Steps.pdf		Steps.pdf
final_dashboard.png		final_dashboard.png