Data Engineering Portfolio

Welcome to my collection of production-grade data engineering projects. This portfolio demonstrates hands-on experience designing, building, and optimizing data pipelines across cloud platforms and big data technologies.

🌟 Highlight Projects

🏆 Featured Implementations

Project	Technologies	Key Features
Real-Time IPL Analytics	Spark, Delta Lake, AWS Glue	Processed 15GB match data with 60% faster queries
Smart City IoT Pipeline	AWS Kinesis, Lambda, Redshift	50K events/sec streaming architecture
Azure Fabric ETL Framework	Data Factory, Synapse, Power BI	End-to-end CI/CD pipeline deployment

☁️ Cloud Specializations

Azure

• Olympic Analytics Pipeline (Synapse, Blob Storage)
• Fabric Real-Time Tutorial (Event Hubs, Stream Analytics)

AWS

• Airlines Data Ingestion (Kinesis, S3, Athena)
• YouTube Trend Analysis (Glue, EMR, QuickSight)

🛠️ Technical Stack

# Sample Spark ETL Snippet
from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .config("spark.sql.shuffle.partitions", 100) \
    .getOrCreate()

df = spark.read.parquet("s3://ipl-data/raw/") \
    .transform(clean_data) \
    .write.format("delta") \
    .save("/mnt/processed/")