Implementing best practices for PySpark ETL jobs and applications.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

Near real time ETL to populate a dashboard.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Conductor OSS SDK for Python programming language

Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management

🚹 💾 Script to import issues from a JIRA instance into a database.

Airflow DAGs for the Stellar ETL project

Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.

Solution for IBM Data Engineer Professional Certificate

ETL pipeline combined with supervised learning and grid search to classify text messages sent during a disaster event

A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and Delta Lake.

A tutorial to setup and deploy a simple Serverless Python workflow with REST API endpoints in AWS Lambda.

ETL process which downloads, transforms, and loads Freddie Mac/Fannie Mae mortgage data

Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator