Beginner data engineering project - batch edition

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Apache Airflow Website

Visualize dependencies between Airflow DAGs

Integrating Apache Airflow, dbt, Great Expectations and Apache Superset to develop a modern open source data stack.

A data warehousing application on NYC Food Inspection

The Earthquake Emergency Response Robots project aims to create, develop, and implement systems specifically designed to handle post-earthquake situations. The main focus of the project is to build adaptable robots that come equipped with sensors and communication capabilities.

Data engineering monorepo with Airflow, dbt and postgres

This ETL (Extract, Transform, Load) project employs several Python libraries, including Airflow, Soda, Polars, YData Profiling, DuckDB, Requests, Loguru, and Google Cloud to streamline the extraction, transformation, and loading of CSV datasets from the U.S. government's data repository at https://catalog.data.gov.

Live COVID-19 tracker with Airflow

A data pipeline to extract News articles from BBC News, storing it to MongoDB, performing NLP analysis (Topic modeling, sentiment analysis), and orchestrating with airflow.

As a Data Engineer for a fictional E-commerce startup, this project addresses the task of analyzing the web server logs to find the number of product pages visited and the number of items in the cart.

I generated the list of tables using Schemaspy and hosting it on my github. Definitely helps in understanding the backend tables of airflow.

Alumni Profile Matching is a project aimed at facilitating networking between graduate students and alumni with similar backgrounds and career goals. By leveraging machine learning techniques and data processing pipelines, the project aims to provide graduate students with personalized recommendations of alumni profiles to connect with.

SPI or I²C differential pressure sensor and IMU unit.

The main aim of this project is to migrate a data warehouse tech stack from MySQL to PostgreSQL data warehouse and redash to superset dashboard.

Sentiment Analysis and SparkML modeling on Financial Data using HuggingFace, Spark, MongoDB, Airflow and GCS.

White and Red Wine classification using logistic regression

Ask Ubuntu Logs analysis with PySpark on GCP | Pipeline with Airflow (Cloud Composer)