💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Python framework for building Google Cloud Composer workflows.

Airflow DAG for sentiment analysis on GCP

Auto run a Cloud Composer DAG when an object is uploaded to a Google Cloud Storage bucket

Scheduled ETL pipeline with Apache Airflow to fetch 2hr & 24hr weather forecast data from data.gov.sg's API, load it into BigQuery. Visualisation done with Plotly & Dash and deployed with Cloud Run.

Welcome to the MiniProjects Playground—an interactive space where learning meets doing! This repository is a collection of hands-on mini-projects that I've crafted after delving into various tech stacks and frameworks. From theory to application, each project is a testament to the practical side of coding.

This project demonstrates a seamless integration of Apache Airflow, Snowflake, and Google Cloud Composer to create an automated ETL pipeline for fetching, transforming, and storing stock price data. The workflow highlights the power of cloud-native orchestration and scalable data warehousing to handle real-time data processing efficiently

End-to-End data engineering project with Google services