A series of Plugins for Apache Airflow (https://airflow.incubator.apache.org/)

A flexible data integration tool to help nonprofits connect to their data collection tools and ERP systems

This project showcases the implementation of a data pipeline using Apache Airflow. Leveraging the OpenWeather API, it efficiently fetches real-time weather data and performs ETL processing. Results are seamlessly stored in AWS S3 buckets for further analysis. Moreover, the integration of Slack notifications ensures timely alerts to myself.

Data Engineering using Apache Airflow

An EL pipeline built with Apache Airflow that downloads a file from the web uploads it to Google Cloud Storage, and creates an external table in BigQuery for data storage and analysis.

Airflow sandbox

Data pipelines created and monitored using Airflow to feed data into Redshift

Data pipeline that fetches recently played songs in the past 24 hours using Spotify API and saves the data in the SQLite database. Scheduled to run daily using Apache Airflow.

This is a tool to collect title and url from videos posted on youtube.

Fetch Data from a simple csv file, send the data in GCP BigQuery table, run dbt to automate the DWH and run SODA to check Data Quality.

This is End-To-End Data Engineering Project using Airflow and Python. In this project, we will extract data using Twitter API, use python to transform data, deploy the code on Airflow/EC2 and save the final result on Amazon S3

Small sample project to test Apache Airflow workflow engine.

Corrent: Experimental Airflow functional DAG API

An on-going application to track and therefore analyse personal activities and goal setting.

In this project, files were ingested to Google Cloud Storage and later to moved to BigQuery so as to perform some queries and the result moved back to Google Cloud Storage.

Entire ETL pipeline project with different tasks orchestration using Apache airflow. And data storage in PostgreSQL database using Python

This repo contains code to support the tutorial, Automate the Provisioning of Your Apache Airflow Environments.