This is a infrastructure project run with Apache Airflow.

Starter package to setup Apache Airflow locally.

Process of scheduled data extraction, transform and load is done using Apache Airflow and PySpark

An example Apache Airflow DAG-definition source repository, to be used with the Airflow DAG Aggregator.

This repo contains the concepts of Apache Airflow and the practical implemetation I'll be doing while learning.

Apache Airflow Cheatsheet

A simple dag for triggering the Cloud Data Fusion Pipeline using Apache Airflow.

Setup for Apache Airflow with Docker.

Project in Course of Udacity's Data Engineering Nano-Degree

The ETL Pipeline using a way autoscaling

Celery and Kubernetes operators are used in order to manage data engineering pipelines of stocks and cryptocurrencies prices

Airflow pipeline using aws S3 and Redshift cluster

A simple DataOps for wine dataset on Docker

Repoitory to learn Airflow.

Udacity project within the Data Engineer Nanodegree

Playing around with Airflow

This repository contains the projects I completed in the Udacity Data Engineering Nanodegree.

This repository showcases an end-to-end data engineering pipeline built with Airflow and Docker. The pipeline extracts data from a .tsv file, performs transformation operations, and loads the transformed data into a CSV file.

Api integration using Airbyte/Airflow Mega Docker Cluster - Airbyte handles the Api connection & Airflow does the orchestration.

ETL Pipeline for Twitter Data using Apache Airflow