Utilizing Airflow's built-in functionalities creating a reusable ETL pipeline. Source data resides in a S3 bucket, and the pipeline should include data quality checks and data should be processed within AWS Redshift.

A POC to demonstrate how to work with Apache Airflow

With this app, you can see what programming skills are most in-demand in the current job market.

Repositório criado para estudar Airflow baseado nos cursos do Marc Lamberti

Hands up project for learning purposes

Создание дагов для автоматизации отчетности

This project focuses on utilizing Apache Airflow to orchestrate an ETL (Extract, Transform, Load) process using data from the Stack Overflow API. The primary objective is to determine the most prominent tags on Stack Overflow for the current month.

Ingesting NYC Yellow Taxi Data to GCP using Airflow running on Docker

This repository contains the code of Airflow DAG, Snowflake Tasks and Snowflake Procedures

Data Pipeline Analytics Platform is an end-to-end generic Big Data pipeline. Involves following tech stack: AWS S3, AWS Redshift, AWS EMR Cluster, Apache Spark, Apache Airflow.

Apache Airflow source code to begin with

Pulumi airflow reosurce provider

Analysis of Economic Trends through Alternative Data.

This project demonstrates how to build an ELT pipeline using dbt, Snowflake, and Airflow. Follow the steps below to set up your environment, configure dbt, create models, macros, tests, and deploy on Airflow.

A Python script extracts data from Zillow and stores it in an initial S3 bucket. Then, Lambda functions handle the flow: copying the data to a processing bucket and transforming it from JSON to CSV format. The final CSV data resides in another S3 bucket, ready to be loaded into Amazon Redshift for in-depth analysis. QuickSight for visualizations

Airflow DAGs for the Manifold (TUL Website) application

Airflow DAG project using Python and Bash

load data to snowflake

Airflow ETL for Earthquake Data