Airflow DAG project using Python and Bash
-
Updated
Mar 31, 2023 - Python
Airflow DAG project using Python and Bash
Data Engineering Project Using Airflow, S3, & Redshift
Example: Apache Airflow for creating, monitoring and scheduling workflows.
Utilizing Airflow's built-in functionalities creating a reusable ETL pipeline. Source data resides in a S3 bucket, and the pipeline should include data quality checks and data should be processed within AWS Redshift.
A POC to demonstrate how to work with Apache Airflow
Hands up project for learning purposes
Создание дагов для автоматизации отчетности
This project focuses on utilizing Apache Airflow to orchestrate an ETL (Extract, Transform, Load) process using data from the Stack Overflow API. The primary objective is to determine the most prominent tags on Stack Overflow for the current month.
Cloud Composer: Copying BigQuery tables across different locations.
Repositorio para realizar el curso llamado "Apache Airflow" en Udemy, expuesto por Javier López Tomás (Data Engineer)
AirflowDataPipeline is a daily data collection project that extracts JSON data from a website's API and loads it into a SQL database using Airflow. This project offers an automated and reliable solution for managing data pipelines.
ETL pipeline to extract webscraped forex data and google news library data on daily basis and storing it in a postgres database for market analysis and insights
An end-to-end pipeline that ingests raw data from CSV files through Airflow DAGS into BigQuery. From there, it uses dbt to normalize and clean the data and afterwards to make the transformations and come up with relevan reports.
An airflow pipeline which scrapes daily items data from rocket-league.com, alerts if condition is met and sends data to MongoDB
Data Engineer projects
A data pipeline for the popular pagila database system.
Utilizando o conceito de Docker Operator para execução local do Airflow com Docker em execuções de código ETL fora da DAG do Airflow. 🇧🇷
Add a description, image, and links to the airflow-dags topic page so that developers can more easily learn about it.
To associate your repository with the airflow-dags topic, visit your repo's landing page and select "manage topics."