Airflow Unit Tests and Integration Tests
-
Updated
Nov 16, 2022 - Python
Airflow Unit Tests and Integration Tests
Gerador de DAGs no Apache Airflow para fazer clipping do Diário Oficial da União.
Zero configuration Airflow plugin that let you manage your DAG files.
Orchestration of data science and earth observation models in Apache Airflow, scale-up with Celery Executor, experiment with jupyter notebook using a docker containers composition
My self-learning about Apache Airflow
This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
🎵 LyricWave - Your AI Music Composer 🎶 Compose Unique MP4 Songs Effortlessly! LyricWave uses AI to create personalized music by harmonizing lyrics with captivating melodies and synthetic vocals. Unleash your musical creativity today! 🚀🎶
Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache Airflow Guide
An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
A starting point for a data stack using Python, Apache Airflow and Metabase.
Automate your data pipelines using Apache Airflow with this ready-to-use DAG for data integration, ETL and workflow automation.
Here I added 9 projects which have been made by me during my apprenticeship in Yandex.Practicum as data engineer.
This project creates a basic web service for solving image-based CAPTCHAs. Using the Flask framework, it allows users to upload CAPTCHA images and employs an Optical Character Recognition (OCR) pipeline to extract the embedded text.
End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
Airflow Data Processing Pipeline for TUL Catalog on Blacklight Data
Built functional python ETL script with functions that initialized spark clusters using pyspark library to extract songs stored in S3 bucket. Partitioned songs data by year and artist_id and compressed in parquet output files to increase load performance. Used the overwrite mode in spark to ensure every new run of ELT script is overwritten in th…
Scraping and analyzing corona virus data from minsal.cl.
Shares ETL to scrape historical data of one shares and import in metabase
Add a description, image, and links to the airflow-dags topic page so that developers can more easily learn about it.
To associate your repository with the airflow-dags topic, visit your repo's landing page and select "manage topics."