O objetivo deste projeto é contribuir com a formação de iniciantes que almejam entrar na área de dados, fornecendo uma visão baseada em dados sobre as habilidades e conhecimentos mais demandados pelo mercado. Através da coleta e análise de vagas de emprego/estágio, o projeto visa responder à pergunta: “Como se tornar um profissional de dados?"

An end-to-end pipeline that ingests raw data from CSV files through Airflow DAGS into BigQuery. From there, it uses dbt to normalize and clean the data and afterwards to make the transformations and come up with relevan reports.

Gerador de DAGs no Apache Airflow para fazer clipping do Diário Oficial da União.

EEA Crawler contains the tasks (DAGs) used by Apache Airflow to index content from various EEA-Eionet websites into a central Elasticsearch (aka content hub).

Airflow Data Processing Pipeline for TUL Catalog on Blacklight Data

Airflow ETL with AWS, Docker and Postgres consuming TMDb API

This project showcases an ELT pipeline that extracts JSON data, loads it into a PostgreSQL database, applies transformations using Python scripts, saves the transformed data in a CSV file, and shares it through a FastAPI endpoint.

Multi-Modal Representational Learning for Social Media Popularity Prediction

A simple DAG for NewsAPI and Apache Airflow: https://airflow.apache.org/

Airflow DAGs for the Manifold (TUL Website) application

Airflow DAGs for PA Digital aggregation processes

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows

Status BI python DAGs for Airflow

A dbt data pipeline capstone project.

Using yfinance, we grab minute-by-minute BTC-USD data, dump it into PostgreSQL, and link Excel via ODBC for quick analysis!

A Python script extracts data from Zillow and stores it in an initial S3 bucket. Then, Lambda functions handle the flow: copying the data to a processing bucket and transforming it from JSON to CSV format. The final CSV data resides in another S3 bucket, ready to be loaded into Amazon Redshift for in-depth analysis. QuickSight for visualizations

Ochestraction of ETL process with Apache Airflow

This project demonstrates how to build an ELT pipeline using dbt, Snowflake, and Airflow. Follow the steps below to set up your environment, configure dbt, create models, macros, tests, and deploy on Airflow.

A program that analyzes the correlation between Airflow DAGs and tasks, enabling anomaly detection for error tasks and operator failures.