Weather ETL Pipeline

Project Description

This project demonstrates a complete data engineering pipeline using the OpenWeather API, Supabase PostgreSQL, and Apache Airflow. The pipeline collects, processes, and stores transformed weather data to database to showcase essential data engineering skills.

Features

• ETL Pipeline: Fetch, transform, and load weather data into a PostgreSQL database.
• Batch Processing: Scheduled data collection with Apache Airflow.
• Cloud Integration: Supabase used for database and storage.

Tech Stack

• Programming Languages: Python, SQL
• Orchestration: Apache Airflow
• Database: Supabase PostgreSQL
• APIs: OpenWeather API