This project aims to fetch weather data from an API, load it into a Snowflake data warehouse, perform data transformations using DBT (Data Build Tool), and orchestrate the entire workflow using Apache Airflow running in a Docker container.
-
Data Ingestion: A Python script is used to fetch weather data from openweather API.
-
Data Loading: The fetched data is loaded into a Snowflake data warehouse using appropriate Snowflake utilities or libraries.
-
Data Transformation: DBT (Data Build Tool) is employed to perform data transformations on the loaded data within the Snowflake environment. This includes defining models, writing SQL transformations, and managing the transformation pipeline.
-
Workflow Orchestration: Apache Airflow is used to orchestrate the entire data pipeline workflow. An Airflow DAG (Directed Acyclic Graph) is defined to execute the Python script for data ingestion, trigger the data loading process, and run the DBT models for data transformation.
-
Docker Containerization: The Airflow environment is containerized using Docker, allowing for consistent and reproducible deployments across different environments.
- Python
- Snowflake Data Warehouse
- DBT (Data Build Tool)
- Apache Airflow
- Docker
- Ensure you have Docker and Docker Compose installed on your machine.
- Clone the repository:
git clone https://github.com/rohithlanka/weatherdatapipeline.git - Install the required dependencies (Python packages, DBT, Airflow, Docker, etc.).
- Configure the project settings:
- Set up the API credentials and endpoint for data ingestion.
- Configure the Snowflake connection details.
- Set up the DBT project and define the data models.
- Configure the Airflow connections and variables.
- Navigate to the repository directory and run docker-compose up.
- Once all containers are up and running, access the Airflow web UI and trigger the DAG execution to run the entire data pipeline.
- After the ELT process completes, you can access the snowflake UI to view the transformed and old tables.