In this project, we leverage the synergy of Apache Airflow, DBT, and Snowflake to construct a data pipeline. Our objective is to import raw data into Snowflake, apply DBT transformations, and furnish clean, analysis-ready datasets for BI Tools visualization. This undertaking showcases the seamless integration and cooperation of these cutting-edge data engineering technologies. We use a Maven Analytics sample dataset, comprising two CSV files, featuring churn customer data from a fictional telecom firm. The primary CSV file contains 38 columns encompassing personal, contractual, and revenue details of 7,043 California-based customers. The second CSV file includes zip code and population information.
Created by Guillermo Parlade and Josep Sugrañes
The project contains the following files and folders:
- dags: This folder contains the Python files for your Airflow DAGs. By default, this directory includes two example DAGs:
poc_dbt_snowflake: PoC
- Dockerfile: This file contains a versioned Astro Runtime Docker image that provides a differentiated Airflow experience. If you want to execute other commands or overrides at runtime, specify them here.
- include: This folder contains any additional files that you want to include as part of your project. It is empty by default.
- packages.txt: Install OS-level packages needed for your project by adding them to this file. It is empty by default.
- requirements.txt: Install Python packages needed for your project by adding them to this file. It is empty by default.
- plugins: Add custom or community plugins for your project to this file. It is empty by default.
- airflow_settings.yaml: Use this local-only file to specify Airflow Connections, Variables, and Pools instead of entering them in the Airflow UI as you develop DAGs in this project.
- Start Airflow on your local machine by running 'astro dev start'.
This command will spin up 4 Docker containers on your machine, each for a different Airflow component:
- Postgres: Airflow's Metadata Database
- Webserver: The Airflow component responsible for rendering the Airflow UI
- Scheduler: The Airflow component responsible for monitoring and triggering tasks
- Triggerer: The Airflow component responsible for triggering deferred tasks
- Verify that all 4 Docker containers were created by running 'docker ps'.
Note: Running 'astro dev start' will start your project with the Airflow Webserver exposed at port 8080 and Postgres exposed at port 5432. If you already have either of those ports allocated, you can either stop your existing Docker containers or change the port.
- Access the Airflow UI for your local Airflow project. To do so, go to http://localhost:8080/ and log in with 'admin' for both your Username and Password.
You should also be able to access your Postgres Database at 'localhost:5432/postgres'.