Apache Airflow ETL Workflow using Docker

📌 Project Overview

This project demonstrates a production-ready data engineering workflow built using Apache Airflow and Docker.

It showcases how to orchestrate end-to-end ETL pipelines, perform data transformations, implement conditional workflows, and handle success/failure notifications in a reproducible containerized environment.

The solution includes five independent DAGs, each designed to highlight a real-world Airflow orchestration pattern commonly used in data engineering teams.

🏗️ Architecture

Technology Stack

Apache Airflow 2.8.0

Webserver

Scheduler

LocalExecutor

PostgreSQL

Airflow metadata database

Data warehouse

Docker & Docker Compose

Parquet

pyarrow engine

snappy compression

Data Flow

CSV → PostgreSQL (raw) → PostgreSQL (transformed) → Parquet

  ↘ Conditional Workflow

  ↘ Notification Workflow

📂 Repository Structure

project-root/

├── docker-compose.yml

├── requirements.txt

├── README.md

├── dags/

│   ├── dag1\_csv\_to\_postgres.py

│   ├── dag2\_data\_transformation.py

│   ├── dag3\_postgres\_to\_parquet.py

│   ├── dag4\_conditional\_workflow.py

│   └── dag5\_notification\_workflow.py

├── tests/

│   ├── test\_dag1.py

│   ├── test\_dag2.py

│   └── test\_utils.py

├── data/

│   └── input.csv

├── output/

│   └── (generated parquet files)

└── plugins/

⚙️ Prerequisites

Ensure the following are installed on your system:

Docker

Docker Compose

Git

🚀 Setup Instructions

1️⃣ Clone the Repository

git clone

cd project-root

2️⃣ Start Airflow Using Docker

docker-compose up

This command starts the following services:

PostgreSQL database

Airflow initialization container

Airflow webserver

Airflow scheduler

🌐 Access Airflow UI

Open your browser and navigate to:

http://localhost:8080

Default Credentials

Username: admin

Password: admin

🔁 DAGs Overview

DAG 1: CSV to Postgres Ingestion

DAG ID: csv_to_postgres_ingestion

Schedule: @daily

Functionality

Reads employee data from a CSV file

Creates the target table if it does not exist

Truncates the table to ensure idempotency

Loads data into PostgreSQL

Output

Table: raw_employee_data

DAG 2: Data Transformation Pipeline

DAG ID: data_transformation_pipeline

Schedule: @daily

Transformations Applied

full_info → name - city

age_group → Young / Mid / Senior

salary_category → Low / Medium / High

year_joined → extracted from join date

Output

Table: transformed_employee_data

DAG 3: Postgres to Parquet Export

DAG ID: postgres_to_parquet_export

Schedule: @weekly

Functionality

Validates source table existence and data

Exports transformed data to Parquet format

Uses snappy compression

Validates schema and file integrity

Output

Parquet files in /opt/airflow/output/

Filename includes execution date

DAG 4: Conditional Workflow

DAG ID: conditional_workflow_pipeline

Schedule: @daily

Branching Logic

Monday–Wednesday → Weekday processing

Thursday–Friday → End-of-week processing

Saturday–Sunday → Weekend processing

A unified end task runs regardless of the branch executed.

DAG 5: Notification Workflow

DAG ID: notification_workflow

Schedule: @daily

Functionality

Simulates a risky operation

Triggers success or failure callbacks

Executes cleanup logic regardless of outcome

Demonstrates Airflow trigger rules and callbacks

▶️ How to Trigger DAGs

Open the Airflow UI

Enable the DAG

Click Trigger DAG

Monitor execution using Graph View and Logs

📦 Output Verification

PostgreSQL Tables

raw_employee_data

transformed_employee_data

Parquet Files

Location: /opt/airflow/output/

Example:

employee_data_2024-01-01.parquet

🧪 Running Unit Tests

Unit tests validate DAG structure only and do not require Airflow or PostgreSQL to be running.

pytest tests/ -v

Tests Cover

DAGs load without import errors

Correct DAG IDs

Correct number of tasks

Task dependencies

Schedule intervals

No cyclic dependencies

All DAGs load successfully

🛠️ Troubleshooting

Airflow UI Not Accessible

Ensure port 8080 is free

Check logs:

docker-compose logs airflow-webserver

DAGs Not Appearing

Verify DAG files are inside the dags/ directory

Restart services:

docker-compose restart

Containers Fail to Start

docker-compose down -v

docker-compose up

📜 Dependencies

All Python dependencies are listed in requirements.txt.

✅ Expected Outcome

Airflow environment starts with a single command

Five DAGs visible and operational in the UI

PostgreSQL tables populated correctly

Parquet files generated successfully

Unit tests pass

Clear, evaluator-ready documentation

📌 Notes

Email and alert notifications are simulated via logging

No external services are required

Designed for local execution, learning, and evaluation purposes