Apache Airflow Learning Module

This module is a hands-on introduction to Apache Airflow. It gives you a local Docker Compose environment, three beginner-friendly DAGs, and a technical blog draft that explains the setup from first principles.

The stack follows the official Apache Airflow Docker Compose quick start for Airflow 3.2.2: an API server, scheduler, DAG processor, worker, triggerer, PostgreSQL, and Redis. It is intentionally for learning and local experimentation, not production.

What You Will Build

• A local Airflow 3.2.2 environment running on Docker Compose.
• A classic operator DAG that introduces tasks, dependencies, retries, and templating.
• A TaskFlow API DAG that passes Python return values between tasks.
• A branching DAG that chooses a path at runtime.

Requirements

• Docker Desktop or Docker Engine with Docker Compose v2.
• At least 4 GB of memory available to Docker. The Airflow docs recommend 8 GB when possible.
• A shell from this directory.

Quick Start

Make a copy of .env.example to .env

cp .env.example .env

Initialize the Airflow metadata database and admin account:

docker compose up airflow-init

Start the environment:

docker compose up -d

Open the Airflow UI:

• URL: http://localhost:8080
• Username: airflow
• Password: airflow

List the DAGs from the CLI:

docker compose run --rm airflow-cli dags list

Run one DAG without waiting for the scheduler:

docker compose run --rm airflow-cli dags test hello_airflow 2026-01-01

Shut everything down and remove local database volumes:

docker compose down --volumes --remove-orphans

DAGs Included

hello_airflow

Introduces the basic shape of a DAG with EmptyOperator, PythonOperator, BashOperator, task dependencies, retries, and Jinja templating.

taskflow_sales_summary

Uses the TaskFlow API. Python functions become Airflow tasks with @task, and return values are passed between tasks through Airflow-managed XComs. The final task writes a small JSON summary to data/.

branching_quality_check

Uses BranchPythonOperator to pick one downstream task at runtime, then rejoins the workflow with a trigger rule that accepts one successful branch.

Helpful Commands

The Makefile wraps the most common commands:

make init
make up
make list-dags
make test-sales
make logs
make down

Learning Path

1. Start Airflow and open the UI.
2. Unpause hello_airflow, trigger it manually, and inspect each task log.
3. Trigger taskflow_sales_summary and look at the XCom tab for each task.
4. Trigger branching_quality_check twice with different logical dates and compare which path Airflow skips.
5. Edit one DAG, save the file, and watch the DAG processor pick up the change.

References

• Apache Airflow Docker Compose quick start: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
• Airflow 101 tutorial: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html
• TaskFlow API tutorial: https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html