Airflow DAGs

Airflow DAGs

12 drop-in TaskFlow DAGs in examples/airflow/. Copy the file you want into your Airflow dags/ folder, adjust the knobs at the top, and ship.

These are examples, not a published DAG library. Read, adapt, own.

DAGs by lifecycle stage

Core pipeline

File	What	Cadence
golden_suite_daily_dedupe.py	S3 → Check → Flow → Match → S3 + metrics	Daily 04:00 UTC
golden_suite_incremental_match.py	Match new arrivals against canonical via match_one. Auto-merge ≥0.95, queue 0.75–0.95, append <0.75.	Every 15 min
golden_suite_warehouse_native.py	Snowflake-native variant of daily_dedupe (no S3 hop)	Daily 04:30 UTC
golden_suite_customer_360.py	Multi-source unify (CRM + warehouse + support) with InferMap alignment + multi-pass dedupe	Daily 05:00 UTC

Privacy

File	What	Cadence
golden_suite_pprl_linkage.py	Trusted-third-party PPRL across two parties. Encoded shards in, ID-pair matches out. Raw PII never crosses.	Weekly

Onboarding & monitoring

File	What	Cadence
golden_suite_schema_align_and_load.py	New-source onboarding: InferMap → cache mapping → Flow → upsert	Manual
golden_suite_schema_drift_alarm.py	Daily compare current columns to cached mapping. Alert + log on drift. Never auto-updates.	Daily
golden_suite_quality_gate.py	GoldenCheck as gatekeeper (not preprocessor). Threshold-based. Failure blocks dependent Datasets.	Daily 03:00 UTC

Feedback loop

File	What	Cadence
golden_suite_review_worker.py	Apply steward decisions on review queue every 5 min. Persists into Learning Memory.	Every 5 min
golden_suite_active_learning.py	Weekly retrain boost classifier from labeled pairs. Promote to current.yaml only if F1 strictly beats the running model.	Weekly Sun

Operationalize

File	What	Cadence
golden_suite_reverse_etl.py	Push deduped golden records to Salesforce + HubSpot, watermark-incremental	Every 2 hours
golden_suite_backfill.py	Reprocess N days when match config changes. Dynamic task mapping; isolated _backfill/<run-id>/ output.	Manual

Patterns used across all 12

• TaskFlow API (Airflow 2.7+, compatible with 3.x). Tested at 2.10.
• Idempotent or marker-protected. Safe to retry.
• Fail loudly on empty input. Silent zero-row processing has bitten enough teams that I'd rather a red square than a missed dedupe.
• No giant XComs. Outputs go through S3 / Postgres so each task can retry independently.
• Tunable knobs as top-of-file constants. No magic numbers buried in task bodies.
• Promotion gates. Active-learning, drift, schema-align all require explicit human-or-metric promotion. No silent regressions.

Prerequisites

Each DAG declares its pip deps in its docstring. Common stack:

pip install apache-airflow>=2.7 polars
pip install goldenpipe[full] goldenmatch[postgres,pprl] goldencheck goldenflow infermap
pip install apache-airflow-providers-amazon apache-airflow-providers-postgres

Connections: aws_default, postgres_default, plus snowflake_default / salesforce_default / hubspot_default for the relevant DAGs.

Variables: golden_suite_bucket (your S3 data lake bucket).

DDL for the tracking tables (metrics / canonical / review queue / PPRL audit / mapping cache / drift events / etc.) lives in examples/airflow/README.md.