Orchestration

Scheduled Jobs

prohori pipeline up --orchestration adds Apache Airflow to the Docker stack, with five DAGs that keep the warehouse and models current. Open http://127.0.0.1:18080. The full page is docs/ORCHESTRATION.md.

DAG	When	What it does
prohori_backfill	By hand	Replays the eight simulated scenarios with their labels, then builds the marts
prohori_data_quality	Hourly	dbt build with its 68 checks, and a freshness check on the events
prohori_nightly_evaluation	02:00 UTC	Retrains the flow models; the run fails if recall drops below 0.75
prohori_incident_digest	06:00 UTC	The last day's incidents, rules and dead letters, as Markdown and JSON
prohori_enrichment_refresh	03:30 UTC	Keeps the incident stage's enrichment cache current (offline without API keys)

• Every job also works without Airflow. Each task runs a command: prohori pipeline digest, prohori pipeline check-recall, prohori pipeline refresh-enrichment, or dbt.
• Separate profile. Airflow adds a 3 GB memory limit (about 1.2 GB used when idle), so the default stack leaves it out. prohori pipeline down always stops it.
• Local only. 127.0.0.1:18080, no login. Do not expose the port.
• Datasets stay local. The nightly evaluation reads data/ read-only and skips itself when the dataset is not there.
• Tested. CI imports every DAG with Airflow on Linux and runs each one in the stack.