Client-Server Separation: DAG Processor & Triggerer

[amoghrajesh]

Body

Epic for addressing client-server separation in DAG Processor and Triggerer components.

Why

Both DAG Processor and Triggerer run user code and they should stay in task sdk for that reason. They currently use an in process execution API but they should shift towards communicating with API server via API for DB needs.

Related AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services

Scope

• DAG Processor: Eliminate task SDK dependencies on airflow core for DAG parsing and serialization. Consider moving it to task SDK.
• Triggerer: Eliminate task SDK dependencies on airflow core for trigger definition and execution.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.