Support for managed Airflow via plugin

[jeremyadamsfisher]

Is your feature request related to a problem? Please describe.
I would like to use a managed Airflow cluster like MWAA. Therefore, I cannot swap out the airflow executable. It can, however, take custom plugins.

Describe the solution you'd like
Distribute cwl-airflow as a plugin or the equivalent to work on a managed Airflow system

[michael-kotliar]

Hi @jeremyadamsfisher,

Thank you for your feature request. We are working on moving CWL-Airflow to the cloud, so smooth integration with MWAA is on our TODO list.

If you can install custom python packages on MWAA, you don't need to swap the original airflow executable. You just need to install CWL-Airflow package and create your DAGs using CWLDAG class. CWL-Airflow uses the original Airflow and only extends its functionality with the ability to parse and execute CWL workflows. Let me know if you have any questions about installation and configuration of CWL-Airflow.

[jeremyadamsfisher]

For others' reference, it seems one can install custom packages on MWAA: https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html