Enable DAGS_FOLDER & PLUGINS_FOLDER to be in a remote location such as GCS or AWS

[CorsettiS]

Description

Just like the logs, it would be interesting and very useful to allow the DAGS_FOLDER & PLUGINS_FOLDER to refer to a path in any cloud provider. Ideally, a connection to the cloud provider should be created only when the DAGs are being parsed. A possible implementation I thought about is to create temp dirs with the downloaded contents of the DAGS_FOLDER & PLUGINS_FOLDER buckets, refer to these temp dirs by DAGS_FOLDER & PLUGINS_FOLDER and every time that the dags are parsed, the contents from the cloud are downloaded to other dynamically-created temp dir and compared with the ones in use, replacing eventual dags or plugins that may have changed.

That is

1. creates new config vars called remote_dags_folder_conn_id & remote_plugins_folder_conn_id so we can fetch the credentials for the cloud providers using similar mechanism that is currently used for remote logs
2. if DAGS_FOLDER or PLUGINS_FOLDER starts with s3:// , gs:// , etc then:
3. A temp dir is created locally
4. The files from the referred cloud path are downloaded into the temp dir and used as the dag_folder & plugins_folder reference for airflow
5. When dags folder is being parsed, the updated version is downloaded to a new dynamically created temp dir that is compared with the current temp dir used, and in the case of eventual changes the latest one overwrites the previous one where it is needed

Use case/motivation

First motivation is that it would make it easier to deal with airflow on kubernetes, where both the scheduler and worker need to have access to up-to-date dags & plugins folders and as of now it is not straightforward to set it up, specially considering that the current best approach involves gitSync, which sometimes may not work due to some blocks from the company's cluster (which is my case). By having those folders reachable from a cloud provider, airflow setup becomes easier as a whole.
Second motivation is that it is a very elegant way of decoupling airflow into infrastructure & components. Many organisations have a unified git repo containing airflow dags & plugins plus infra-specific files (Dockerimage, Docker compose yaml file, .txt files listing libraries, etc), and it would be nice to at least have the possibility of separating that.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This is at most a discussion not feature. And taking the caliber of it, this is something that should be discussed in devlist https://lists.apache.org/list.html?dev@airflow.apache.org and result in Airflow Improvement Proposal - https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals - with all the details and consequences hashed out. Note that before attempting it, you should look at the previous attempts of proposing siumilar issues https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest - https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher (you can see linked discussions in devlist for them)

So far the history shows that those efforts were discontinued (those proposals are in DRAFT stage) - the original autors have not completed them enough to get it through voting and convince community to do so. But if you would like to have a complete proposal - you are absolutely welcome to start and lead discussion on that and hopefully get the Airflow Improvement Proposal that is voted, approved and then lead it to completion (this is how things work in the OSS project like airflow).

Due to the caliber of it however, it needs to be done via devlist and AIP - not by feature or even discussion in GitHub.