New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIP-5 Remote DagFetcher #9555
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
Hi, I wonder what is the status of this issue? I am interested in this. Can I for instance start to work on this or is it still to decide if we want this feature at all in airflow? |
I believe this should be discussed at the devlist on whether and how to implement this one. This is a big change to Airflow and the current consensus is that fetching Dags is "external" to Airflow - there are multiple solutions for fetching the DAGs (Git Sync, GCS/S3 sync, shared volumes, sync sidecars in Kubernetes etc. I think we are rather far from reaching common understanding and consensus on:
And last but not least - how it plays together with Dag Versioning. DAG versioning which is another AIP, much closer to being fully fleshed out and it is much closer to reach the consensus - it was dropped from 2.0 release only because we wanted to make sure we deliver 2.0 this year. Some of the questions there (especially atomicity of changes in several dependent files) are common between DAGFetcher and DAGVersioning and need to be answered together I believe. |
Closing as this is a discussion for the mailing list not for Github issue. |
Description
By allowing Airflow to fetch DAG files from a remote source outside the file system local to the service, this grant a much greater flexibility, eases implementation, and standardizes ways to sync remote sources of DAGs with Airflow.
Use case / motivation
deploying dag from remote (s3,git) location
Related Issues
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher
#3138
#8657
The text was updated successfully, but these errors were encountered: