Repo contains sample code for deploying DAGs to Google Cloud Composer, dockerfile, and DAG validation. Details about the CI/CD process can found in this article: https://medium.com/p/f469a09c9db8
| Option Name | Positional Arguments | Description |
|---|---|---|
| get-stop-and-start-dags | n/a | Outputs a list of all the DAG ids that need to be started and stopped. |
| stop-dag | example_dag_4_v1_1_1 | Removes the specified DAG id from the Cloud Composer environment. |
| upload-plugins-and-variables | n/a | Uploads plugins and Airflow variables. |
| start-dag | example_dag_2_v1_1_1 | Adds the specified DAG id to the Cloud Composer. |
| Function Name | Description | Parameters |
|---|---|---|
| get-stop-and-start-dags | Outputs the list of DAGs that need to started and stopped. | n/a |
| handle_delete | Pause DAG, delete file from GCS, and delete from Airflow UI. | $1 DAG id |
| handle_new | Add new files to GCS and unpause DAG. | $1 DAG id |
| handle_variables | Upload and set Airflow variables. | n/a |
| handle_plugins | Upload Airflow Plugins. | n/a |
| wait_for_delete | Delete DAG from Airflow UI. | $1 DAG id |
| wait_for_deploy | Unpause the DAG. | $1 DAG id |
| validate_dags_and_variables | Validate all DAGs in running_dags.txt file using local Airflow environment. | $1 vairables-ENV.json |
| check_files_are_same | Get local and GCS file path and ensure, they are the same. | $1 DAG file name |
| validate_local_vs_gcs_dag_hashes | Compare the hash value of a file | $1 and $2 file path |
| gcs_md5 | Get hash value of a file | $1 file path |