Add wait_policy option to EmrCreateJobFlowOperator.#44055
Add wait_policy option to EmrCreateJobFlowOperator.#44055vincbeck merged 1 commit intoapache:mainfrom adrian-adikteev:emr-job-flow-operator-add-wait-steps-completion
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
vincbeck
left a comment
There was a problem hiding this comment.
Could you cover this new use case in a unit test?
|
Good job! I like that version |
|
I'm not sure what's the next step here but from my side everything is ready. |
vincbeck
left a comment
There was a problem hiding this comment.
Overall looks good! Just one nit and you do not a newsfragment for this change
|
Should be ready for merge |
Possible values: - None: No wait (default) - WaitPolicy.WAIT_FOR_COMPLETION: Previous behaviour when wait_for_completion was True - WaitPolicy.WAIT_FOR_STEPS_COMPLETION: New behaviour - wait for the cluster to terminate.
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Currently, when using
wait_for_complition=Trueit will trigger the node to only wait until the cluster is successfully launched.I added the option
wait_policyso that one can chose what waiter will be used.Possible values:
None: No wait (default)WaitPolicy.WAIT_FOR_COMPLETION: Previous behaviour whenwait_for_completionwasTrueWaitPolicy.WAIT_FOR_STEPS_COMPLETION: New behaviour - wait for the cluster to terminate.This is useful because it will eliminate the need to have an extra node in your DAG.