New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add drain option when canceling Dataflow pipelines #11374
Add drain option when canceling Dataflow pipelines #11374
Conversation
49efb51
to
98673d1
Compare
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
for job_id in job_ids: | ||
for job in jobs: | ||
requested_state = ( | ||
DataflowJobType.JOB_TYPE_STREAMING |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DataflowJobStatus.JOB_STATE_DRAINED
instead of DataflowJobType.JOB_TYPE_STREAMING
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great, thank you.
I know little about airflow, 2 clarifying questions:
- Would users generally decide on drain vs cancel when they are starting the operation? (Would they change their minds before onkill?)
- Would they want to wait until drain is completed? Drain may or may not fully complete in 5 minutes timeout window.
98673d1
to
8e5a516
Compare
Users define this parameter in the code which defines the DAG. AFAIK when DAG is running it is not possible to change this parameters afterwards. User can edit DAG and run it again with the new parameters.
I created separate PR to handle timeout, I think it is good idea to allow configure it so users could decide what timeout is suitable for them. |
8e5a516
to
8525bf9
Compare
Thanks @aaltay |
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
8e51958
to
07e3e33
Compare
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
a34b65e
to
ee87cda
Compare
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*. |
c701cd2
to
766a251
Compare
766a251
to
5a17511
Compare
* Add drain option when cancel Dataflow pipelines * fixup! Add drain option when cancel Dataflow pipelines * fixup! fixup! Add drain option when cancel Dataflow pipelines * fixup! fixup! fixup! Add drain option when cancel Dataflow pipelines
* Add drain option when cancel Dataflow pipelines * fixup! Add drain option when cancel Dataflow pipelines * fixup! fixup! Add drain option when cancel Dataflow pipelines * fixup! fixup! fixup! Add drain option when cancel Dataflow pipelines
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.