Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): Support using pipeline in exit handlers #8220

Merged
merged 8 commits into from Sep 1, 2022

Conversation

chensun
Copy link
Member

@chensun chensun commented Aug 31, 2022

Description of your changes:

  • Support using pipeline in exit handlers
  • Also fix bugs in component spec merging logic.

Checklist:

@chensun
Copy link
Member Author

chensun commented Aug 31, 2022

/retest

Copy link
Contributor

@zichuan-scott-xu zichuan-scott-xu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Chen, the test looks great to me!

)
exit_task_component_spec = exit_task_pipeline_spec.root
else:
raise RuntimeError
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should we add an error message? (e.g. Exit task is missing both container spec and pipeline spec)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.


with dsl.ExitHandler(exit_task, name='my-pipeline'):
print_op(message=message)
fail_op(message='Task failed.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, do we support nested exit handling? For example, there may be ways to handle a failed inner task but let the outer task run as usual.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. While I believe IR can possibly support nested exit handler, I'm not sure if Argo can support it.
So unless we have concrete use case/strong feature demand, I think we'll hold off adding such a feature that could potentially break symmetry between KFP and Vertex (again, this needs investigation on whether Argo can support it).

@google-oss-prow google-oss-prow bot added size/XL and removed size/L labels Aug 31, 2022
@zichuan-scott-xu
Copy link
Contributor

/lgtm

@chensun
Copy link
Member Author

chensun commented Sep 1, 2022

/approve

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@chensun
Copy link
Member Author

chensun commented Sep 1, 2022

/test kubeflow-pipelines-samples-v2

@google-oss-prow google-oss-prow bot merged commit d883582 into kubeflow:master Sep 1, 2022
@connor-mccarthy
Copy link
Member

/lgtm

Thank you for this, @chensun! And thanks for reviewing @zichuan-scott-xu.

@chensun chensun deleted the pipeline-as-exit-task branch September 1, 2022 20:17
jlyaoyuli pushed a commit to jlyaoyuli/pipelines that referenced this pull request Jan 5, 2023
* Support using pipeline in exit handlers

* release note

* remove dead code

* fix component spec merging flakiness

* address comments in PR#8209

* address review comments

* fix bad merge in release note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants