Integrate custom tasks into Pipelines #3463

GregDritschler · 2020-10-27T14:02:21Z

Changes

A Pipeline Task now can reference a custom task. This causes the PipelineRun reconciler to create a Run instead of a TaskRun.
Pipeline parameters can be passed to the custom task. The custom task's results can be used by other Pipeline tasks and by the Pipeline results.

/kind feature

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices
Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

In package pkg/apis/pipeline/v1alpha1, the type of field RunStatusFields.Results has changed from
v1beta1.TaskRunResult to v1alpha1.RunResult.

GregDritschler · 2020-10-27T14:03:50Z

@imjasonh @vdemeester This PR supercedes Jason's original PR #3115. I made the following changes:

Rebased to latest master.
Resolved most of the TODOs.
Resolved the import cycle problem getting the v1beta1 PipelineRun status to include the v1alpha1 Run status.
Added code to propagate PR cancel request to Runs.
Added code to handle result references to results produced by custom tasks.
Improved unit test coverage for Runs.
Updated docs.

The main TODO which I did not address was to handle Conditions in pipeline tasks that use custom tasks. At the time of Jason's PR, I believe that "when" support was not yet merged. Now that "when" support is merged and Conditions are being deprecated, I would like to propose that Conditions NOT be supported in pipeline tasks that use custom tasks. If everyone agrees then I can update this PR to validate that restriction or I can do that in a follow-on PR.

Thanks in advance for your review.

tekton-robot · 2020-10-27T14:04:17Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

afrittoli

Thanks for this, I was really looking forward to it!
I'll be digging into the PR in the next days!

Quick question, without having looked at the code: would it make sense to split the part that solves the import cycle in a dedicated initial PR to reduce the size of this one?

tekton-robot · 2020-10-27T16:33:43Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

tekton-robot · 2020-10-27T17:11:47Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

vdemeester

We should move all Run related types to pkg/apis/run/v1alpha1 instead of just the one that were a problem for import cycle. This would allow us to still not import anything from "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1alpha1" but "github.com/tektoncd/pipeline/pkg/apis/run/v1alpha1" instead — similar to what we do with pkg/apis/resource/v1alpha1.

pkg/apis/run/v1alpha1/runstatus_types.go

imjasonh · 2020-10-28T12:58:09Z

Thanks so much for doing this Greg! 🎉

Now that "when" support is merged and Conditions are being deprecated, I would like to propose that Conditions NOT be supported in pipeline tasks that use custom tasks.

This seems reasonable to me, so long as it's documented and produces a descriptive error message. I'd prefer it to happen in a followup PR though to keep this one relatively narrowly scoped.

afrittoli · 2020-10-28T13:27:03Z

/test check-pr-has-kind-label

pkg/apis/run/v1alpha1/runstatus_types.go

tekton-robot · 2020-10-30T14:01:14Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

Review to redo, as discuss off

tekton-robot · 2020-10-30T14:47:44Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

tekton-robot · 2020-10-30T15:26:43Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	80.3%	81.0%	0.6

bobcatfish · 2020-11-02T16:39:34Z

@GregDritschler could you fill in the checklist:

Discussed in OWNERS meeting today, sounds like merging this fixes #3133 (correct me if I'm wrong @imjasonh @GregDritschler !)

GregDritschler · 2020-11-02T17:06:55Z

Discussed in OWNERS meeting today, sounds like merging this fixes #3133 (correct me if I'm wrong @imjasonh @GregDritschler !)

There are a few things left to do (as separate PRs since this one is already very big):

Disallow use of deprecated Conditions with custom tasks
Enable use of service account name and workspaces with custom tasks
Provide a sample custom task controller (I have started one)

tekton-robot · 2020-12-03T20:25:24Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/config/feature_flags.go	72.4%	71.0%	-1.4
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipeline_validation.go	99.5%	99.6%	0.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	75.0%	75.8%	0.8
pkg/reconciler/pipelinerun/cancel.go	94.1%	84.0%	-10.1
pkg/reconciler/pipelinerun/pipelinerun.go	84.6%	82.6%	-2.0
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go	91.9%	92.0%	0.1
pkg/reconciler/pipelinerun/resources/pipelinerunstate.go	71.0%	64.1%	-6.9
pkg/reconciler/pipelinerun/resources/resultrefresolution.go	91.9%	91.0%	-0.9
test/controller.go	25.8%	24.5%	-1.3

GregDritschler · 2020-12-03T21:35:36Z

I've opened issues for the follow-on work to this PR.

Custom Tasks: Disallow use of Conditions #3592
Custom Tasks: Allow use of workspaces #3593
Custom Tasks: Allow use of service accounts #3594
Custom Tasks: Support Pipeline Results #3595
Custom Tasks: Move all Run-related Go types to pkg/apis/run #3596

@vdemeester @afrittoli @pritidesai FYI. I believe I've resolved all the review issues. If not let me know. I'd like to get this one closed out.

vdemeester

/meow
Let's get this in for 0.19 😝

tekton-robot · 2020-12-04T06:47:13Z

@vdemeester:

In response to this:

/meow
Let's get this in for 0.19 😝

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot · 2020-12-04T06:47:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afrittoli

Thanks for the updates and for the great work on this!

There are a few more comments that we could add to into the code to track design decisions for posterity, but nothing blocking.

/lgtm

vdemeester · 2020-12-04T09:11:45Z

arf, @GregDritschler needs yet another rebase, sorry 😓 🙇

tekton-robot · 2020-12-04T13:37:45Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/config/feature_flags.go	72.4%	71.0%	-1.4
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipeline_validation.go	99.5%	99.6%	0.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	74.6%	75.4%	0.8
pkg/reconciler/pipelinerun/cancel.go	94.1%	84.0%	-10.1
pkg/reconciler/pipelinerun/pipelinerun.go	84.6%	82.6%	-2.0
test/controller.go	25.8%	24.5%	-1.3

A PipelineTask now can reference a custom task. This causes the PipelineRun reconciler to create a Run instead of a TaskRun. Pipeline parameters can be passed to the custom task. The custom task's results can be used by other Pipeline tasks and by the Pipeline results.

tekton-robot · 2020-12-04T13:50:13Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/config/feature_flags.go	72.4%	71.0%	-1.4
pkg/apis/pipeline/v1alpha1/run_types.go	22.7%	43.8%	21.0
pkg/apis/pipeline/v1beta1/pipeline_validation.go	99.5%	99.6%	0.0
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	74.6%	75.4%	0.8
pkg/reconciler/pipelinerun/cancel.go	94.1%	84.0%	-10.1
pkg/reconciler/pipelinerun/pipelinerun.go	84.6%	82.6%	-2.0
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go	91.9%	92.0%	0.1
pkg/reconciler/pipelinerun/resources/pipelinerunstate.go	71.0%	64.1%	-6.9
pkg/reconciler/pipelinerun/resources/resultrefresolution.go	92.6%	91.0%	-1.6
test/controller.go	25.8%	24.5%	-1.3

GregDritschler · 2020-12-04T14:17:34Z

@afrittoli @vdemeester Rebase completed.

afrittoli

Thanks!
/lgtm

tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Oct 27, 2020

tekton-robot requested review from dlorenc and a user October 27, 2020 14:02

tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2020

afrittoli reviewed Oct 27, 2020

View reviewed changes

GregDritschler force-pushed the pipelinerun branch from 9df4757 to 535913a Compare October 27, 2020 16:31

GregDritschler force-pushed the pipelinerun branch from 535913a to 07a9932 Compare October 27, 2020 17:09

vdemeester previously requested changes Oct 28, 2020

View reviewed changes

pkg/apis/run/v1alpha1/runstatus_types.go Show resolved Hide resolved

imjasonh reviewed Oct 28, 2020

View reviewed changes

pkg/apis/run/v1alpha1/runstatus_types.go Show resolved Hide resolved

GregDritschler force-pushed the pipelinerun branch from 07a9932 to 6e7a4ac Compare October 30, 2020 13:59

GregDritschler requested a review from vdemeester October 30, 2020 14:12

GregDritschler force-pushed the pipelinerun branch from 6e7a4ac to ccd82b9 Compare October 30, 2020 14:45

GregDritschler force-pushed the pipelinerun branch from ccd82b9 to 5dec217 Compare October 30, 2020 15:24

afrittoli added this to the Pipelines v0.18 milestone Oct 30, 2020

dibyom mentioned this pull request Nov 2, 2020

Implement Custom Tasks #3133

Closed

pritidesai modified the milestones: Pipelines v0.18, Pipelines v0.19 Nov 6, 2020

vdemeester approved these changes Dec 4, 2020

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 4, 2020

afrittoli reviewed Dec 4, 2020

View reviewed changes

tekton-robot assigned afrittoli Dec 4, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2020

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2020

Create Run resources from PipelineTask references

16aef05

GregDritschler force-pushed the pipelinerun branch from e43a794 to 4004bfe Compare December 4, 2020 13:34

tekton-robot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 4, 2020

GregDritschler force-pushed the pipelinerun branch from 4004bfe to d1dfa4b Compare December 4, 2020 13:47

afrittoli reviewed Dec 4, 2020

View reviewed changes

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2020

tekton-robot merged commit 75f3776 into tektoncd:master Dec 4, 2020

GregDritschler mentioned this pull request Dec 4, 2020

Do not allow use of deprecated Conditions with custom tasks #3601

Merged

4 tasks

Tomcli mentioned this pull request Jan 12, 2021

Custom Tasks: question with taskSpec #3682

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate custom tasks into Pipelines #3463

Integrate custom tasks into Pipelines #3463

GregDritschler commented Oct 27, 2020 •

edited

Loading

GregDritschler commented Oct 27, 2020

tekton-robot commented Oct 27, 2020

afrittoli left a comment

tekton-robot commented Oct 27, 2020

tekton-robot commented Oct 27, 2020

vdemeester left a comment

imjasonh commented Oct 28, 2020

afrittoli commented Oct 28, 2020

tekton-robot commented Oct 30, 2020

tekton-robot commented Oct 30, 2020

tekton-robot commented Oct 30, 2020

bobcatfish commented Nov 2, 2020

GregDritschler commented Nov 2, 2020

tekton-robot commented Dec 3, 2020

GregDritschler commented Dec 3, 2020

vdemeester left a comment

tekton-robot commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

afrittoli left a comment

vdemeester commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

GregDritschler commented Dec 4, 2020

afrittoli left a comment

Integrate custom tasks into Pipelines #3463

Integrate custom tasks into Pipelines #3463

Conversation

GregDritschler commented Oct 27, 2020 • edited Loading

Changes

Submitter Checklist

Reviewer Notes

Release Notes

GregDritschler commented Oct 27, 2020

tekton-robot commented Oct 27, 2020

afrittoli left a comment

Choose a reason for hiding this comment

tekton-robot commented Oct 27, 2020

tekton-robot commented Oct 27, 2020

vdemeester left a comment

Choose a reason for hiding this comment

imjasonh commented Oct 28, 2020

afrittoli commented Oct 28, 2020

tekton-robot commented Oct 30, 2020

tekton-robot commented Oct 30, 2020

tekton-robot commented Oct 30, 2020

bobcatfish commented Nov 2, 2020

GregDritschler commented Nov 2, 2020

tekton-robot commented Dec 3, 2020

GregDritschler commented Dec 3, 2020

vdemeester left a comment

Choose a reason for hiding this comment

tekton-robot commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

afrittoli left a comment

Choose a reason for hiding this comment

vdemeester commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

tekton-robot commented Dec 4, 2020

GregDritschler commented Dec 4, 2020

afrittoli left a comment

Choose a reason for hiding this comment

GregDritschler commented Oct 27, 2020 •

edited

Loading