Design: Conditional execution

[bobcatfish]

The work for this task is to design this feature and present one or more proposals (before implementing).

Expected Behavior

Pipelines need to be able to express Tasks that execute conditionally, e.g.:

• If a Task fails, do X (e.g. notify slack)
• If anything in the Pipeline fails, do Y (e.g. notify slack)
• Regardless of success/failure, do Z (e.g. cleanup)

Additionally (as pointed out by @abayer !) Pipelines may need more complex logic, e.g.:

• Only run this Task/follow this branch of tasks if we're not on a pull request (e.g. only publish an image, etc., from master branch)

This means we need to be able to express conditional execution generally against the state of the Pipeline. We can start with a specific subset, e.g.:

• What Tasks have run
• What Tasks have failed
• The state of the Resources available to this Task (i.e. where you'd get information such as the branch)

Actual Behavior

At the moment, execution of a Pipeline will continue to traverse DAG the user has expressed until either the Pipeline completes or one of the Tasks fails (a step exits with a non-0 code).

Additional Info

For other designs:

• Concourse conditional execution
• Argo's conditionals and exit handlers
• Jenkins pipeline cleanup
• Prow's RunAfterSuccess

This doc has a (very) rough start at expressing some of the requirements for this (note doc visible to members of knative-dev@googlegroups.com).

[bobcatfish]

One possibility: all Tasks and Pipelines could have a finally clause, which could be provided with params/env variables which indicate whether there was a success, failure, abort, etc. (vs. providing "on failure", "on success", and "finally" as separate concepts)

[abayer]

I think there's probably a need for both "do X after a Task or Pipeline if $SOMETHING is true" and "only run this Task or Pipeline if $SOMETHING is true before we start" - the most obvious use case that we see for the latter in Jenkins Declarative Pipelines is "only run this stage if we're not on a pull request" and the like. Having different environments, etc for PR runs does cut down on the need for that to some extent, but there's always going to be some fairly sizable chunks of work that people will want to do if-and-only-if some condition is met.

[bobcatfish]

Interesting point @abayer ! So it sounds like you would recommend that we make our requirements for this design a bit more general, e.g. instead of If a Task fails, do X (e.g. notify slack) it should be If Y is true, do X, with some mechanism to provide state information which includes things such as which Tasks have failed, succeeded, and maybe any other info available to a Pipeline (e.g. which git revision/branch is being used) ?

[abayer]

@bobcatfish - exactly! Otherwise, people will end up needing to have separate but nearly identical Pipelines for a PR build vs a master build etc. What other information needs to be available is a wider question, of course, and I've got a whole spiel I'm working up on extensibility in that area as well as others. =)

[bobcatfish]

Makes sense to me @abayer - I'll update the task description!

[jchesterpivotal]

We hit a similar PRs -> many pipelines problem in Concourse, which has been a standing pain point. The solution being worked on is "spaces". A resource can pump versions into spaces which don't need to be defined when the pipeline is set.

Take git commits, since PRs are the motivating case. Right now, a git resource follows a single branch. A PR resource can follow commits in PRs. But what I don't get is a separate pipeline, unless I either have (a) a shared pipeline for everything, (b) separate pipelines for PRs vs main or (c) tooling that creates and destroys pipelines on need.

None of these is particularly palatable. The first is confusing and difficult to set up, the second introduces duplication and busywork, the third is cool but requires an investment in tooling and testing that might not be commensurate with the value. Most CF teams settle on (b), given the relative rareness of PR-based activity. Teams with a PR-based workflow gravitate towards (c). Teams working in private repos enjoy the luxury of ignoring the problem altogether.

The space concept would essentially add (c) as a feature to Concourse itself. A git resource can create a "space" for each PR that lands, which essentially allows a parallel pipeline to be created on demand, but which shares its history with the overall pipeline it belongs to.

Other examples of spaces I've seen discussed are upstream dependency versions and IaaSes. So you might create a new space when a new Ruby interpreter version lands (a problem buildpacks faces for literally hundreds of upstream dependencies), or have spaces to corral a single pipeline that runs deployment, upgrade and smoke tests across multiple platforms (Master Pipeline has this problem).

[abayer]

@jchesterpivotal - yeah, Jenkins' multibranch Pipelines are basically (c). I should really get more familiar with Concourse. =)

[bobcatfish]

We hit a similar PRs -> many pipelines problem in Concourse, which has been a standing pain point.

The way we've been talking about handling this is a decoupling Resources from Pipelines (if we can), discussion in #200 - so you can use the same Pipeline with different Resources.

[bobcatfish]

another use case here that @fejta brought up yesterday:

conditionally executing based on which files changed in a commit, e.g. if you only changed markdown files, maybe you don't want to run your whole test suite

my gut reaction is that the theoretical "metadata" that is provided to each Pipeline's Task can include information like this, and the Pipeline's Task can express if there are certain file changes it is interested in or not.

[abayer]

@bobcatfish - hey, yeah, I had a note for myself scribbled down to try to understand what role "changelog"s could play in this world. Actually, that's part of a broader area I'm not clear on - historical data from run to run. I'm still getting my head around the CRD and API model here, so I don't know if we'll be recording run metadata (like, in this example, the commit hash for the git resource used in a particular run, which we'd need to calculate the changes in a subsequent run) in the PipelineRun's CRD or some other data store. Either way, that metadata definitely needs to be available to the Tasks - not just SCM info, but, say, whether the previous run was successful or not (people do like their "build fixed!" notifications, and you can't do that if you can't tell what the previous run's result was), etc...

[bobcatfish]

Actually, that's part of a broader area I'm not clear on - historical data from run to run

This is something that probably isn't clear in our current docs or plans, but you can see there is a hint of it in the fact that there is a runs section in the results configuration:

https://github.com/knative/build-pipeline/blob/4270f16403e90cfd9f900110f37a078736fdfd9c/examples/pipelineparams.yaml#L9-L12

The idea was that this is a location where we would persist a copy of all of the Runs (i.e. TaskRun, PipelineRun) and probably all of the related resources that were used to execute that Run.

• @tejal29 who is owning the results portion of the Pipeline design in general.

[abayer]

@tejal29 @bobcatfish - might be worth taking a look at Jenkins X's PipelineActivity CRD - see https://jenkins-x.io/architecture/custom-resources/ and https://github.com/jenkins-x/jx/blob/master/pkg/apis/jenkins.io/v1/types_pipeline.go. @rawlingsj can speak more to it - I'm just starting to dig in.