provide a pipeline concurrency limit

[jstrachan]

Expected Behavior

Its common in CI systems to limit the number of concurrent pipelines that can execute on a given repository and branch. e.g. process PRs concurrently, but only allow a maximum of 1 release to be performed at once in case releases clash with each other. e.g. to avoid race conditions between concurrent pipeline steps that operate on shared git repositories/buckets/kubernetes clusters.

e.g. imagine a simple pipeline of

• get the next incrementing version number
• spend some time building artifacts/images
• use kubectl to deploy some resources using this version number or update the website/changelog

If you run this pipeline concurrently all kinds of things could happen due to the wonders of concurrency (e.g. seeing the version number go forwards then backwards).

When working on separate PRs concurrency is not usually an issue; but working on shared resources (e.g. producing a sequential stream of artifacts or updating a shared cluster) we often want to force a clean ordering on the pipelines to avoid confusion or worse things.

Actual Behavior

There's currently no way to force a pipeline to not execute until all other pipelines for that repository + branch have completed without writing some kind of leader election step.

We're pondering writing a little leader election step as a workaround (which would be Jenkins X specific jenkins-x/jx#5471); but figure it would be nice to be able to add this kind of capability into the tekton controller.

If you squint its a little like the tekton controller being like the ReplicaSet controller; if replicas = 1 for a unique string (e.g. the git repository URL + branch name), only start a new Pod for a PipelineRun when no others are running for that string.

Steps to Reproduce the Problem

1. run lots of pipelines like the above example and watch the version number go up and down

Additional Info

If there was some kind of MaximumConcurrency for a specific source repository and branch we could modify the tekton controller to only create a new Pod when it knows there are no other running pods for a given source repository + branch.

[vdemeester]

/kind feature
/kind question

I wonder if this should be a feature for pipeline (core) or some integration tooling (like jenkins-x)

[bobcatfish]

This sounds like it could be pretty cool!! I agree with @vdemeester in that I'm not 100% sure where it would best belong - but there have been some related ideas that have come up lately (but are slightly more complicated) such as:

• Keeping track of how much $ a Pipeline is costing (how many resources it's consuming) and limiting how frequently it can run as a result

I think it could be pretty cool to think about what it would be like to apply generic policies to Pipeline execution, maybe in an admission controller, so it could be decoupled from the Tekton Pipelines codebase but could be ultra flexible.

[assertion]

Any updates about this issue? @bobcatfish @vdemeester
We are alse facing this situation when switching repo based CI/CD pipelines to tekton.

[bobcatfish]

Hey @assertion ! I don't think there has been any movement but if you (or anyone else in the community) wants to take this on I think we'd be happy to see it! Let me know if you want any pointers re. next steps to work on this.

[Fabian-K]

I looked into the approach using an admission controller. To ensure a pipeline has no concurrent executions, I registered a validating admission controller for creation of taskruns. As the incoming AdmissionReview contains the taskrun definition, I can query for other running pipelines and as a result return either true or false.

This works fine as the tekton controller retries creating the taskrun after it was rejected.

As the retry seems to follow some exponential backoff pattern, I´m a bit worried that if this is requested multiple times, a lot of time might be wasted. Any ideas about that? Do you know where I can find details about the backoff pattern?

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bobcatfish]

I'm gonna reopen this one, I wonder if we should add it to the roadmap also 🤔

/reopen

[tekton-robot]

@bobcatfish: Reopened this issue.

In response to this:

I'm gonna reopen this one, I wonder if we should add it to the roadmap also 🤔

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bobcatfish]

We're adding this to our roadmap, I changed the name a bit to reflect that we feel like we'd want to address this in general but maybe not specifically for branch + repo only

[imjasonh]

Some thoughts on how this could possibly work, feel free to propose alternatives:

• introduce a "concurrency bucket" CRD with a cap on task runs and/or resources, and/or something else

• have PipelineRuns and TaskRuns state what bucket they're counting against; possibly using an annotation?

• triggers could populate with some key based on repo+branch (or just repo, or org, etc.)

• PipelineRun controller holds runs in a concurrency-limited state until there's room in the bucket's cap

Open Questions:

• should items be unblocked FIFO? At random? Scheduled based on requests?
• how should this interact with existing K8s features for limiting resource usage in a namespace? Can operators use these features effectively today as a stopgap?

[bobcatfish]

Related issue: #2591

[afrittoli]

@jstrachan work on this is happening in experimental tektoncd/experimental#699, please continue the investigation / discussion on this there. Closing this one for now.