Promote Remote Resolution to beta

[ghost]

Feature request

Remote resolution is currently behind the alpha feature gate in Tekton Pipelines. This issue is intended to track the work required to promote resolution into beta, removing the feature gate.

Use case

Platforms and users would like to access Tekton resources in locations other than OCI registries or the cluster their taskruns and pipelineruns execute in.

Criteria for Promotion

• Poll Pipeline maintainers for their preferred direction with resolution: integrate with pipelines + add out-of-box resolvers, keep entirely separate, go a different direction entirely.
• Once ^ is done, document the rest of the criteria for promotion here.

[ghost]

Potential for integrating remote resolution as a direct feature of Tekton Pipelines

This comment outlines improvements to the system design of remote resolution if Pipelines directly integrated the functionality into its codebase.

---

Simpler code

A lot of resolution's codebase exists to provide abstractions and helpers that hide the resolution specifics from clients (like Pipelines) and resolvers (like the git resolver). Examples of these are:

• The framework.Resolver interface for resolvers to implement, which hides a lot of the underlying knative controller "stuff". (Example of implementation here)
• The resource.GenerateDeterministicName helper for pipelines to use when creating ResolutionRequests so that the name of a request remains constant across multiple reconciles of the same taskrun or pipelinerun. This is the mechanism by which we could introduce "caching" by adding ownerRefs to existing ResolutionRequest objects instead of creating new ones.
• The resource.Requester interface for pipelines to use to submit resolution requests, allowing easier mocking and stubbing of resolver behaviour in pipeline's tests.
• The resource.Request and resource.OwnedRequest interfaces that are designed to give a client like pipelines more control over the resolution process without exposing much of the machinery.
• ... and so on ...

A lot of this code exists in service of the idea that Tekton Resolution could be leveraged by clients other than Tekton Pipelines. But without concrete examples of other clients this abstraction is unnecessary and probably just confusing. Integrating remote resolution's concepts directly into Pipelines would likely mean a lot of this could be removed entirely.

... and easier dependencies

Currently Tekton Resolution shares a lot of the same dependencies as Tekton Pipelines. Examples include knative, go-containerregistry + k8schain, various k8s.io modules, etc. Since Tekton Pipelines depends on Resolution for some of its client code pulling in updates from resolution can cause a bit of churn in Pipeline's go.mod. This would be eliminated entirely if Resolution shared the same go.mod with Pipelines.

---

Re-using existing concepts

Reduced to its essence a "resolver" is really the same thing as Pipeline's CustomTask concept. The only real differences are that resolvers have a fixed interface (the ResolutionRequest CRD) and a tiny supporting SDK that developers can use to get started quickly without knowing anything about controllers or webhooks: Writing a basic resolver starts with one go file less than 90 lines and one yaml file for its deployment.

Given the similarity between Resolvers and CustomTasks it would make sense to eliminate Resolvers entirely (while still enforcing the specific CRD and helpful "sdk" parts that are useful to make remote resolution work).

Duck typing would be another approach: register a shape of object that Pipelines will accept for remote resolution and allow resolvers to meet that shape however they want.

---

Changes in system design

How resolution works today

Currently Tekton Resolution maintains a "core" reconciler whose responsibility is to monitor all ResolutionRequests and does bookkeeping: marks them Succeeded, initializes status and enforces global timeouts.

Here is approximately the order of operations today for a new ResolutionRequest:

sequenceDiagram
    Tekton Pipelines->>ResolutionRequest: CREATE request for git resource
    ResolutionRequest-->>Tekton Resolution: WATCH triggers reconcile
    Tekton Resolution->>ResolutionRequest: UPDATE with initial status
    ResolutionRequest-->>GitResolver: WATCH triggers reconcile
    GitResolver->>Git Repo: Clone & get file contents
    GitResolver->>ResolutionRequest: PATCH with contents of file from git
    ResolutionRequest-->>Tekton Resolution: WATCH triggers reconcile
    Tekton Resolution->>ResolutionRequest: UPDATE with Succeeded status
    ResolutionRequest-->>Tekton Pipelines: WATCH triggers reconcile of pipelines

... and this is the simplest case where a ResolutionRequest succeeds. The sequencing expands a lot once it starts to account for things like managing and configuring global timeouts of requests, caching responses correctly, eliminating duplicate requests for the same resource, and so on.

How it could work if pipelines did remote resolution itself

In a world where Tekton Pipelines merges remote resolution directly into its codebase, the basic flow of control would be simplified quite a bit:

  sequenceDiagram
    Tekton Pipelines->>ResolutionRequest: CREATE request for git resource w/ initial status
    ResolutionRequest-->>GitResolver: WATCH triggers reconcile
    GitResolver->>Git Repo: Clone & get file contents
    GitResolver->>ResolutionRequest: UPDATE with contents of file + Succeeded status
    ResolutionRequest-->>Tekton Pipelines: WATCH triggers reconcile of pipeline

[chitrangpatel]

/assign

[vdemeester]

Note: we also need to support Remote Resolution for Task specs.

[lbernick]

Potential improvement for remote resolution: If you try to use remote resolution without both the resolution project and the correct resolver installed, your pipelinerun will just hang until it times out. I'm not sure if it's possible for the pipelinerun/taskrun reconcilers to know if you have resolution installed, or for the resolution reconciler to know if you have the correct resolvers installed, but if so it would be helpful to return an error in these cases instead.

[chitrangpatel]

Potential improvement for remote resolution: If you try to use remote resolution without both the resolution project and the correct resolver installed, your pipelinerun will just hang until it times out. I'm not sure if it's possible for the pipelinerun/taskrun reconcilers to know if you have resolution installed, or for the resolution reconciler to know if you have the correct resolvers installed, but if so it would be helpful to return an error in these cases instead.

At the API WG call we discussed the possibility of integrating tekton resolution into pipeline. I dont think anyone at the WG expressed ideas against it. If anything, the team wanted to merge the two projects. Since many team members were away this week, the decision was not finalized (I still need to send out an email to the maintainers of pipeline). But, if we do take that route and merge the two projects, we might not need to worry about this?

[lbernick]

Potential improvement for remote resolution: If you try to use remote resolution without both the resolution project and the correct resolver installed, your pipelinerun will just hang until it times out. I'm not sure if it's possible for the pipelinerun/taskrun reconcilers to know if you have resolution installed, or for the resolution reconciler to know if you have the correct resolvers installed, but if so it would be helpful to return an error in these cases instead.

At the API WG call we discussed the possibility of integrating tekton resolution into pipeline. I dont think anyone at the WG expressed ideas against it. If anything, the team wanted to merge the two projects. Since many team members were away this week, the decision was not finalized (I still need to send out an email to the maintainers of pipeline). But, if we do take that route and merge the two projects, we might not need to worry about this?

This would definitely solve the problem of the user needing to install the resolution project separately, but it would still be nice to return an error if they are trying to use a resolver they don't have installed.

[vdemeester]

At the API WG call we discussed the possibility of integrating tekton resolution into pipeline. I dont think anyone at the WG expressed ideas against it. If anything, the team wanted to merge the two projects. Since many team members were away this week, the decision was not finalized (I still need to send out an email to the maintainers of pipeline). But, if we do take that route and merge the two projects, we might not need to worry about this?

I think this is not about where the code lives but how the api/controller behave. Today, if you submit a PipelineRun (or TaskRun) with a non-existent resolver (resolver: foo-is-bar), it will hang out for the pipelinerun timeout (so by default 1h).. Although it could make sense as, maybe the resolver will be available in that time, I don't think we handle that case today. So having a very small timeout for the resolver to answer and timeout otherwise would make sense yes. And this is indepedent of where the code lives 👼🏼

[abayer]

Yeah, ResolutionRequestSpec needs a timeout field for sure. I feel like it'll make more sense to add that than to hack together timeout behavior on the PipelineRun/TaskRun reconciler side.

[abayer]

Oh wait, https://github.com/tektoncd/resolution/blob/408cd52add36156c4cbbb0819404ea9daee1cfc2/pkg/reconciler/resolutionrequest/resourcerequest.go#L59-L61 - there is a global timeout of 1 minute currently for ResolutionRequests, so we do specifically need PipelineRun/TaskRun reconciler logic to deal with that. I'll get on that now.

[abayer]

Ok, another thing Resolution needs - unit tests. The reconciler has none, so while I've verified that a PipelineRun will fail if the ResolutionRequest fails, I can't actually verify that the ResolutionRequest will fail!

[chitrangpatel]

I'm summarizing the action items here (cc @jerop @vdemeester @abayer @lbernick @dibyom @sbwsg): Feel free to suggest additional items and correct me if I've made an error summarizing the items 😄 .

Criteria for Promotion

• Poll Pipeline maintainers for their preferred direction with resolution: integrate with pipelines + add out-of-box resolvers, keep entirely separate, go a different direction entirely.
  • This is likely heading towards integrating resolution with pipelines
  • What about resolvers? They seem custom and can be created according to use cases. Should they be integrated into pipelines as well?
• Unit tests for Resolution. This is important before merging the two projects so that the resolution codebase is not under-tested.
• Somehow error-out the pipelinerun/taskrun reconcilers when the resolution project (after merging into pipelines, this is irrelevant) or the resolvers are not installed (see context).
• Prioritize and tackle Issues in the resolution project. Note that it makes sense to go after some of those after integrating resolution into pipeline. Once integrated, those issues can be transferred to the pipeline repo.
• Dogfooding remote resolution.
• Collect feedback on usage from the users and solve hidden bugs over <?> extended
  time

[vdemeester]

What about resolvers? They seem custom and can be created according to use cases. Should they be integrated into pipelines as well?

Some should probably (cluster task / pipeline, maybe the git resolver), but most might be better off living outside indeed.

Somehow Inform the pipelinerun/taskrun reconcilers that the resolution project (after merging into pipelines, this is irrelevant) or the resolvers are installed (see #4710 (comment)).

I don't think it's about "informing" but just about "erroring" out properly in case we are trying to use a resolver that doesn't exists. We might even want to do this in the admission controller.

• Custom tasks : Dogfooding.

Do you mean "Remote resolution" here instead of Custom Task ?

[chitrangpatel]

What about resolvers? They seem custom and can be created according to use cases. Should they be integrated into pipelines as well?

Some should probably (cluster task / pipeline, maybe the git resolver), but most might be better off living outside indeed.

I agree.

Somehow Inform the pipelinerun/taskrun reconcilers that the resolution project (after merging into pipelines, this is irrelevant) or the resolvers are installed (see #4710 (comment)).

I don't think it's about "informing" but just about "erroring" out properly in case we are trying to use a resolver that doesn't exists. We might even want to do this in the admission controller.

• Custom tasks : Dogfooding.

Do you mean "Remote resolution" here instead of Custom Task ?

Yes, I meant remote resolution. I updated it.

[abayer]

@jerop and I chatted about the resolvers question on Friday - I'd advocate for moving the existing git and bundle resolvers into Pipeline, along with an eventual catalog resolver, since those all seem pretty fundamental. But additional resolvers going forward (like the GitHub-specific one I intend to write eventually, inspired by tektoncd/resolution#42) should, IMO, be outside the Pipeline repo.

That does bring up another matter relating to the catalog - we do need a way to provide catalogs of custom tasks and resolvers in the catalog. But that's not relevant to Remote Resolution going to beta. =)

[vdemeester]

Added this to the v1 project as I would really want to have this in beta at least so that we can deprecate taskref.bundle and rely on remote resolver for oci bundle.

[abayer]

Oops, shouldn't have said closes on that PR - we haven't actually promoted to beta yet, we've just folded everything from https://github.com/tektoncd/resolution into the Pipeline repo and deployment. Reopening!

[afrittoli]

I'm rather late to the discussion here, but I still wanted to mention that something that I think we should have in remote resolution is support for bundles, i.e. a way for a parent resource to use "relative references" for children, as long as they belong to the same bundle.

That was one of the most requested changes to the OCI bundles feature that we have today.
The only place where we have this feature today is in-cluster resources - a Pipeline may reference to Tasks without specifying where the Tasks come from, it's a relative reference which implies "the same place where the Pipeline comes from i.e. the cluster.

This is something we can add after the beta as well, as I don't expect it would really require changes to the API surface, but I wanted to add it here to start gathering feedback about it.

[abayer]

Interesting - that'll need new syntax for "this Task needs to be resolved, so here's where to find it based on its name/type/version (for the Hub) and the Pipeline's resolution parameters".

[vdemeester]

Could it be something the resolver support itself ? For example, a Pipeline in a bundle could use the bundle resolver that would make the bundle resolver resolving the Pipeline to also resolve the Task from the same bundle, in case the bundle parameter is missing, or is using a special syntax ? We could even envision that the resolver, when seeing "normal" TaskRef could list all the bundle layers (aka resources), find the in-the-bundle referenced Task and mutate the Pipeline to use the bundle resolver instead.

In a gist, I think it is a feature on the resolvers themselves, and not necessary on "tekton resolution", and I don't think it should even bubble in the API at all.

[abayer]

So this is a bunch of interesting questions!

I'm pretty sure (I'm going to add e2e tests for this today) that if you've got a resolver reference for a PipelineTask inside a Pipeline that gets resolved remotely, it'll work just fine. So what's "missing" is automatically resolving Tasks without them needing to have an explicit resolver - automatically detecting when we've got an unresolved Task inside a resolved Pipeline and trying to resolve that Task using the same method as the Pipeline.

It gets a bit more complex for resolvers other than bundles - there, if we already know the OCI bundle containing both the Pipeline and the Task, as well as the name of the Task, we've already got the information we need. So doing something like Vincent's talking about where we first try to resolve some-task within bundle-pipeline inside the same bundle that the Pipeline came from should be doable implicitly, same with the cluster resolver (assuming the Task is in the same namespace as the Pipeline).

The hub and git resolvers don't seem to have such a simple trick, though - hub needs a version for the Task, not just the name, and git needs to know the path to the resource in the repo. But I don't think that should hold up implementing this behavior for bundles and cluster - @afrittoli, could you open an issue for this?

[abayer]

(and down the road, it may make sense for a PipelineRun to be able to say "here are the default org/repo (or URL for the anonymous clone approach) and revision for all Pipelines and Tasks within this PipelineRun which are marked as to be resolved by the git resolver", so that all that needs to be specified on the pipelineRef and the PipelineTasks would be the pathInRepo for the individual Task)

[abayer]

Also, I keep thinking that it may make sense to eventually add real fields (with relevant structs) to ResolverRef for the built-in resolvers, so that you can just do

pipelineRef:
  git:
    org: foo
    repo: bar
    revision: main
    pathInRepo: a/b/c.yaml

Rather than the more elaborate generic params approach of

pipelineRef:
  resolver: git
  params:
  - name: org
    value: foo
  - name: repo
     value: bar
  - name: revision
     value: main
  - name: pathInRepo
    value: a/b/c.yaml

We could only support the git: ... syntax for built-in resolvers, but for those, we'd be able to do admission control-time validation, a simpler syntax, PipelineRun-wide defaults, etc.

I'm gonna open an issue for this. =)