Reducing ETCD pressure from ResolutionRequest CRDs at scale

[vdemeester]

Problem

At scale, ResolutionRequest CRDs create significant ETCD pressure — independent of whether the resolved content is cached or not.

Each time a PipelineRun or TaskRun needs to resolve a remote Task or Pipeline, a ResolutionRequest custom resource is created in ETCD, reconciled by the resolver controller, updated with the result, and eventually garbage collected. That's at minimum 3 ETCD writes per resolution (create → update status → delete), even when the resolver cache returns an instant hit.

Based on feedback from customer engagements, we've seen environments running 80,000+ TaskRuns/hour (projected to grow 3×), where ETCD capacity — not compute — is the practical scalability ceiling. At that throughput, ResolutionRequest churn is a meaningful contributor to ETCD pressure.

What the current cache does (and doesn't do)

The resolver cache (introduced in v0.76 / late 2025, with several important fixes through early 2026) successfully reduces remote fetch latency and upstream load — on a cache hit, no network call is made.

However, the cache operates inside the resolver's Resolve() method, which is called by the ResolutionRequest reconciler. By the time the cache is consulted, the ResolutionRequest CRD has already been written to ETCD. The cache eliminates redundant fetches but does not eliminate the CRD round-trip.

In short: the cache solves latency, not ETCD pressure.

Possible directions

A few approaches could address this (not mutually exclusive):

1. Short-circuit resolution before CRD creation

Move the cache (or a layer of it) into the PipelineRun/TaskRun reconciler so that on a cache hit, no ResolutionRequest is created at all. This is the least disruptive change — the existing CRD-based flow remains for cache misses.

2. Non-CRD resolution path ("resolver v2")

Replace the ResolutionRequest CRD coordination mechanism with a direct HTTP/gRPC call from the pipeline controller to resolver endpoints. Resolution becomes a synchronous service call with built-in caching, eliminating CRD writes entirely. This is a larger architectural change but removes the fundamental coupling between resolution and ETCD.

Importantly, this doesn't have to be a big-bang migration. The CRD-based and non-CRD resolution paths can coexist — which path is used would be a configuration switch (e.g. a feature flag in the feature-flags ConfigMap). Operators can adopt the new path at their own pace, and the existing CRD-based flow remains available as a fallback. From the user's perspective (PipelineRun/TaskRun specs), nothing changes — it's purely an implementation detail of how resolution is dispatched.

3. Hybrid

Option 1 as a near-term improvement, option 2 as a longer-term evolution.

Why not bring back ClusterTask?

We've received requests to reintroduce a cluster-scoped Task type that bypasses resolvers entirely. While we understand the motivation (avoiding ResolutionRequest overhead for shared, well-known tasks), ClusterTask was removed deliberately and reintroducing it would:

• Re-create the namespace/cluster scope confusion that motivated its removal
• Only solve the problem for one specific use case (cluster-scoped tasks), not for git-resolved, bundle-resolved, or hub-resolved tasks
• Not address the underlying architectural issue

The right fix is to make resolution itself efficient at scale, regardless of resolver type.

Discussion

• Are others seeing ETCD pressure from ResolutionRequests at scale?
• Is there appetite for option 1 (cache short-circuit) as a near-term improvement?
• Should we pursue a TEP for option 2 (non-CRD resolution)?
• Are there other approaches worth considering?

[waveywaves]

I want to pursue a TEP for non-CRD resolution, I believe that makes more sense instead of using another CRD for resolving. It will offload the resolution completely to a different service and won't touch etcd as well.

[waveywaves]

To expand on my earlier comment, I think a TEP should compare reference architectures before assuming a particular transport or storage backend. Hatchet is interesting because it separates the orchestration engine, durable state, and workers. kube-shard takes a different approach: it preserves the Kubernetes API while routing selected API groups to a secondary API server backed by Kine and SQL.

For Tekton, the key boundary may be between state that is part of our public Kubernetes-native contract:
such as PipelineRun and TaskRun, and internal coordination state such as ResolutionRequest. The latter may not need to be represented as a CRD. A resolver service with its own cache or persistence layer could remove the ResolutionRequest lifecycle while leaving the user-facing Tekton API unchanged.

kube-shard looks useful as a platform-level mitigation because it moves writes away from the primary etcd, but it retains the create/update/delete churn and introduces a secondary API server, Kine, and database stack. I am therefore unsure whether it should become a dependency of Tekton itself.

The TEP should also quantify how much pressure comes specifically from ResolutionRequest objects versus other Tekton and Kubernetes resources before choosing an architecture.

@gbenhaim @chmouel , do you have pointers to other reference architectures or constraints we should consider, particularly around HA, retries, consistency, authentication/authorization, observability, and upgrades?

[afrittoli]

If option 1 turns out to be enough, is it worth it building option 2, which introduces a lot of additional complexity, in the code base and from a service deployment POV?

[vdemeester]

@afrittoli maybe, but it still put pressure on etcd (if cache is not enabled, on cache miss, especially at that scale) or for example by default for ClusterTask (that are not cached).

[afrittoli]

For large scale, not enabling the cache is not supported. Even with a service based solution, cache must be used at scale.
In terms of cache-miss, I would like to better understand if there are use cases where a miss is expected to be a significant e tough fraction of the requests to create a discernible pressure on etcd.
WRT to ClusterTask, is the lack of cache by design or not implemented yet? If the latter, the fix is easy.

[vdemeester]

For the cluster resolver, it is implemented, but not enable by default as it's not "automatic" as with digest for bundles or git sha for the git resolvers.. for cluster resolver, there is, at least today, no way for being sure you are fetching the latest version from the cache (without querying the API.. which defeat the purpose).

I agree for the cache setup, but at a large scale you also easily have multiple concurrent querying the same "object" that might be cached by the first query but not seen by the other ones right away 😓

[aThorp96]

1. Short-circuit resolution before CRD creation
   Move the cache (or a layer of it) into the PipelineRun/TaskRun reconciler so that on a cache hit, no ResolutionRequest is created at all. This is the least disruptive change — the existing CRD-based flow remains for cache misses.

This seems like the most tenable .However I think this understates the complexity required. This would change the caching API and break some of the separation of responsibilities since right now caching configuration is resolver-specific and the Pipelines Controller is by design ignorant of the resolvers and their implementation details. I believe in order for this to work, we would need to do one of two things, each with their own tradeoffs:
a) make caching configuration independent of the resolver, losing the cache: auto behavior.
b) import the individual resolvers into the pipelines controller, keeping the cache: auto behavior but breaking the extensibility of the resolvers and making caching unsupported for custom resolvers

[vdemeester]

Agreed, it also moves the "memory footprint" of the cache to the main core controller. But I agree, it kind of breaks a lot of assumption we make today.

[aThorp96]

Related finding here: since ResolutionRequests are not removed until after the PipelineRun is deleted, they can incur a pretty significant etcd footprint. Deleting the ResolutionRequests as soon as they're no longer needed could reduce the etcd footprint of a PipelineRun quite as much as 30% after the pipelineRun starts, which I think is likely the majority of a pipelineRun's lifetime: #10692

[aThorp96]

An additional thought here which would address the etcd load (in terms of storage) would be to keep the caching logic where it is, but move the cache storage layer to the cluster. This would allow a ResolutionRequest to reference a shared in-cluster object which the Pipelines Controller can fetch. The below diagram illustrates how it works now compared to how it might work to store the resolved objects in the cluster.

[vdemeester]

@aThorp96 in a nutshell (very naive nutshell), you suggest storing the object definition AS real object (CR) in the cluster right or in a wrapper object ? How would this reduce the etcd load ? it would create more objects right ? I guess then the pipeline controller could have a deterministic way to check if the in-cluster cache object is present or not and thus not creating a ResolutionRequest object altogether, right ?

[aThorp96]

you suggest storing the object definition AS real object (CR) in the cluster right or in a wrapper object ?

Yeah, in a wrapper object.

How would this reduce the etcd load ? it would create more objects right ?

It would result in more objects, but the objects would be much smaller. Right now, if a 1kb Pipeline is referenced and cached by 10 PLRs, even though only one remote fetch is made the pipeline ends up as 10kb in etcd (1kb duplicated into each of the 10 ResolutionRequest's status.data). With this change, The pipeline would be cached into one 1kb wrapper object in etcd, and each of the 10 ResolutionRequests status.data would be like 100b since they only store the reference to the 1kb wrapper object.

I guess then the pipeline controller could have a deterministic way to check if the in-cluster cache object is present or not and thus not creating a ResolutionRequest object altogether, right ?

Possibly, but I'm not sure how this would best work without the ResolutionRequests. There's a couple ways I think the Pipelines Controller could work without the intermediary ResolutionRequest, but it gets tricky since the Resolvers are designed to scale, cache, and run separately from the controllers.

a) ResolutionRequest caching is determined in the pipelines controller - requires the resolvers all be imported into the pipelines controller for caching checks. Caching only works for our resolvers (or a plugin system is introduced).
b) Resolvers have an /cached API endpoint which the Pipelines Controller hits before creating a ResolutionRequest, and returns either 404 (no cache hit), a cache key (a reference to a cached in-cluster wrapper object), or the full resolved data from the cache. Similarly, request routing is tough here; when there are multiple resolver replicas, which one receives the request and how do we ensure that's the one which would have cached the item in memory? If a custom resolver is running in a different namespace, is are they able to get this request at all? If not, they can't take advantage of the caching.
c) Pipelines Controller can get the cached-object directly If the cached wrapper-object's cache key is deterministic and regardless of the resolver, the pipelines controller can try and fetch the object via the kubeapi before creating the ResolutionRequest. This would work well I think, so long as there's a way to know which namespace the object will live in and custom resolvers can create their objects in that namespace. Creating the cache-object in the same NS as the PLR could undermine the cache's performance, but putting the objects in the tekton-pipelines or tekton-pipelines-resolvers namespace will require additional permissions for the custom resolver. In a cluster with a lot of cache misses this may cause more kubeapi reads and load, but maybe it could be disabled in configuration?

Option C seems like the simplest to me and also the best tradeoffs.