Reduce `Secret` watch pressure on seeds by splitting `ManagedResource` data

[rfranzke]

/area control-plane
/area scalability
/area performance
/kind poc
/label teamsize/medium

What is the topic about?:

Problem

gardener-resource-manager stores rendered manifests in Secret objects referenced by ManagedResource.spec.secretRefs. This was deliberate — ManagedResources can contain Kubernetes Secret resources that must be encrypted at rest in etcd.

However, this means all manifest data (Deployments, Services, RBAC, etc.) lives in Secrets, even when it contains nothing sensitive. On a seed cluster, this leads to:

• A large number of Secrets in etcd (roughly 1:1 with ManagedResources)
• Elevated memory and network I/O for every pod that watches Secrets (even if it only cares about a handful)
• Unnecessary etcd pressure from watch events on non-sensitive data

Proposed Solution

Split the ManagedResource data store by sensitivity:

1. Sensitive manifests (i.e., Kubernetes Secret resources) stay in Secret objects — encrypted at rest as today.
2. Non-sensitive manifests (everything else) move to a new dedicated custom resource (e.g., ManagedResourceData or similar) that is not in the Secret watch surface.

gardener-resource-manager would be adapted to read from both sources. During reconciliation, it already knows which resource types it is deploying, so the split can be determined at write time by the deploying controller.

Tasks

1. Audit Secret watchers on seeds: Identify which pods in seed clusters are watching Secret resources (e.g., shoot-dns-service in control plane namespaces, extension controllers, cert-manager, etc.). Measure the memory and network I/O impact of the current Secret volume on these pods. This provides the data to justify the change and quantifies the expected benefit.
2. Design the new resource type: Define a CRD for non-sensitive ManagedResource data. Decide on schema, naming, and namespace scoping.
3. Adapt gardener-resource-manager: Support reading manifests from both Secret refs (sensitive) and the new CRD refs (non-sensitive). Maintain backward compatibility with existing spec.secretRefs.
4. Adapt deploying controllers: Modify the controllers that create ManagedResources (e.g., in pkg/component/) to split manifests by sensitivity at write time.
5. Migration path: Ensure a smooth transition — existing ManagedResources with all data in Secrets must continue to work. Consider a phased rollout.

[plkokanov]

Hackathon Summary

Initial Measurements

There was not enough time during the hackathon to conduct a full audit of Secret watchers on seeds (task 1 from the issue). Following are some initial findings.

Using a small go tool that lists all ManagedResources, collects their secretRefs, and computes the in-memory heap size of each referenced Secret (approximating the cost in client-go's informer cache), produced the following:

Cluster	Shoots	ManagedResources	Secrets	In-memory size
1	193	11,753	12,453	204.90 MB
2	272	15,605	16,374	281.22 MB

Generally this leads to around 1 MB of secret data per shoot control plane. The data in the garden namespace and other non-shoot namespaces is negligible.

On average, this is more than half of secret data on seed clusters.

---

Additionally, all provider extensions in the gardener org were checked and it seems that they explicitly disable the controller-runtime cache for Secrets. The same is true for extensions like shoot-dns-service, registry-cache, and shoot-rsyslog-relp.

This leaves some control plane components that still have to be checked (mainly in control planes of Shoots backing ManagedSeeds) like the cert-management, external-dns-management, etc ...

Implementation

A sample implementation to check what changes would be necessary and how difficult adding such a feature would be was developed. The code is available here: plkokanov/gardener@04acab4.

A new CRD ManagedResourceData was introduced as the non-sensitive data store:

apiVersion: resources.gardener.cloud/v1alpha1
kind: ManagedResourceData
metadata:
  name: managedresource-blackbox-exporter
  namespace: garden
data:
  data.yaml.br: <brotli-compressed non-Secret manifests>

The ManagedResource spec was extended with dataRefs alongside the existing secretRefs:

spec:
  secretRefs:
  - name: managedresource-blackbox-exporter-abc123  # K8s Secret manifests only
  dataRefs:
  - name: managedresource-blackbox-exporter         # everything else (Deployments, RBAC, etc.)

The split is determined at write time by the deploying controller, as suggested in the issue. No changes to function signatures in pkg/utils/managedresources were required — the existing Create*/Update functions internally route data to either a Secret or ManagedResourceData based on content.

Registry (registry.go):
Objects are classified at serialization time using the generated filename which includes the GVK (e.g. v1, kind=secret__foo__my-secret.yaml). SerializedObjects() groups objects by whether the filename contains "secret" (this could be changed to group by wither the file contents for the resource kind, instead):

• Secret-kind objects → brotli-compressed under "data.yaml.br" (preserves immutable Secret hash stability)
• Non-secret objects → compressed under "plain.data.yaml.br"

Create/Update logic (managedresources.go):
A splitDataIntoSecretAndPlain function routes the data map based on key patterns:

• Keys matching "data.yaml.br" or containing "secret" or not containing plain. prefix (due to data coming from other utility functions) → K8s Secret → referenced via secretRefs
• Everything else → ManagedResourceData → referenced via dataRefs

For chart-rendering paths, raw YAML documents are inspected by parsing the kind field to separate Secret manifests from non-Secret manifests before compression.

A NewManagedResourceData builder was added mirroring NewSecret. Delete and GetObjects were updated to handle both reference types.

Results

On a local dev setup with a single-node cluster and one Shoot:

	Secrets	ManagedResourceData
Total	26	105
Shoot control plane	13	38

As can be seen, the vast majority of manifests moved out of Secrets into ManagedResourceDatas

Open Items

• More detailed analysis of impact of Secrets: Identify which controllers still enable watches and cache for Secrets and whether this feature is still worthwhile considering the implementation
• ManagedResourceData schema: Decide whether ManagedResourceData needs spec/status fields or if the flat key/value structure is sufficient
• Splitting heuristic: The current key-based classification (strings.Contains(key, "secret")) is fragile. A cleaner approach would be an explicit interface (e.g. DataProvider with GetSecretData()/GetData()) so callers declare intent rather than relying on naming conventions.
• Garbage collection: Decide on a strategy for ManagedResourceData lifecycle (currently mutable, unlike the immutable hashed Secrets)