feature: Webhook: read AgentRuntime CR for per-workload config merging at admission time

[cwiklik]

Feature Description

Summary

Extend the admission webhook to look up an AgentRuntime CR at Pod CREATE time and merge its configuration with platform defaults. This gives developers per-workload control over identity and observability settings without modifying the webhook's ConfigMap defaults.

Today the webhook reads configuration exclusively from the kagenti-webhook-defaults ConfigMap. With this change, the webhook resolves the Pod's owner workload, looks up a matching AgentRuntime CR, and merges the CR's spec fields over the ConfigMap defaults. If no AgentRuntime CR exists, the webhook continues to use ConfigMap defaults — no behavior change for existing workloads.

Context

The AgentRuntime CRD and controller are implemented in kagenti-operator. The controller applies kagenti.io/type labels and kagenti.io/config-hash annotations to the Deployment's PodTemplateSpec, which triggers a rolling update. The webhook intercepts the resulting Pod CREATE and needs the AgentRuntime CR to determine per-workload configuration.

Current config architecture

The webhook has a well-structured config system:

• config.PlatformConfig (internal/webhook/config/types.go) — struct covering images, proxy settings, resources, token exchange, SPIFFE, observability, and per-sidecar defaults
• config.ConfigLoader (internal/webhook/config/loader.go) — loads from a YAML file (mounted from ConfigMap), watches for changes, hot-reloads with debounce
• injector.PrecedenceEvaluator (internal/webhook/injector/precedence.go) — multi-layer precedence chain: feature gates → namespace labels → workload labels → CR overrides (stub) → platform defaults
• injector.PodMutator (internal/webhook/injector/pod_mutator.go) — applies the injection decision to the PodSpec using a ContainerBuilder

The PrecedenceEvaluator already accepts a tokenExchangeOverrides *TokenExchangeOverrides parameter — this is the designed extension point for CR-based overrides, currently passed as nil.

Proposed Solution

Required changes

1. Owner chain resolution — Pod to Deployment

File: new file, e.g. internal/webhook/resolver/owner_chain.go

The webhook receives a Pod admission request but the AgentRuntime CR's targetRef points to a Deployment (or StatefulSet). The webhook needs to walk the owner chain to find the workload:

• Read pod.OwnerReferences to find the ReplicaSet (for Deployments) or StatefulSet
• For ReplicaSet: read its OwnerReferences to find the Deployment
• For StatefulSet: the Pod's owner is the StatefulSet directly
• For Jobs/CronJobs: Pod → Job owner, Job → CronJob owner (if applicable)
• Return the resolved workload reference: {apiVersion, kind, name, namespace}
• Handle edge cases: Pod with no owner (skip), Pod with unexpected owner chain (skip gracefully)
• Add RBAC: get ReplicaSets in all namespaces

2. AgentRuntime CR lookup

File: new file, e.g. internal/webhook/resolver/agentruntime_resolver.go

• Define a lightweight AgentRuntime type (or import from kagenti-operator's api/v1alpha1) for deserializing the CR
• Query the API server (or cached informer) for AgentRuntime CRs in the Pod's namespace
• Match the AgentRuntime CR whose spec.targetRef matches the resolved workload (apiVersion, kind, name)
• Return the matched AgentRuntime CR or nil if not found
• Add RBAC: get/list AgentRuntime CRs (agent.kagenti.dev group, v1alpha1 version)

3. AgentRuntime informer cache

File: cmd/main.go or webhook setup

• Set up a shared informer for AgentRuntime CRs to avoid per-admission API calls
• The informer should watch all namespaces (AgentRuntime CRs can be in any namespace)
• Use the informer cache for lookups in the resolver (step 2)
• Ensure the cache is synced before the webhook starts accepting requests
• Consider memory impact: AgentRuntime CRs are small and expected count is low (hundreds, not thousands)

4. Config merging — AgentRuntime spec over PlatformConfig

File: new file, e.g. internal/webhook/config/merge.go

Merge AgentRuntime CR spec fields over the PlatformConfig loaded from the ConfigMap. The AgentRuntime CR is the higher-priority source — its fields override defaults where present.

• Define merge function: MergeWithAgentRuntime(base *PlatformConfig, cr *AgentRuntime) *PlatformConfig
• Merge identity fields (v1alpha1 scope):
  • spec.identity.spiffe.trustDomain → overrides PlatformConfig.Spiffe.TrustDomain
  • spec.identity.clientRegistration.realm → affects token exchange URL and client registration config
  • spec.identity.clientRegistration.adminCredentialsSecret → overrides the secret used for Keycloak admin access
• Merge trace fields (v1alpha1 scope):
  • spec.trace.endpoint → overrides OTEL exporter endpoint (passed as env var to sidecar)
  • spec.trace.protocol → overrides OTEL protocol
  • spec.trace.sampling.rate → overrides sampling rate
• Return a new *PlatformConfig — do not mutate the shared base config
• Fields not present in the AgentRuntime CR keep their ConfigMap default values

5. Wire it into the injection path

File: internal/webhook/injector/pod_mutator.go, internal/webhook/v1alpha1/authbridge_webhook.go

• In the webhook handler (Handle()), after decoding the Pod:
  1. Resolve the owner chain (Pod → workload)
  2. Look up AgentRuntime CR for the resolved workload
  3. If found: merge CR config with platform defaults
  4. Pass merged config to PodMutator
• Update PodMutator.InjectAuthBridge() to accept an optional merged config (or update its GetPlatformConfig to return the merged version for this request)
• Update PrecedenceEvaluator call: populate the tokenExchangeOverrides parameter from the AgentRuntime CR (replaces the current nil stub)
• Update ContainerBuilder if needed: it currently takes a *PlatformConfig at construction time — may need per-request config for merged values

6. Tests

Files: new test files + updates to existing tests

• Unit test: owner chain resolution
  • Pod owned by ReplicaSet owned by Deployment → returns Deployment
  • Pod owned by StatefulSet → returns StatefulSet
  • Pod with no owner → returns nil
  • Pod owned by unknown kind → returns nil gracefully
• Unit test: AgentRuntime CR lookup
  • CR exists matching targetRef → returns CR
  • No CR in namespace → returns nil
  • Multiple CRs, one matches → returns correct one
  • CR targets different workload → returns nil
• Unit test: config merging
  • CR with trustDomain override → merged config has new trustDomain
  • CR with trace endpoint override → merged config has new endpoint
  • CR with no overrides (empty spec) → merged config equals defaults
  • Merge does not mutate base config
• Integration test: full webhook flow
  • Pod with matching AgentRuntime CR → sidecars configured from merged config
  • Pod without AgentRuntime CR → sidecars configured from defaults only
  • AgentRuntime CR with partial overrides → only overridden fields differ
• make test passes

7. Documentation

• Update ARCHITECTURE.md to document the config merging flow
• Add sequence diagram: Pod CREATE → owner resolution → CR lookup → config merge → injection
• Document which AgentRuntime fields map to which PlatformConfig fields

Progress tracker

Phase 1: Resolution and lookup

• Owner chain resolution
• AgentRuntime CR lookup
• AgentRuntime informer cache
• Unit tests for resolution and lookup

Phase 2: Config merging

• Merge function
• Wire into injection path
• Unit tests for merging

Phase 3: Integration and docs

• Integration tests
• Documentation
• make test passes

Dependencies

Dependency	Repo	Notes
AgentRuntime CRD definition	kagenti-operator	Need the Go types or CRD schema to deserialize the CR
Webhook targets Pods at CREATE time	kagenti-extensions (this repo)	Owner chain resolution only works when the webhook receives Pod objects

Design decisions

Import vs. lightweight type: The webhook can either import the AgentRuntime types from the kagenti-operator Go module, or define a lightweight local struct with only the fields it needs. Importing ensures type consistency but adds a module dependency. A lightweight local type keeps the webhook independent but may drift. Recommend: import from kagenti-operator — the types are stable (v1alpha1) and the dependency is one-directional.

Informer vs. per-request API call: Use an informer cache. Per-request API calls add latency to every Pod admission (the webhook has a 10-second timeout). AgentRuntime CRs are small and low-cardinality — the informer memory cost is negligible.

Merge semantics: Field-level override, not full replacement. If the AgentRuntime CR specifies spec.identity.spiffe.trustDomain but not spec.trace.endpoint, the trust domain is overridden and the trace endpoint keeps the ConfigMap default. This matches the existing PlatformConfig overlay pattern used by the ConfigLoader (YAML file overlays compiled defaults).

Want to contribute?

• I would like to work on this issue.

Additional Context

No response