fix: prevent Namespace informer cache errors and fallbacks#1686
Merged
EItanya merged 3 commits intokagent-dev:mainfrom Apr 21, 2026
Merged
fix: prevent Namespace informer cache errors and fallbacks#1686EItanya merged 3 commits intokagent-dev:mainfrom
EItanya merged 3 commits intokagent-dev:mainfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Disables controller-runtime’s cached client for Namespace objects to prevent cluster-scoped Namespace informer 403 loops in namespaced RBAC mode, and adds watched-namespace-based fallbacks where Namespace reads may be forbidden.
Changes:
- Configure manager client to bypass the informer cache for
corev1.Namespace. - Add watched-namespace fallback logic to namespace listing (
/api/namespaces) and internal-K8s-URL detection. - Improve behavior when
AllowedNamespaces.from=Selectoris used without Namespace read permissions (clear error) and add/extend related tests.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| go/core/pkg/app/app.go | Disables cached client for Namespace; passes watched namespaces into translator constructor. |
| go/core/internal/httpserver/handlers/namespaces.go | Adds Forbidden/Unauthorized fallback to configured watched namespaces; introduces helper for name-only responses. |
| go/core/internal/httpserver/handlers/handlers.go | Adds WatchedNamespaces onto handler base dependencies. |
| go/core/internal/httpserver/handlers/agents.go | Constructs translator with watched namespaces from handler base. |
| go/core/internal/httpserver/handlers/namespaces_test.go | Adds test covering fallback to watched namespaces when Namespace reads are forbidden. |
| go/core/internal/controller/translator/agent/adk_api_translator.go | Adds watched-namespace fallback in isInternalK8sURL; introduces new constructor variant. |
| go/core/internal/controller/translator/agent/proxy_test.go | Adds test ensuring proxy translation falls back to watched namespaces when Namespace reads are forbidden. |
| go/api/v1alpha2/common_types.go | Returns a clearer error when selector-based allowed namespaces can’t read namespace labels due to RBAC. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
EItanya
reviewed
Apr 20, 2026
Comment on lines
+419
to
+427
| Client: client.Options{ | ||
| Cache: &client.CacheOptions{ | ||
| // Prevent the cached client from starting a cluster-scoped | ||
| // Namespace informer. In namespaced RBAC mode a Role cannot | ||
| // grant access to cluster-scoped resources, so an informer | ||
| // list/watch would keep crashing and cannot be handled. | ||
| DisableFor: []client.Object{&corev1.Namespace{}}, | ||
| }, | ||
| }, |
Contributor
There was a problem hiding this comment.
Can this be optional based on the clusterRole setting? Otherwise caching this is definitely better
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
3098a15 to
6ef0e38
Compare
EItanya
approved these changes
Apr 21, 2026
Huimintai
pushed a commit
to Huimintai/kagent
that referenced
this pull request
Apr 21, 2026
…v#1686) ## Summary In namespaced RBAC mode, any code path that read a `Namespace` object via the cached client would lazily start a cluster-scoped list/watch informer. Since a Role cannot grant access to cluster-scoped resources, this informer repeatedly fails with a 403 error. The simple repro is listing namespaces from the UI and the following will show up repeated in the controller logs: ``` failed to list *v1.Namespace: namespaces is forbidden: User \"system:serviceaccount:kagent:kagent-controller\" cannot list resource \"namespaces\" in API group \"\" at the cluster ``` This PR disables controller-runtime's informer cache for `Namespace` objects, which automatically uses direct API reads, and thus we have the ability to catch authorization errors and use a fallback and prevent the error loop. When available, we use `watchedNamespaces` as a fallback for `isInternalK8sURL` and `/api/namespaces` call sites. For `AllowedNamespaces.from=Selector`, it returns a clear error since reading namespace labels requires cluster role. ## Testing The following scenarios have been manually tested: - Listing namespaces from the UI no longer causes UI and controller error - The above informer error spam is no longer there - Cross NS reference with `AllowNamespaces.from=All`, `AllowedNamespaces.from=Selector` (both allowed and rejected paths and unwatched namespace rejection) - `isInternalK8sURL` will fallback to `watchedNamespaces` during agent translation ## Alternatives Other than catching the error, I've also thought of using some mechanism to "inform" the controller if it has cluster wide access or not. I considered `watchedNamespaces` (which is always set if using namespaced RBAC) or creating another flag, but eventually both has some complex logic or edge cases troubles and gracefully handling errors / fallback seems the best solution to me right now. --------- Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In namespaced RBAC mode, any code path that read a
Namespaceobject via the cached client would lazily start a cluster-scoped list/watch informer. Since a Role cannot grant access to cluster-scoped resources, this informer repeatedly fails with a 403 error. The simple repro is listing namespaces from the UI and the following will show up repeated in the controller logs:This PR disables controller-runtime's informer cache for
Namespaceobjects, which automatically uses direct API reads, and thus we have the ability to catch authorization errors and use a fallback and prevent the error loop. When available, we usewatchedNamespacesas a fallback forisInternalK8sURLand/api/namespacescall sites. ForAllowedNamespaces.from=Selector, it returns a clear error since reading namespace labels requires cluster role.Testing
The following scenarios have been manually tested:
AllowNamespaces.from=All,AllowedNamespaces.from=Selector(both allowed and rejected paths and unwatched namespace rejection)isInternalK8sURLwill fallback towatchedNamespacesduring agent translationAlternatives
Other than catching the error, I've also thought of using some mechanism to "inform" the controller if it has cluster wide access or not. I considered
watchedNamespaces(which is always set if using namespaced RBAC) or creating another flag, but eventually both has some complex logic or edge cases troubles and gracefully handling errors / fallback seems the best solution to me right now.