Skip to content

Add shared/multi-tenant cluster note (oidc-reviewer collision)#2870

Open
pnalubandhu-a11y wants to merge 5 commits into
mainfrom
sa/openshift-shared-cluster-oidc-reviewer
Open

Add shared/multi-tenant cluster note (oidc-reviewer collision)#2870
pnalubandhu-a11y wants to merge 5 commits into
mainfrom
sa/openshift-shared-cluster-oidc-reviewer

Conversation

@pnalubandhu-a11y

Copy link
Copy Markdown

What

Adds a "Shared or multi-tenant clusters" note to the self-managed operator page.

Why

The chart creates cluster-scoped objects named after the release (e.g. <release>-oidc-reviewer). If two installs on the same cluster use the same release name, the second one fails with a Helm ownership error and it isn't obvious why. The docs currently assume a dedicated cluster.

Change

Short note under "Environment-specific considerations": use a unique release name per install, set app.internalJWTMap: [] when not needed, and install the operator with namespaceIsolation.enabled=true.

Testing

Reproduced the collision on a shared OCP 4.21 cluster — a second release named wandb failed with:
ClusterRoleBinding "wandb-oidc-reviewer" ... cannot be imported ... release-namespace must equal ...
A unique release name avoids it.

On a shared cluster, two W&B installs with the same release name collide on
the cluster-scoped <release>-oidc-reviewer ClusterRoleBinding. Add a short
note to use a unique release name (plus internalJWTMap and namespaceIsolation).
@pnalubandhu-a11y pnalubandhu-a11y requested a review from a team as a code owner July 6, 2026 21:13
@pnalubandhu-a11y

pnalubandhu-a11y commented Jul 6, 2026

Copy link
Copy Markdown
Author

Narrowing the ping. cc @wandb/delivery-tooling-team 🙂

Context: docs PR in the self-managed OpenShift series. Adds a shared/multi-tenant cluster note — two installs sharing a cluster collide on the cluster-scoped <release>-oidc-reviewer ClusterRoleBinding; remedies are a unique release name and app.internalJWTMap: []. Companions: #2869, #2872, #2874. @wandb/docs-team is the reviewer.

@mintlify

mintlify Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
wandb 🟢 Ready View Preview Jul 6, 2026, 9:39 PM

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

📚 Mintlify Preview Links

🔗 View Full Preview

📝 Changed (1 total)

📄 Pages (1)

File Preview
platform/hosting/self-managed/operator.mdx Operator

🤖 Generated automatically when Mintlify deployment succeeds
📍 Deployment: 0dd9224 at 2026-07-09 22:34:19 UTC

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

🔗 Link Checker Results

All links are valid!

No broken links were detected.

Checked against: https://wb-21fd5541-sa-openshift-shared-cluster-oidc-reviewer.mintlify.app

@johndmulhausen

Copy link
Copy Markdown
Contributor

📋 Validation of technical assertions

Every claim below was checked against the truth on the ground using Glean (Slack, Jira, Zendesk, GitHub) and Sourcegraph (the implementing Helm chart plus the vendored Helm ownership-validation code), then put through an adversarial verification pass.
5 assertions · ✅ 4 confirmed (1 qualified) · 🟡 0 partial · ⚠️ 0 reframed · ⛔ 0 could not validate · ❌ 1 refuted.
What didn't survive: the namespaceIsolation.enabled=true remedy is inverted — namespace isolation is the cause of this class of failure, not a fix, because it stops the operator's service account from managing cluster-scoped objects like the oidc-reviewer binding. That row is ❌ Refuted and blocking. The four naming/collision/internalJWTMap claims all hold against the chart source.

# Assertion in the docs Verdict Evidence (truth on the ground) Source
1 The chart creates cluster-scoped objects named after the release, specifically a <release>-oidc-reviewer ClusterRoleBinding · operator.mdx:335 ✅ Confirmed kind: ClusterRoleBinding with name: {{ .Release.Name }}-oidc-reviewer — object kind and <release>- naming are exact. The binding references the built-in ClusterRole system:service-account-issuer-discovery; the chart creates the binding, not that role. helm-charts:oidc.yaml#L13-L28
2 A second same-named install fails with the Helm ownership import error shown (... cannot be imported into the current release: ... key "meta.helm.sh/release-namespace" must equal ...) · operator.mdx:335,338-339 ✅ Confirmed Helm's own validator emits "%s exists and cannot be imported into the current release: %s", and checkOwnershiprequireValue emits key %q must equal %q: current value is %q for the meta.helm.sh/release-namespace annotation. The doc's text is a faithful (abbreviated) rendering of this exact code path. watchtower:validate.go#L83-L84; validate.go#L126-L134
3 Remedy: use a unique release/CR name per install to avoid the collision · operator.mdx:344 ✅ Confirmed The binding name is literally {{ .Release.Name }}-oidc-reviewer; a unique release name yields a unique object name, so no ownership conflict. This is a mechanical consequence of the template naming (see also the analogous shared-cluster ownership-conflict pattern in NO-2020). helm-charts:oidc.yaml#L17
4 Remedy: set app.internalJWTMap: [] to skip rendering the oidc-reviewer ClusterRoleBinding · operator.mdx:345 ✅ Confirmed (qualified) Binding is guarded by {{- if and .Values.app.internalJWTMap (gt (len .Values.app.internalJWTMap) 0) }} — an empty list is length 0, so the CRB is not rendered. The app. values path is correct (.Values.app.internalJWTMap). Guard added in helm-charts #525; shared-tenant deployment relies on the same trick (internalJWTMap: null). Qualifier: a reviewer (Daniel Panzella) flagged that len on this value could break native-Kubernetes-OIDC on-prem installs that legitimately need the binding — set it to [] only when internal JWT mapping is genuinely unused. helm-charts:oidc.yaml#L13; wandb/helm-charts#525; deployments:hacks.yaml#L1-L4
5 Remedy: install the operator with namespaceIsolation.enabled=true (listed as a way to "avoid this on a shared cluster") · operator.mdx:346 ❌ Refuted — resolve before merge Inverted: namespace isolation is the cause of cluster-scoped RBAC failures, not a fix. It only gates the operator's --isolation-namespaces arg and the operator's namespaced Role/RoleBinding (a different chart, operator); it never co-occurs with oidc-reviewer in helm-charts. With it on, the operator SA is scoped to one namespace and cannot manage the cluster-scoped oidc-reviewer binding (customer report: "namespace isolation on so it can't access the cluster based roles by design"). The real remedy is a unique release name (#3) or app.internalJWTMap: [] (#4). helm-charts:operator/values.yaml#L248-L250; operator/deployment.yaml#L37-L38

Every source is a clickable link pinned to a commit SHA — display text is the short repo:path#Lline / PR ref; the target is the GitHub blob or commit.

Open items — recommended next steps

Scope notes

  • The exact import-error string (row 2) is confirmed from Helm's own ownership-validation code (validate.go), which generates it for any chart including operator-wandb — not from a W&B-specific ticket. Internal tickets more often capture the namespace-isolation RBAC forbidden error, which is the separate failure mode described in row 5.
  • Chart evidence is pinned at helm-charts@de864305 and watchtower@030dbf84; the oidc-reviewer template name and the internalJWTMap guard are current on the default branch.

🤖 Generated by the CoreWeave Docs Team's Validator (beta-validator v1.0.0). Sources are linked inline; this is an accuracy aid, not a substitute for SME review.

@johndmulhausen johndmulhausen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated documentation fact-check — see the summary comment for the full validation table.


- Use a unique release/CR name for each install (for example `wandb-team-a`, not `wandb`).
- Set `app.internalJWTMap: []` unless you need internal JWT mapping. This skips rendering the `oidc-reviewer` ClusterRoleBinding.
- Install the operator with `namespaceIsolation.enabled=true`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refuted — the source says this is wrong. namespaceIsolation.enabled=true does not avoid the oidc-reviewer collision; it's the cause of cluster-scoped RBAC failures. That key only gates the operator's --isolation-namespaces arg and the operator's namespaced Role/RoleBinding (in the separate operator chart) — it never touches the cluster-scoped oidc-reviewer ClusterRoleBinding. With isolation on, the operator's service account is scoped to one namespace and can no longer manage that cluster-scoped object (customer report: "namespace isolation on so it can't access the cluster based roles by design").

Removing this bullet is the safe fix — the two working remedies (unique release name, app.internalJWTMap: []) are already listed above.

Suggested change
- Install the operator with `namespaceIsolation.enabled=true`.

Source says: namespace isolation gates only the operator's namespaced RBAC, unrelated to the cluster-scoped oidc-reviewer binding · Source: helm-charts:operator/deployment.yaml#L37-L38, operator/values.yaml#L248-L250

🤖 CoreWeave Docs Team's Validator — refuted

Review feedback: namespaceIsolation doesn't avoid the oidc-reviewer collision —
with it on, the operator's namespaced SA can't manage the cluster-scoped binding
at all. The working remedies (unique release name, app.internalJWTMap: []) stay.
@pnalubandhu-a11y

Copy link
Copy Markdown
Author

Thanks @johndmulhausen / validator — good catch. Removed the namespaceIsolation.enabled=true bullet (fcf2723); you're right it doesn't avoid the collision and actually prevents the operator SA from managing the cluster-scoped binding. Kept the two working remedies (unique release name + app.internalJWTMap: []).

@mdlinville mdlinville enabled auto-merge (squash) July 8, 2026 22:35

@mdlinville mdlinville left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing my approval until the technical assertion flagged by John is resolved one way or the other. Thanks. :)

@pnalubandhu-a11y

Copy link
Copy Markdown
Author

@mdlinville thanks — both of John's flagged items are now resolved:

#5 (the blocking one — namespaceIsolation.enabled=true): removed in fcf2723. The validator was right that it's inverted — namespace isolation is the cause of the cluster-scoped RBAC failure, not a fix — so that bullet is gone. Only the two working remedies remain (unique release name; app.internalJWTMap: []).

#4 (advisory — app.internalJWTMap: []): the validator actually confirmed this one against the chart source (the CRB is guarded by {{- if and .Values.app.internalJWTMap (gt (len .Values.app.internalJWTMap) 0) }}, so an empty list skips it). Its only qualifier was scope — don't blank it on installs that genuinely use internal JWT mapping. I've made that explicit in 0dd9224: the bullet now reads "…unless you need internal JWT mapping, such as native-Kubernetes-OIDC on-premises installs that rely on the oidc-reviewer binding."

Net: the ❌ refuted assertion is gone and the remaining item is advisory + now caveated in-text. Good to re-review when you have a moment — thanks! 🙏

@pnalubandhu-a11y

Copy link
Copy Markdown
Author

@mdlinville when you get a moment — both items you and John flagged are now resolved:

  • namespaceIsolation.enabled=true bullet removed (fcf2723)
  • advisory on app.internalJWTMap: [] scoped in-text (0dd9224) — now caveated for native-Kubernetes-OIDC on-prem installs

CI is green. Would you mind re-reviewing to clear the changes-requested? Thanks! 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants