Skip to content

fix(nvca-operator): restore ngc-managed default and fix nil pointer on 3.0.x upgrade - #706

Open
apartha-nv wants to merge 3 commits into
mainfrom
fix/nvca-operator-cluster-source-defaults
Open

fix(nvca-operator): restore ngc-managed default and fix nil pointer on 3.0.x upgrade#706
apartha-nv wants to merge 3 commits into
mainfrom
fix/nvca-operator-cluster-source-defaults

Conversation

@apartha-nv

@apartha-nv apartha-nv commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Why

Two bugs in the nvca-operator 3.1.0 chart prevented successful NGC-managed installation and upgrade from 3.0.x:

Bug 1 - Wrong clusterSource default (blocking NGC-managed installs)

The 3.1.0 vendoring script changed ngcConfig.clusterSource from ngc-managed to self-managed. NGC UI-generated install commands do not include --set ngcConfig.clusterSource=..., so any cluster installed using the NGC UI with the 3.1.0 chart ends up in self-managed mode and the NVCA agent never registers.

Bug 2 - Nil pointer crash on upgrade from 3.0.x (all cluster types)

self-managed-nvcfbackend-cm.yaml evaluates .Values.selfManaged.imageCredHelper.imageRepository and .Values.selfManaged.sharedStorage.imageRepository in the ConfigMap metadata block. This block runs unconditionally for every cluster type, before the clusterSource guard on line 37. Both imageCredHelper and sharedStorage were added as sub-keys in 3.1.0. When upgrading from 3.0.x with --reuse-values, these keys are absent from stored values, making their parent maps nil. Accessing .imageRepository on a nil interface panics:

nil pointer evaluating interface {}.imageRepository

What changed

  • deploy/helm/nvca-operator/scripts/ci_vendor_nvca_operator_chart: restore ngcConfig.clusterSource = "ngc-managed" (reverts accidental change to "self-managed")
  • deploy/helm/nvca-operator/nvca-operator/values.yaml: restore clusterSource: ngc-managed and update comment to list all three valid values
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml: same source chart update
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl: add explicit ngcConfig.clusterSource: self-managed so self-hosted stack installs continue to work with the restored default
  • deploy/helm/nvca-operator/nvca-operator/templates/self-managed-nvcfbackend-cm.yaml: use | default dict before accessing .imageRepository on selfManaged.imageCredHelper and selfManaged.sharedStorage to guard against nil parents on upgrade

Customer Release Notes

Fixes helm upgrade from nvca-operator 3.0.x to 3.1.0 crashing with "nil pointer evaluating interface {}.imageRepository" on all cluster types. Fixes NGC-managed clusters failing to register the NVCA agent after installing nvca-operator 3.1.0 via NGC UI.

Plan Summary

Not applicable.

Usage

No operator change required. After this fix, helm upgrade from 3.0.x to 3.1.0 and NGC UI install commands work without additional flags.

Testing

  • helm lint passes with nil selfManaged.imageCredHelper and selfManaged.sharedStorage
  • helm template with --set selfManaged.imageCredHelper=null --set selfManaged.sharedStorage=null renders without error
  • Verified clusterSource: ngc-managed is the new default in both the vendored and source chart

QA should validate:

  • Helm upgrade from nvca-operator 3.0.x to 3.1.0 on an NGC-managed cluster
  • Fresh NGC UI install on an NGC-managed cluster (no explicit clusterSource flag)
  • Self-managed stack helmfile install (make install from deploy/stacks/nvcf-compute-plane)

Notes

The self-managed stack helmfile now explicitly sets ngcConfig.clusterSource: self-managed, making the install independent of the chart default.

References

Closes #704

Related Pull Requests

None.

Dependencies

None.

Summary by CodeRabbit

  • New Features

    • Added clearly defined cluster source modes for NGC-managed, Helm-managed, and self-managed deployments.
    • Self-managed compute plane deployments now explicitly select the self-managed mode.
    • The default cluster source is now NGC-managed.
  • Bug Fixes

    • Improved handling when image credential helper or shared storage settings are omitted.
  • Documentation

    • Updated cluster source descriptions, ordering, defaults, and terminology for greater clarity.

@apartha-nv
apartha-nv requested a review from a team as a code owner August 6, 2026 07:53
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d40b0820-1d12-41ad-832c-d78ecc3a4e67

📥 Commits

Reviewing files that changed from the base of the PR and between 5d78e34 and 4879050.

📒 Files selected for processing (6)
  • deploy/helm/nvca-operator/nvca-operator/templates/self-managed-nvcfbackend-cm.yaml
  • deploy/helm/nvca-operator/nvca-operator/values.yaml
  • deploy/helm/nvca-operator/scripts/ci_vendor_nvca_operator_chart
  • deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml
  • deploy/helm/nvca-operator/scripts/ci_vendor_nvca_operator_chart
  • deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh
  • deploy/helm/nvca-operator/nvca-operator/values.yaml

📝 Walkthrough

Walkthrough

The NVCA Operator chart now defaults to ngc-managed. The self-managed stack sets self-managed explicitly. Optional self-managed configuration sections are guarded before image repository lookups. A Helm rendering test validates both modes.

Changes

NVCA Operator configuration

Layer / File(s) Summary
Cluster source defaults and stack override
deploy/helm/nvca-operator/nvca-operator/values.yaml, deploy/helm/nvca-operator/scripts/ci_vendor_nvca_operator_chart, src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml, deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
The chart and vendoring script use ngc-managed by default. The self-managed stack sets self-managed explicitly. Documentation lists all supported modes.
Optional self-managed value guards
deploy/helm/nvca-operator/nvca-operator/templates/self-managed-nvcfbackend-cm.yaml
Missing selfManaged.imageCredHelper and selfManaged.sharedStorage sections default to empty dictionaries before imageRepository is accessed.
Cluster source rendering validation
deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh
The test renders default and self-managed configurations. It checks NVCA_CLUSTER_SOURCE and managed ConfigMap contents.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title follows Conventional Commits and accurately describes the two bug fixes in the changeset.
Linked Issues check ✅ Passed The changes restore the ngc-managed default, preserve self-managed installs, guard nil maps, and add coverage for the required behavior.
Out of Scope Changes check ✅ Passed All changes support the linked issue, including configuration documentation, deployment overrides, nil guards, and regression tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/nvca-operator-cluster-source-defaults

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
deploy/helm/nvca-operator/nvca-operator/values.yaml (1)

241-251: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Add render assertions for ngc-managed and self-managed.

Both chart defaults are ngc-managed, but the current test checks version overrides only. Assert NVCA_CLUSTER_SOURCE and the selected configuration resources for both modes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/helm/nvca-operator/nvca-operator/values.yaml` around lines 241 - 251,
Add Helm render assertions covering both ngc-managed and self-managed
clusterSource values, including the rendered NVCA_CLUSTER_SOURCE and the
configuration resources selected for each mode. Extend the existing
version-override test coverage without changing the chart defaults or unrelated
behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@deploy/helm/nvca-operator/nvca-operator/values.yaml`:
- Around line 241-251: Add Helm render assertions covering both ngc-managed and
self-managed clusterSource values, including the rendered NVCA_CLUSTER_SOURCE
and the configuration resources selected for each mode. Extend the existing
version-override test coverage without changing the chart defaults or unrelated
behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ea708f2f-4cdd-4236-9221-b49946d43a1e

📥 Commits

Reviewing files that changed from the base of the PR and between 6bb0cde and e4d033b.

📒 Files selected for processing (5)
  • deploy/helm/nvca-operator/nvca-operator/templates/self-managed-nvcfbackend-cm.yaml
  • deploy/helm/nvca-operator/nvca-operator/values.yaml
  • deploy/helm/nvca-operator/scripts/ci_vendor_nvca_operator_chart
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml

Comment thread deploy/helm/nvca-operator/nvca-operator/values.yaml Outdated
Comment thread src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh`:
- Around line 36-44: Extend the self-managed rendering test around the existing
manifest_self_managed flow to use a legacy values fixture where
selfManaged.imageCredHelper and selfManaged.sharedStorage are omitted or null.
Assert helm template succeeds for that fixture, then verify
nvcfbackend-self-managed contains the supplied ICMS, revalidation, and NATS
service URLs rather than relying only on a nonzero data-key count.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ee3298f7-5da0-4120-aa4c-ee4565b571ed

📥 Commits

Reviewing files that changed from the base of the PR and between e4d033b and 8e27edd.

📒 Files selected for processing (1)
  • deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh

Comment thread deploy/helm/nvca-operator/tests/cluster_source_defaults_test.sh
@apartha-nv
apartha-nv enabled auto-merge August 7, 2026 08:37
…n 3.0.x upgrade

Two bugs in the nvca-operator 3.1.0 chart prevented successful installation
and upgrade:

1. The vendoring script changed ngcConfig.clusterSource from "ngc-managed" to
   "self-managed". NGC UI-generated install commands do not pass this flag
   explicitly, so NGC-managed clusters installed with 3.1.0 never registered
   their NVCA agent. Restore "ngc-managed" as the chart default and add an
   explicit "self-managed" override in the self-managed stack helmfile so that
   self-hosted deployments continue to work.

2. self-managed-nvcfbackend-cm.yaml evaluates selfManaged.imageCredHelper and
   selfManaged.sharedStorage in the ConfigMap metadata block, which runs for
   all cluster types before the clusterSource guard. These sub-keys were
   introduced in 3.1.0, so upgrading from 3.0.x with --reuse-values leaves
   their parent maps nil. Accessing .imageRepository on a nil interface
   panics with "nil pointer evaluating interface {}.imageRepository". Use
   | default dict on the parent before accessing the field.

Closes #704
…d self-managed

Add a chart render test that verifies:
- default render produces NVCA_CLUSTER_SOURCE=ngc-managed with empty
  nvcfbackend config maps
- self-managed render produces NVCA_CLUSTER_SOURCE=self-managed with
  populated nvcfbackend-self-managed and empty nvcfbackend-helm-managed

Closes #704
Use compute plane instead of control plane in the self-managed mode
description, matching the correct NVCF terminology.
@apartha-nv
apartha-nv force-pushed the fix/nvca-operator-cluster-source-defaults branch from b999026 to 4879050 Compare August 7, 2026 08:38
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: nvca-operator chart clusterSource defaults to self-managed instead of ngc-managed in 3.1.0

3 participants