Skip to content

chore(recipes): update nodewright to v0.17.1 (fixes immutable selector)#1502

Merged
lockwobr merged 1 commit into
mainfrom
chore/nodewright-0.17.1
Jun 26, 2026
Merged

chore(recipes): update nodewright to v0.17.1 (fixes immutable selector)#1502
lockwobr merged 1 commit into
mainfrom
chore/nodewright-0.17.1

Conversation

@lockwobr

Copy link
Copy Markdown
Contributor

Summary

Bump the nodewright-operator Helm chart pin from v0.17.0 to v0.17.1 across the registry, base overlay, and example recipes, and regenerate the container-images BOM.

Motivation / Context

deploy.sh could not upgrade nodewright-operator over a pre-rename install: the chart's controller-manager Deployment has an immutable spec.selector that changed across the skyhook to nodewright rename, so helm upgrade failed with field is immutable and the retry loop could not recover (manual kubectl delete was required).

The v0.17.1 chart fixes this upstream by shipping a selector-migration pre-upgrade hook (ServiceAccount + Role + Job, hook-weight: -10) that deletes the stale Deployment before the upgrade so helm upgrade recreates it cleanly. AICR's generated install.sh runs helm upgrade --install with no --no-hooks, so the hook executes as part of the normal deploy.

Fixes: #1382
Related: #1361

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Component(s) Affected

  • Recipe engine / data (pkg/recipe)
  • Docs/examples (docs/, examples/)

Implementation Notes

  • v0.17.1 is a chart-only patch: its appVersion is still v0.17.0, so the rendered operator image legitimately remains ghcr.io/nvidia/nodewright/operator:v0.17.0. The regenerated BOM reflects this.
  • The BOM regen also folds in pre-existing OCP manifest-component drift (empty *-ocp entries from Add OpenShift Container Platform (OCP) as a service type #1380 that were never regenerated). These are zero-image manifest entries and harmless.

Testing

make qualify

make qualify passed: 23 chainsaw tests passed / 0 failed, golangci-lint + yamllint clean, Grype found no vulnerabilities, license headers OK. No Go source changed, so the coverage gate does not apply.

Risk Assessment

  • Low — Isolated chart-version pin bump, well-tested, easy to revert

Rollout notes: Patch chart upgrade; the new pre-upgrade hook recovers stuck pre-rename installs automatically. Backwards compatible.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

The v0.17.1 nodewright chart adds a selector-migration pre-upgrade hook
(ServiceAccount + Role + Job) that deletes the stale, immutable-selector
Deployment before upgrade, so helm upgrade recreates it cleanly across
the skyhook->nodewright rename instead of failing on spec.selector.

Bumps the pin in registry.yaml, base overlay, and example recipes, and
regenerates the container-images BOM.

Fixes #1382

Signed-off-by: Brian Lockwood <lockwobr@gmail.com>
@lockwobr lockwobr requested review from a team as code owners June 26, 2026 22:39
@lockwobr lockwobr added dependencies Pull requests that update a dependency file theme/deployer Helm, ArgoCD, and deployment bundle generation labels Jun 26, 2026
@lockwobr lockwobr self-assigned this Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

Recipe evidence check

Broad impact: recipes/overlays/base.yaml changed; every leaf recipe is
potentially affected. Recipes that carry committed evidence are verified below;
the rest have no evidence yet (best-effort).

Protected recipes

Recipes with committed evidence (recipes/evidence/<slug>/<source>/<digest>.yaml) that this PR affects: 2

Recipe Source Pointer Verify Digest match
gb200-eks-ubuntu-training 7c4c0edc8c765a95a0f3afdb3bbb8e91 sha256-8274b6a1da24aa9782dc12162bf6a38265c30a852585ca64cfad5718efbbdec3 ❌ invalid — registry-forbidden (HTTP 401): registry not accessible (make the fork's aicr-evidence package public, or provide registry credentials) ⚠️ skipped (no signed digest)
h100-gke-cos-training 7c4c0edc8c765a95a0f3afdb3bbb8e91 sha256-33d4cf36622ead990c43a596f6f53c62b87d9fa4708f59b7e3f356f215e54317 ❌ invalid — registry-forbidden (HTTP 401): registry not accessible (make the fork's aicr-evidence package public, or provide registry credentials) ⚠️ skipped (no signed digest)
Other affected recipes without evidence yet: 61

These recipes are affected by this PR but carry no committed evidence pointer, so there is
nothing to verify. This is expected — evidence is hardware-gated and added over time.

  • a100-aks-training
  • a100-aks-ubuntu-training-kubeflow
  • a100-aks-ubuntu-training
  • a100-eks-training
  • a100-eks-ubuntu-training-kubeflow
  • a100-eks-ubuntu-training
  • a100-gke-cos-training-kubeflow
  • a100-gke-cos-training
  • a100-oke-training
  • a100-oke-ubuntu-training-kubeflow
  • a100-oke-ubuntu-training
  • b200-gke-cos-inference-dynamo
  • b200-gke-cos-inference
  • b200-gke-cos-training-kubeflow
  • b200-gke-cos-training
  • gb200-eks-inference
  • gb200-eks-training
  • gb200-eks-ubuntu-inference-dynamo
  • gb200-eks-ubuntu-inference
  • gb200-eks-ubuntu-training-kubeflow
  • gb200-oke-inference
  • gb200-oke-training
  • gb200-oke-ubuntu-inference-dynamo
  • gb200-oke-ubuntu-inference
  • gb200-oke-ubuntu-training-kubeflow
  • gb200-oke-ubuntu-training
  • h100-aks-inference
  • h100-aks-training
  • h100-aks-ubuntu-inference-dynamo
  • h100-aks-ubuntu-inference
  • h100-aks-ubuntu-training-kubeflow
  • h100-aks-ubuntu-training
  • h100-bcm-training
  • h100-bcm-ubuntu-training
  • h100-eks-inference
  • h100-eks-training
  • h100-eks-ubuntu-inference-dynamo
  • h100-eks-ubuntu-inference-nim
  • h100-eks-ubuntu-inference
  • h100-eks-ubuntu-training-kubeflow
  • h100-eks-ubuntu-training-slurm
  • h100-eks-ubuntu-training
  • h100-gke-cos-inference-dynamo
  • h100-gke-cos-inference
  • h100-gke-cos-training-kubeflow
  • h100-gke-cos-training-slurm
  • h100-kind-inference-dynamo
  • h100-kind-inference
  • h100-kind-training-kubeflow
  • h100-kind-training-slurm
  • h100-kind-training
  • h200-eks-inference
  • h200-eks-training
  • rtx-pro-6000-eks-inference
  • rtx-pro-6000-eks-ubuntu-inference-dynamo
  • rtx-pro-6000-eks-ubuntu-inference-nim
  • rtx-pro-6000-eks-ubuntu-inference
  • rtx-pro-6000-lke-inference
  • rtx-pro-6000-lke-training
  • rtx-pro-6000-lke-ubuntu-inference
  • rtx-pro-6000-lke-ubuntu-training

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
# Copy to the per-source path printed in the emit 'copyTo' hint:
#   recipes/evidence/<slug>/<source>/<bundle-digest>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 77.9%
Threshold 75%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-77.9%25-green)

No Go source files changed in this PR.

@lockwobr lockwobr merged commit ec48bbd into main Jun 26, 2026
223 of 224 checks passed
@lockwobr lockwobr deleted the chore/nodewright-0.17.1 branch June 26, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs area/recipes dependencies Pull requests that update a dependency file size/S theme/deployer Helm, ArgoCD, and deployment bundle generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: nodewright-operator Helm upgrade fails on immutable spec.selector (skyhook rename)

2 participants