Release v1.29.0

Release v1.29.0

This release lets node-drainer select a drain mode by the kind of a pod's controller and adds SELinux support for RKE2 and SUSE Linux Enterprise. It also stops the generic reboot from rebooting a node twice, stops fault-remediation from creating a second maintenance CR after a read error, and fixes node drains that never finished on a bundled MongoDB with fewer than three members. Several behaviours change on upgrade — please read the Upgrade Notes below.

Note

DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.

Upgrade Notes

Generic reboot image needs sh and cat (#2110, #2077)

With janitor-provider.csp.provider: generic, every reboot Job now runs its command through sh -c, because the command first checks the node boot ID. The default command was chroot /host reboot, which ran without a shell. The image in janitor-provider.csp.generic.rebootImage must now include sh and cat. The default busybox image has both. If your custom image has no sh, the reboot container does not start and the node does not reboot.

Bundled MongoDB disruption budget (#2112, #2111)

The bundled Bitnami MongoDB set mongodb-store.mongodb.pdb.minAvailable: 2. With fewer than three members, the PodDisruptionBudget allowed no evictions, so a drain of a node that hosted a member never finished. The chart now leaves pdb.minAvailable and pdb.maxUnavailable empty, and the subchart renders maxUnavailable: 1. With one or two members, a drain can now evict a member, and the replica set has no primary until the pod runs again. With more than three members, the budget allows one eviction at a time, where minAvailable: 2 allowed more.

helm upgrade --reuse-values keeps minAvailable: 2. If you run the bundled MongoDB with a member count other than three, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

RepeatedXIDErrorOnSameGPU skips XID 45 events (#2064, #2050)

The RepeatedXIDErrorOnSameGPU rule in health-events-analyzer never counted XID 45, but each XID 45 event still started the rule's 24-hour MongoDB query. A job teardown can produce a burst of XID 45 events, and these queries loaded the MongoDB primary and delayed change-stream processing. XID 45 events no longer start the rule.

The fix is in the default value of health-events-analyzer.config. helm upgrade --reuse-values keeps your previous value, so XID 45 events still start the query. If you run health-events-analyzer, upgrade with --reset-then-reuse-values. If you set your own config, copy the two XID 45 exclusions from the RepeatedXIDErrorOnSameGPU rule in the v1.29.0 chart.

Major New Features

Drain Modes by Pod Owner Kind (#2049, #1689)

A node-drainer.podDrainPolicies entry can now select pods by the kind of their controller, with the new ownerKinds list. A policy needs podSelector, ownerKinds, or both. With both, a pod must match both. node-drainer compares ownerKinds exactly with the kind in the pod's controller owner reference, and does not follow owners further, so Deployment pods match ReplicaSet and CronJob pods match Job. Pods without a controller do not match an ownerKinds policy. podDrainPolicies and userNamespaces are mutually exclusive, and userNamespaces is not empty by default, so set it to []:

node-drainer:
  userNamespaces: []
  podDrainPolicies:
    - name: stateful-workloads
      ownerKinds: ["StatefulSet"]
      mode: AllowCompletion

Bug Fixes & Reliability

• Generic reboot reboots a node once (#2110, #2077): with the generic CSP, the reboot stopped the reboot pod before the kubelet recorded a final status. After the node booted, the kubelet started the same pod again, and the pod rebooted the node a second time. janitor-provider now passes the node boot ID to the reboot container in PRE_REBOOT_BOOT_ID. The container reboots the node only when the node still has that boot ID. When the boot ID is different, the container exits without a reboot. When it cannot read either boot ID, it fails and does not reboot. See the Upgrade Notes for the image requirement.
• Fault-remediation does not create a second maintenance CR after a read error (#2093, #2092): fault-remediation treated any error from reading the maintenance CR as NotFound. It then removed the CR from the node annotation and created a new one, so a transient read error while the first CR ran gave the node a second CR. Only a NotFound error now counts as a missing CR. Fault-remediation counts other read errors in fault_remediation_processing_errors_total as cr_status_check_error and retries the event. In a partial recovery, an unreadable CR no longer sets the node to remediation-succeeded.
• Node-drainer dry run does not create custom drain CRs (#2076, #2075): in dry run, node-drainer sent evictions with DryRun: [All], but with custom drain enabled it created a real custom drain CR, and the external drain controller then drained the node. node-drainer now creates and deletes the custom drain CR with DryRun: [All]. The API server validates the CR but does not store it.
• Bundled MongoDB does not block drains below three members (#2112, #2111): see the Upgrade Notes.
• Kubernetes-object-monitor publishes healthy events after a restart (#2078, #1542): the monitor saves each unhealthy match in the nvsentinel.nvidia.com/k8s-object-monitor-policy-matches node annotation, so that it can publish the healthy event after a restart. It never loaded these matches, because it read them before its cache started. Also, a resource deleted while the monitor was stopped was not in its first list, so nothing reconciled it. In both cases, the node stayed cordoned until someone uncordoned it by hand. The monitor now loads the saved matches at startup and reconciles each one after its cache syncs. The first start after the upgrade also processes the matches that earlier versions saved. If the monitor cannot load the matches, it now exits instead of starting without them.
• Platform-connectors keeps the node condition of a fatal event (#2071, #2070): the connector wrote node conditions with UpdateStatus at the resourceVersion it read. For a fatal event, fault-quarantine, node-drainer and fault-remediation update the same Node within a few hundred milliseconds, and each update made the condition write fail with a conflict. When the reboot stopped the connector before its next retry, the node never showed the condition. The connector now writes the conditions with a JSON patch that tests only the current conditions, so label, annotation, taint and cordon changes do not conflict with it. The deployment platform connector's ClusterRole now also grants patch on nodes/status.
• Janitor waits for the reset Job before it restores GPU services (#2022, #1991): when a GPUReset was deleted after its services were torn down but before its reset Job finished, janitor re-enabled the managed GPU services while nvidia-smi --gpu-reset still ran. DCGM, and in GPUCluster mode the DRA kubelet plugin, could attach to GPUs during the reset. The deletion now waits for the Job to finish. If the Job runs past the GPUReset timeout, janitor restores the services and logs an error.
• A GPUReset without spec.selector starts (#2066, #2065): without spec.selector, a GPUReset resets every GPU on the node. The update webhook read the selector without a nil check, so it panicked on the first update, which adds the finalizer, and denied it. The reset never started. The webhook now treats a missing selector as an empty UUID list.
• platform-connectors and nic-health-monitor run on SELinux-enforcing nodes (#1959, #1953): on SELinux-enforcing nodes, for example RKE2 installed from RPM on SUSE Linux Enterprise Server 16 or RHEL, unprivileged pods run as container_t. platform-connectors could not create its socket in /var/run/nvsentinel, so no monitor could publish, and the nic-health-monitor init container could not read /var/run/nic_health_monitor. The new global.seLinuxOptions value sets the pod SELinux options of the platform-connectors DaemonSet, syslog-health-monitor (regular and Kata) and nic-health-monitor. On these nodes, set global.seLinuxOptions.type: spc_t.
• RepeatedXIDErrorOnSameGPU skips XID 45 events (#2064, #2050): see the Upgrade Notes.
• logLevel takes effect in six more modules (#2058, #2057, #2074, #2073): janitor, janitor-provider, event-exporter, lifecycle-manager, metadata-collector and preflight read LOG_LEVEL, but their charts did not set it, so <module>.logLevel had no effect. Each chart now has a logLevel value, default info, and passes it to the container as LOG_LEVEL.
• Event-exporter uses the HTTP proxy settings (#2099, #2094): the HTTP sink and the OIDC token client ignored HTTP_PROXY, HTTPS_PROXY and NO_PROXY. In clusters where egress must go through a proxy, event-exporter could not get a token or publish events. Both clients now use these variables. The new event-exporter.extraEnv value sets them, and the event-exporter configuration docs show an example.
• Event-exporter reuses token-endpoint connections (#2123, #2122): the OIDC token client created a new HTTP client and transport for each token request, so it never reused a connection, and each old connection stayed open until the server closed it. The client is now created once and reused.

Security & Infrastructure

• Go 1.27.3 (#2090): all Go components now build with Go 1.27.3, which includes upstream security fixes.
• Preflight NCCL images (#2091, #2083): nccl-allreduce now uses PyTorch NGC 26.09-py3 (PyTorch 2.14, CUDA 13.4.1, NCCL 2.31.2), and nccl-loopback uses nccl-tests v2.21.1. Every base image is now pinned by digest.

Documentation

• Deploying on SUSE Linux Enterprise and RKE2 (#1958): a new guide collects the settings for SUSE Linux Enterprise and RKE2: GPU Operator NRI mode, SELinux, host-installed drivers on SL Micro, DCGM embedded-mode, journal persistence, and the PostgreSQL and MongoDB images on 64 KB-page arm64.
• MongoDB open-file limit (#2062, #1555): mongod uses one file descriptor for each client connection and gets its open-file limit from the container runtime on the node. With a soft limit of 1024, which some node images set, mongod crashes as the cluster grows. The MongoDB store documentation now gives a soft limit of at least 64000 as a node prerequisite for both backends, with check commands and an alert on mongodb_ss_connections. The MongoDB connection-error runbook covers the crash.

Acknowledgments

This release includes contributions from:

• @abonillabeeche
• @akhil9tiet
• @avgvi
• @bhaskargurram-ai
• @blacksoxx
• @cbumb
• @deesharma24
• @jianzhangbjz
• @kdotmanoj
• @lalitadithya
• @msnow-nv
• @tanishagoyal2
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @akhil9tiet, @bhaskargurram-ai, @blacksoxx, and @kdotmanoj.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.29.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.28.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.29.0 \
  --namespace nvsentinel \
  --reuse-values