Release v1.30.0

Release v1.30.0

This release adds support for the XID 154 recovery actions of R610 and later drivers, lets the labeler's default GPU class follow a GPU Operator mode switch without manual configuration, adds an option to run metadata-collector without its pod device mapper, and labels platform_connector_health_events_received_total by processing strategy and health. It also fixes the bundled MongoDB network policy for a release that is not named after its namespace, and stops a truncated node condition from staying True after every GPU recovers.

Note

DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.

Upgrade Notes

Bundled MongoDB network policy follows the release name (#2186, #2185)

mongodb-networkpolicy selected the bundled Bitnami MongoDB pods by the release namespace, but the pods carry the release name. With the default networkPolicy.enabled: true, the metrics-access policy still isolated the MongoDB pods, so it blocked port 27017, the replica set could not form, and platform-connectors could not reach MongoDB. The policy now selects the pods by release name.

If your release name is different from its namespace and you set networkPolicy.enabled: false, no policy selected the MongoDB pods before. Now mongodb-networkpolicy selects them and allows only port 27017 from the release namespace, so a Prometheus in another namespace can no longer scrape the MongoDB exporter on port 9216. To allow it, add a NetworkPolicy for port 9216, for example through mongodb-store.mongodb.extraDeploy.

Labeler GPU class with --reuse-values (#2108, #1892)

The new default GPU class (see Major New Features) is a change to labeler.expectedDeviceCounts.classes. helm upgrade --reuse-values keeps your previous class list, so GPUCluster mode still needs manual configuration. If you run GPUCluster mode, or switch between modes, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults. If you set your own classes, for example to enable the gpu-dra example class, replace your GPU classes with the new default GPU class, as the labeler configuration docs show. Without sources, a GPU DRA class can report 0 on a node that has only the ResourceSlices of another driver.

Major New Features

R610 and Later Driver Support for XID 154 (#2166, #2174, #2165)

XID 154 reports the recovery action that the GPU driver asks for. R610 and later drivers print PF FLR and OS Reboot where earlier drivers printed GPU Reset Required and Node Reboot Required, and they add Bus Reset and System Reboot. syslog-health-monitor now maps PF FLR to COMPONENT_RESET, and OS Reboot, Bus Reset and System Reboot to RESTART_BM. A bus reset covers more than the single-GPU reset that COMPONENT_RESET runs, so it gets a node reboot. With fault-remediation enabled, NVSentinel resets the GPU or reboots the node for these recovery actions on R610 and later drivers.

Labeler GPU Count Follows the GPU Operator Mode (#2108, #1892)

The labeler could already count GPUs in GPU Operator GPUCluster (DRA) mode, but only after you enabled the gpu-dra example class, and a switch between ClusterPolicy and GPUCluster mode needed a configuration change. The default GPU class now counts GPUs in both modes, so a mode switch needs no manual configuration. The class lists two sources, and the labeler uses the first source that applies to each node:

• dra counts the GPUs in the node's gpu.nvidia.com ResourceSlices, and groups nodes by instance type and by the GPU models in the slices. It applies when the node has slices from that driver. If a pool is incomplete or the expression fails, the labeler skips the class and does not use the GFD label.
• gfd reads the GFD nvidia.com/gpu.count label, but only while the NVIDIA device plugin advertises GPUs. Thus a label left from ClusterPolicy mode does not count on a node whose DRA plugin is down.

When no source applies, the labeler keeps the existing labels. A node that switches modes keeps its expected count. Any device-count class can now list ordered sources. Because the default class reads ResourceSlices, the labeler ClusterRole now gets get, list and watch on resourceslices. See the Upgrade Notes for --reuse-values.

Run metadata-collector Without the Pod Device Mapper (#2082, #2069)

On a vcluster, the kubelet /pods endpoint and the PodResources socket describe host pods. Their names do not match the pods that metadata-collector sees through the virtual cluster's API server, so the pod device mapper always failed and the container exited. The new metadata-collector.podMapper.enabled value, default true, turns the mapper off. With the mapper off, metadata-collector still writes the GPU metadata file, serves metrics, and keeps running. Partial drains need the dgxc.nvidia.com/devices annotation that the mapper writes, so keep node-drainer.partialDrainEnabled: false.

Metrics

• platform_connector_health_events_received_total gains processing_strategy and healthy labels (#2095, #2089): the counter had no labels, so actionable events could not be told apart from STORE_ONLY and recovery events. platform-connectors counts each event after it applies its defaults, so processing_strategy is the stored strategy, and UNSPECIFIED counts as EXECUTE_REMEDIATION. healthy is "true" or "false". The counter has at most eight series. It no longer counts rejected batches; platform_connector_request_duration_seconds{outcome="rejected"} shows them. A query that reads the counter without sum() now returns one series for each label pair.
• Labeler device-count metrics (#2108): the new labeler_device_count_source{node, class, source} gauge shows the source that the labeler used for each node. With the new default GPU class, the partition label of labeler_device_count_expected starts with the source, for example source=dra|node.kubernetes.io/instance-type=p5|hardware=NVIDIA H100 80GB HBM3. Update dashboards and alerts that match the earlier partition values of the gpu class.

Bug Fixes & Reliability

• Bundled MongoDB works when the release name is different from the namespace (#2186, #2185): see the Upgrade Notes.
• maxRemediationAttempts holds through post-remediation validation (#2149, #2148): fault-quarantine ends the quarantine session when it hands a node to post-remediation validation. fault-remediation cleared the attempt count at that point, so a fault found while validation still held the node started a new budget, and maxRemediationAttempts did not stop a node that kept failing validation. fault-remediation now keeps the count until validation releases the node, and the next fault after that starts from zero. With the cap set, the latestFaultRemediationState node annotation now stays on the node after the hand-off to validation.
• Fault-remediation remediates nodes with names longer than 63 characters (#2151, #2150): with the log collector enabled, fault-remediation put the node name in the dgxc.nvidia.com/node-name label of the log collector Job. Kubernetes limits a label value to 63 characters, so for a longer node name the API server rejected the Job lookup and creation, and fault-remediation never remediated the node. fault-remediation now cuts a longer name and adds the first 8 hex characters of the SHA-256 hash of the full name.
• Node conditions clear after every GPU recovers (#2168, #2167): when the entries of a node condition did not fit in platformConnector.k8sConnector.maxNodeConditionMessageLength after compaction, platform-connectors cut the last entry at a byte boundary. A fragment cut before its GPU and PCI tokens, such as Error;, matched no recovery event, so the condition stayed True after every GPU recovered. platform-connectors now first shortens each entry to its error code, entity tokens and recommended action, and cuts at a byte boundary only when the entries still do not fit. This fix does not repair a condition that an earlier version left in this state; clear it with one patch of the node status.
• Fault-quarantine refuses a non-positive circuit breaker window (#2201, #2200): a fault-quarantine.circuitBreaker.duration of 0s made fault-quarantine panic at the first cordon, and a negative value made it panic at startup. The pod then crash-looped, because the cold start replays the cordon. fault-quarantine now refuses such a value at startup with a clear error.
• nic-health-monitor can report ports that stay down from boot (#2068, #2067): the first poll suppresses an unhealthy port when its peers give no verdict, for example on a card that is alone in its role group. Such a port was never reported, so a node that booted with half its compute links down stayed silent. The new opt-in nic-health-monitor.suppressedPortReportAfter value reports a suppressed port on a compute card when it stays unhealthy for that long. An InfiniBand port that is still DOWN is fatal with REPLACE_VM. INIT and ARMED ports are non-fatal, so a subnet manager outage at boot does not make nodes fatal. A RoCE port is reported only when it is DOWN. The wait continues across pod restarts. A port that an earlier version suppressed is picked up only after the next reboot. The option is off by default (0s).
• Fault-quarantine and node-drainer log fixes (#2183, #2182, #2197, #2196, #2180, #2179): fault-quarantine wrote three INFO lines, one with the full event, for each healthy event that it skipped. After a monitor restart, these lines rotated the cordon and circuit breaker lines out of kubectl logs within minutes. These lines are now DEBUG and do not include the event. The fault-quarantine ruleset lines showed every event with CreatedAt 0001-01-01T00:00:00Z; they now log the event itself. node-drainer logged each completed pod that it skipped, on each scan, with a printf format string through slog; it now logs one structured DEBUG line.

Documentation

• Node condition timestamps (#2199, #2198): the platform-connectors documentation now explains that the timestamps of a node condition come from the health event that last changed it. A check that stays healthy writes the condition once, so the timestamps are not a liveness signal. The documentation lists the metrics that show whether a check still runs.

Chores

• E2E tests are divided across jobs (#2156, #2137): each E2E architecture runs five jobs, and each job ran every test. The tests are now divided across the jobs, so a job takes 38 to 44 minutes instead of 81 to 86.
• Tilt CI passes on its first attempt (#2143, #2136): in make tilt-ci, the datastore started about 5.5 minutes into the first tilt ci attempt, so the attempt failed in almost every E2E job. The datastore now starts about 3 minutes earlier, and the first attempt passed in 9 of 10 jobs instead of 1 of 20.
• Tilt CI retries recover (#2173, #2172, #2195, #2194): after a failed first attempt, the retries failed too. Each attempt now removes the Helm releases that a stopped attempt left pending, and deletes the stale kai-scheduler webhook configurations before it installs kai-scheduler again. health-events-analyzer now starts after the datastore and platform-connectors. The kai-scheduler admission webhook gets only the pods of the preflight test namespaces, so it cannot reject NVSentinel pods while kai-scheduler installs.
• kind cluster creation retries (#2154): make cluster-create now runs ctlptl apply up to three times. With kind 0.33.0, about 2% of E2E jobs failed when the second control plane joined the cluster.
• Five flaky unit tests fixed (#2181, #2139): in labeler, fault-remediation, janitor and slinky-drainer. Three tests reconciled before the informer cache had their write, one had a data race on a shared scheme, and one had a timing assertion that included the labeler's start-up time.
• Dead-code check (#2113, #2134): removes about 7,000 lines of Go code that no binary uses, including the old PostgreSQL client and change-stream watcher in store-client. make dead-code-lint and a new dead-code CI job fail when new code is unreachable or unused.
• Docs pipeline (#2189, #2188, #2193, #2241, #2141): Renovate now updates the Fern CLI version in fern/fern.config.json. The fern check job now creates the content of the older docs versions before it runs, as the build and publish jobs do. The Fern preview comment workflow now skips a build that uploaded no preview, instead of failing on main for each PR push without docs changes.

Acknowledgments

This release includes contributions from:

• @bfs2021
• @blacksoxx
• @gengwg
• @jianzhangbjz
• @jtschelling
• @kdotmanoj
• @lalitadithya
• @lblanc-together
• @ranrubin
• @tanishagoyal2
• @XRFXLP
• @ZeXi0308

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @bfs2021, @gengwg, @lblanc-together, and @ZeXi0308.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.30.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.29.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.30.0 \
  --namespace nvsentinel \
  --reuse-values