Repository navigation
Release v1.32.0
This release replaces nodes on AWS for REPLACE_VM events, makes node-drainer and fault-remediation record their decisions on the health event, and attaches a signed OpenVEX document to every published image. It also stops a new quarantine from removing an operator's own taint, keeps platform-connectors from storing a recovery before its fault, fixes a PostgreSQL change-stream stall under load, and lets gpu-health-monitor reconnect after an unclean DCGM disconnect.
Note
DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.
Major New Features
Node Replacement on AWS (#2342)
On AWS, every TerminateNode failed with SendTerminateSignal not implemented for AWS, so a REPLACE_VM health event cordoned and drained the node but never replaced it. janitor-provider now terminates the EC2 instance of the node with TerminateInstances, and janitor then waits for the node to go NotReady and deletes it. With fault-remediation enabled, a REPLACE_VM event on AWS now terminates the instance. The janitor-provider IAM role needs ec2:TerminateInstances.
Drain and Remediation Decisions on the Health Event (#2354, #2205, #2341, #2207)
This release adds recorded decisions (ADR-064) to node-drainer and fault-remediation:
- node-drainer (#2354, #2205): node-drainer chose the drain method, built-in eviction or a custom drain CR, again on each pass from the node labels. A label change during a drain could make a custom drain CR and the built-in eviction run on the same node. node-drainer now chooses the method and the scope, a full drain or the entity of a partial drain, one time for each health event, and stores them in the event's
drainDecisionstatus. Later passes and cold start follow the stored decision, so a change tocustomDrain.nodeSelectororpartialDrainEnabledduring a drain does not affect that drain. If the stored method can no longer run, node-drainer waits, logs an error and counts it innode_drainer_processing_errors_total. For example, if custom drain is turned off during aCustomdrain, end the quarantine and delete the CRdrain-<node name>-<event ID>to end the drain. - fault-remediation (#2341, #2207): fault-remediation now stores the action that it selects in the event's
remediationDecisionstatus, before it starts the log collector or creates the maintenance CR. If the write fails, fault-remediation creates no CR and retries the event.
Quieter syslog-health-monitor Logs (#2372, #2371)
syslog-health-monitor polls the journal every 15 seconds, and each poll logged about a dozen INFO lines for each check, also when no new entries arrived. These routine messages are now DEBUG. To see them, set syslog-health-monitor.logLevel: debug.
Metrics
node_drainer_awaiting_completion_event_timestamp_seconds{node}(#2334, #2329):AllowCompletion, the chart default drain mode, has no deadline, and node-drainer exported nothing while it waited, so a drain that could never finish looked like one that was waiting. The new gauge holds the creation time of the health event behind a drain that still waits for pods to complete, and is removed when the wait ends. To find a drain that waits too long, alert ontime() - node_drainer_awaiting_completion_event_timestamp_seconds > 7200. The time counts from the health event, not from the start of the drain.- New error labels (#2354, #2341, #2365):
node_drainer_processing_errors_totalgains theerror_typevaluesrecord_drain_decision_error,recorded_custom_drain_unavailable,recorded_builtin_no_pod_policiesandrecorded_drain_method_unknown.fault_remediation_processing_errors_totalgains the error typerecord_decision_error. In gpu-health-monitor,dcgm_api_failures_totalgains theerror_namevaluesstale_group_cleanupanddcgm_initialize_monitoring, anddcgm_api_latencygains theoperation_namevaluestale_group_cleanup.
Bug Fixes & Reliability
- A new quarantine keeps an operator's taint (#2287, #2286): on a new quarantine, fault-quarantine treated a node taint as already present only when its key, value and effect all matched a rule-set taint. A taint with the same key and effect but a different value, for example an operator's
nvidia.com/gpu-xid-error=owned-by-operator:NoSchedule, was overwritten, recorded as an NVSentinel taint, and then removed at unquarantine. Fault-quarantine now leaves such a taint on the node, as it already did for a node that was already quarantined. - platform-connectors stores events in order and drains its queues at shutdown (#2322, #2321): when an insert failed, the store connector took the next batch while the failed batch waited for its retry, so a recovery could be stored before the fault it followed. The connector now retries a failed batch before it takes the next one. It drops a document that the datastore can never store at once, with an error log. At shutdown, platform-connectors could stop before its queues were empty and lose batches that it had already acknowledged. It now waits up to 10 seconds for the queues to drain, and the platform-connectors DaemonSet gets a termination grace period of 45 seconds.
- The PostgreSQL change stream keeps up under load (#2275, #2171): with the PostgreSQL datastore, the LISTEN loop fetched each notified row itself and waited for the consumer. When fault-quarantine fell behind, the loop stopped reading, fault-quarantine processed only about 300 events a minute, and the PostgreSQL backend held an open transaction that blocked VACUUM and
CREATE INDEX CONCURRENTLY. The LISTEN loop now only records notifications, a separate loop fetches the rows, and the watcher polls until it is caught up, also at startup. A notification can no longer move the read position past older rows that nobody has read, so a restart behind a backlog no longer loses those rows. - gpu-health-monitor reconnects after an unclean DCGM disconnect (#2365, #2338): the DCGM hostengine keeps the groups of a connection until it sees the connection close, and a field-group name must be unique. After an unclean disconnect, for example a liveness restart during a DCGM outage longer than about 2 minutes, every new connection failed with
Duplicate key passed to function. The node kept a false fatalGpuDcgmConnectivityFailurecondition until someone restarted DCGM. gpu-health-monitor now deletes its own stale groups before it creates new ones. It deletes only the groups nameddcgm_health,nvsentinel_field_monitorsandgpu_temp_limit. - MongoDB setup Job deadline (#2274, #2257): the
create-mongodb-databaseJob had a fixed deadline of 600 seconds. When image pulls queued on a cold node, MongoDB was not reachable in time, the Job failed, Helm did not run it again, and every module crash-looped withUserNotFound. The deadline is nowmongodb-store.job.activeDeadlineSeconds, default1800. The setup Job for an external MongoDB readsglobal.datastore.setupJob.activeDeadlineSeconds, default600. A value of0removes the deadline. - The generic reboot pod on a node under disk pressure (#2397, #2184): the kubelet rejects a pod without a critical priority on a node that reports
DiskPressure, so the generic reboot pod never ran on such a node. The newjanitor-provider.csp.generic.rebootJobPriorityClassNamevalue sets the priority class of the reboot pod. Set it tosystem-node-criticalorsystem-cluster-critical. The class must exist, and the reboot Job namespace must be allowed to use it. The value is empty by default. - GPU faults clear after a GPU index change (#2377): after a reboot, a GPU can come back with a new index or PCI address but the same UUID. Its healthy event did not clear the old entry from the node condition, so the condition stayed
True. A healthy GPU event now also clears an entry that names the same full GPU UUID. An entry with a shortened UUID does not match. - No false fabric probe failures on GPUs without NVLink (#2366, #2360): DCGM raised
DCGM_FR_FABRIC_PROBE_STATEon GPUs without NVLink, for example L40 PCIe GPUs with driver 535. gpu-health-monitor now suppresses it when the GPU has no NVLink hardware and NVML reports 0 NVLinks present. metadata-collector records that count in the GPU metadata asnvlink_present_link_count. - One event for a GPU that falls off the bus after an XID (#2369, #2367): syslog-health-monitor skips a "fallen off the bus" line when the same GPU had an XID in the last 5 minutes. The two lines write the PCI address in different forms, so they never matched, and the node got a second
GPU_FALLEN_OFF_BUSevent for the same fault. The addresses are now compared in one form. With the XID check disabled, the monitor always sends the fallen-off-the-bus event. - nic-health-monitor publishes during shutdown (#2364): SIGTERM cancelled every publish, so the monitor lost the
NICPollStallCheckevent and the recoveries of a poll that completed during shutdown. These events come when liveness restarts the monitor. Publishes that start after SIGTERM now share one 10-second window, and the DaemonSet sets its termination grace period to 30 seconds. - node-drainer starts with custom drain enabled (#2370, #2368): with
customDrain.enabled: true, node-drainer failed at startup withinvalid duration format: 30m, because the chart default ofcustomDrain.timeoutwent through a parser that accepts only seconds. node-drainer never used this value, so it is removed. The chart and node-drainer ignore it if you still set it. - lifecycle-manager records its leader election event (#2399, #2398): the chart granted
eventsonly withcontrollers.maintenanceRequest.enabled, which is off by default, so the API server rejected theLeaderElectionevent. The lifecycle-manager ClusterRole now always grants it. - Tracing shutdown errors (#2385): at shutdown, the error of the tracer provider that carries the database driver spans was discarded, so a failed final flush lost those spans without a log. Shutdown now stops both tracer providers and returns both errors.
Security & Infrastructure
- OpenVEX attestation on every image (#2379, #2333, #2388, #2380, #2355, #2330): a scanner reports every vulnerability in an image, also those that cannot affect NVSentinel. The new
.openvex.jsonlists those vulnerabilities with the reason for each, in the OpenVEX format that Grype and Trivy read. Each published image now carries this document as a signed attestation. Get it withcosign verify-attestationand pass it to the scanner with--vex.SECURITY.mdgives the commands. In this release, the document has 115 statements for 9 images. The gpu-health-monitor image for DCGM 3.x keeps its findings, because its old base image has vulnerable code that it loads, such as OpenSSL. - Vulnerability scans (#2336, #2331, #2361, #2332, #2412): a daily scan checks the Go and Python dependencies with Grype, and the repository for malware with ClamAV. govulncheck leaves out the Go vulnerabilities in code that NVSentinel does not call. A weekly scan, and a scan at each release, check every published image. The findings appear as code scanning alerts in the Security tab. New scripts in
scripts/openvex/and agent skills inskills/openvex/help triage the findings and keep.openvex.jsoncurrent.
Chores
- CI caches (#2327, #2282): CI jobs failed when
setup-pythongot HTTP 429 while it downloaded Python. CI now caches Python and retries the download once. It saves the Go cache only onmain, so the caches stay within the 10 GB Actions cache limit. - nccl-allreduce base image updates (#2387): a build argument set the PyTorch base image of nccl-allreduce, and Dependabot cannot update a build argument. The image is now on the
FROMline, and Dependabot moved it to PyTorch NGC26.10-py3, the image that nccl-loopback already uses. - Tilt monitoring stack (#2350): the Tilt environment uses an earlier kube-prometheus-stack release, because the latest release has no arm64 images.
Acknowledgments
This release includes contributions from:
- @0z5a
- @blacksoxx
- @cbumb
- @deesharma24
- @extrali
- @glassharbor
- @jianzhangbjz
- @jtschelling
- @lalitadithya
- @pcbeingused333
- @Saibernard
- @SelvanayakiGurusamy
- @TheVicky1
- @XRFXLP
- @ZeXi0308
Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @0z5a, @extrali, and @SelvanayakiGurusamy.
Container Images
See versions.txt for the full list of container images and versions.
Helm Chart
Install with:
helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
--version v1.32.0 \
--namespace nvsentinel \
--create-namespaceTo upgrade from v1.31.0:
helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
--version v1.32.0 \
--namespace nvsentinel \
--reuse-values