Skip to content

Release v1.32.0

Latest

Choose a tag to compare

@github-actions github-actions released this 16 Nov 11:53
· 8 commits to main since this release
v1.32.0
716e483

Release v1.32.0

This release replaces nodes on AWS for REPLACE_VM events, makes node-drainer and fault-remediation record their decisions on the health event, and attaches a signed OpenVEX document to every published image. It also stops a new quarantine from removing an operator's own taint, keeps platform-connectors from storing a recovery before its fault, fixes a PostgreSQL change-stream stall under load, and lets gpu-health-monitor reconnect after an unclean DCGM disconnect.

Note

DCGM 3.x support in GPU Health Monitor is deprecated. It will be removed on 30 November 2026. DCGM 3.x is end-of-life and gets no more patches. After the removal, NVSentinel does not detect faults on nodes that run DCGM 3.x. To find these nodes, run kubectl get nodes -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x. Upgrade DCGM on these nodes to 4.x before you upgrade to the release that removes DCGM 3.x support. For the steps in each global.dcgm.mode, see GPU Health Monitor: DCGM 3.x Support Removal. To give feedback, comment on #2102.

Major New Features

Node Replacement on AWS (#2342)

On AWS, every TerminateNode failed with SendTerminateSignal not implemented for AWS, so a REPLACE_VM health event cordoned and drained the node but never replaced it. janitor-provider now terminates the EC2 instance of the node with TerminateInstances, and janitor then waits for the node to go NotReady and deletes it. With fault-remediation enabled, a REPLACE_VM event on AWS now terminates the instance. The janitor-provider IAM role needs ec2:TerminateInstances.

Drain and Remediation Decisions on the Health Event (#2354, #2205, #2341, #2207)

This release adds recorded decisions (ADR-064) to node-drainer and fault-remediation:

  • node-drainer (#2354, #2205): node-drainer chose the drain method, built-in eviction or a custom drain CR, again on each pass from the node labels. A label change during a drain could make a custom drain CR and the built-in eviction run on the same node. node-drainer now chooses the method and the scope, a full drain or the entity of a partial drain, one time for each health event, and stores them in the event's drainDecision status. Later passes and cold start follow the stored decision, so a change to customDrain.nodeSelector or partialDrainEnabled during a drain does not affect that drain. If the stored method can no longer run, node-drainer waits, logs an error and counts it in node_drainer_processing_errors_total. For example, if custom drain is turned off during a Custom drain, end the quarantine and delete the CR drain-<node name>-<event ID> to end the drain.
  • fault-remediation (#2341, #2207): fault-remediation now stores the action that it selects in the event's remediationDecision status, before it starts the log collector or creates the maintenance CR. If the write fails, fault-remediation creates no CR and retries the event.

Quieter syslog-health-monitor Logs (#2372, #2371)

syslog-health-monitor polls the journal every 15 seconds, and each poll logged about a dozen INFO lines for each check, also when no new entries arrived. These routine messages are now DEBUG. To see them, set syslog-health-monitor.logLevel: debug.

Metrics

  • node_drainer_awaiting_completion_event_timestamp_seconds{node} (#2334, #2329): AllowCompletion, the chart default drain mode, has no deadline, and node-drainer exported nothing while it waited, so a drain that could never finish looked like one that was waiting. The new gauge holds the creation time of the health event behind a drain that still waits for pods to complete, and is removed when the wait ends. To find a drain that waits too long, alert on time() - node_drainer_awaiting_completion_event_timestamp_seconds > 7200. The time counts from the health event, not from the start of the drain.
  • New error labels (#2354, #2341, #2365): node_drainer_processing_errors_total gains the error_type values record_drain_decision_error, recorded_custom_drain_unavailable, recorded_builtin_no_pod_policies and recorded_drain_method_unknown. fault_remediation_processing_errors_total gains the error type record_decision_error. In gpu-health-monitor, dcgm_api_failures_total gains the error_name values stale_group_cleanup and dcgm_initialize_monitoring, and dcgm_api_latency gains the operation_name value stale_group_cleanup.

Bug Fixes & Reliability

  • A new quarantine keeps an operator's taint (#2287, #2286): on a new quarantine, fault-quarantine treated a node taint as already present only when its key, value and effect all matched a rule-set taint. A taint with the same key and effect but a different value, for example an operator's nvidia.com/gpu-xid-error=owned-by-operator:NoSchedule, was overwritten, recorded as an NVSentinel taint, and then removed at unquarantine. Fault-quarantine now leaves such a taint on the node, as it already did for a node that was already quarantined.
  • platform-connectors stores events in order and drains its queues at shutdown (#2322, #2321): when an insert failed, the store connector took the next batch while the failed batch waited for its retry, so a recovery could be stored before the fault it followed. The connector now retries a failed batch before it takes the next one. It drops a document that the datastore can never store at once, with an error log. At shutdown, platform-connectors could stop before its queues were empty and lose batches that it had already acknowledged. It now waits up to 10 seconds for the queues to drain, and the platform-connectors DaemonSet gets a termination grace period of 45 seconds.
  • The PostgreSQL change stream keeps up under load (#2275, #2171): with the PostgreSQL datastore, the LISTEN loop fetched each notified row itself and waited for the consumer. When fault-quarantine fell behind, the loop stopped reading, fault-quarantine processed only about 300 events a minute, and the PostgreSQL backend held an open transaction that blocked VACUUM and CREATE INDEX CONCURRENTLY. The LISTEN loop now only records notifications, a separate loop fetches the rows, and the watcher polls until it is caught up, also at startup. A notification can no longer move the read position past older rows that nobody has read, so a restart behind a backlog no longer loses those rows.
  • gpu-health-monitor reconnects after an unclean DCGM disconnect (#2365, #2338): the DCGM hostengine keeps the groups of a connection until it sees the connection close, and a field-group name must be unique. After an unclean disconnect, for example a liveness restart during a DCGM outage longer than about 2 minutes, every new connection failed with Duplicate key passed to function. The node kept a false fatal GpuDcgmConnectivityFailure condition until someone restarted DCGM. gpu-health-monitor now deletes its own stale groups before it creates new ones. It deletes only the groups named dcgm_health, nvsentinel_field_monitors and gpu_temp_limit.
  • MongoDB setup Job deadline (#2274, #2257): the create-mongodb-database Job had a fixed deadline of 600 seconds. When image pulls queued on a cold node, MongoDB was not reachable in time, the Job failed, Helm did not run it again, and every module crash-looped with UserNotFound. The deadline is now mongodb-store.job.activeDeadlineSeconds, default 1800. The setup Job for an external MongoDB reads global.datastore.setupJob.activeDeadlineSeconds, default 600. A value of 0 removes the deadline.
  • The generic reboot pod on a node under disk pressure (#2397, #2184): the kubelet rejects a pod without a critical priority on a node that reports DiskPressure, so the generic reboot pod never ran on such a node. The new janitor-provider.csp.generic.rebootJobPriorityClassName value sets the priority class of the reboot pod. Set it to system-node-critical or system-cluster-critical. The class must exist, and the reboot Job namespace must be allowed to use it. The value is empty by default.
  • GPU faults clear after a GPU index change (#2377): after a reboot, a GPU can come back with a new index or PCI address but the same UUID. Its healthy event did not clear the old entry from the node condition, so the condition stayed True. A healthy GPU event now also clears an entry that names the same full GPU UUID. An entry with a shortened UUID does not match.
  • No false fabric probe failures on GPUs without NVLink (#2366, #2360): DCGM raised DCGM_FR_FABRIC_PROBE_STATE on GPUs without NVLink, for example L40 PCIe GPUs with driver 535. gpu-health-monitor now suppresses it when the GPU has no NVLink hardware and NVML reports 0 NVLinks present. metadata-collector records that count in the GPU metadata as nvlink_present_link_count.
  • One event for a GPU that falls off the bus after an XID (#2369, #2367): syslog-health-monitor skips a "fallen off the bus" line when the same GPU had an XID in the last 5 minutes. The two lines write the PCI address in different forms, so they never matched, and the node got a second GPU_FALLEN_OFF_BUS event for the same fault. The addresses are now compared in one form. With the XID check disabled, the monitor always sends the fallen-off-the-bus event.
  • nic-health-monitor publishes during shutdown (#2364): SIGTERM cancelled every publish, so the monitor lost the NICPollStallCheck event and the recoveries of a poll that completed during shutdown. These events come when liveness restarts the monitor. Publishes that start after SIGTERM now share one 10-second window, and the DaemonSet sets its termination grace period to 30 seconds.
  • node-drainer starts with custom drain enabled (#2370, #2368): with customDrain.enabled: true, node-drainer failed at startup with invalid duration format: 30m, because the chart default of customDrain.timeout went through a parser that accepts only seconds. node-drainer never used this value, so it is removed. The chart and node-drainer ignore it if you still set it.
  • lifecycle-manager records its leader election event (#2399, #2398): the chart granted events only with controllers.maintenanceRequest.enabled, which is off by default, so the API server rejected the LeaderElection event. The lifecycle-manager ClusterRole now always grants it.
  • Tracing shutdown errors (#2385): at shutdown, the error of the tracer provider that carries the database driver spans was discarded, so a failed final flush lost those spans without a log. Shutdown now stops both tracer providers and returns both errors.

Security & Infrastructure

  • OpenVEX attestation on every image (#2379, #2333, #2388, #2380, #2355, #2330): a scanner reports every vulnerability in an image, also those that cannot affect NVSentinel. The new .openvex.json lists those vulnerabilities with the reason for each, in the OpenVEX format that Grype and Trivy read. Each published image now carries this document as a signed attestation. Get it with cosign verify-attestation and pass it to the scanner with --vex. SECURITY.md gives the commands. In this release, the document has 115 statements for 9 images. The gpu-health-monitor image for DCGM 3.x keeps its findings, because its old base image has vulnerable code that it loads, such as OpenSSL.
  • Vulnerability scans (#2336, #2331, #2361, #2332, #2412): a daily scan checks the Go and Python dependencies with Grype, and the repository for malware with ClamAV. govulncheck leaves out the Go vulnerabilities in code that NVSentinel does not call. A weekly scan, and a scan at each release, check every published image. The findings appear as code scanning alerts in the Security tab. New scripts in scripts/openvex/ and agent skills in skills/openvex/ help triage the findings and keep .openvex.json current.

Chores

  • CI caches (#2327, #2282): CI jobs failed when setup-python got HTTP 429 while it downloaded Python. CI now caches Python and retries the download once. It saves the Go cache only on main, so the caches stay within the 10 GB Actions cache limit.
  • nccl-allreduce base image updates (#2387): a build argument set the PyTorch base image of nccl-allreduce, and Dependabot cannot update a build argument. The image is now on the FROM line, and Dependabot moved it to PyTorch NGC 26.10-py3, the image that nccl-loopback already uses.
  • Tilt monitoring stack (#2350): the Tilt environment uses an earlier kube-prometheus-stack release, because the latest release has no arm64 images.

Acknowledgments

This release includes contributions from:

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @0z5a, @extrali, and @SelvanayakiGurusamy.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.32.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.31.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.32.0 \
  --namespace nvsentinel \
  --reuse-values