Release v1.25.0

Release v1.25.0

This release adds GPU Operator GPUCluster (DRA) mode support to specific modules (labeler and DCGM discovery), completes the MaintenanceRequest feature so it can be deployed, and continues the deployment platform connector work (ADR-052). Several publisher-side changes reach the existing socket path and its metrics today — please read the Upgrade Notes below.

Upgrade Notes

The deployment-platform-connector work refactored every health monitor onto a shared publishing client. The deployment mode itself is off and not yet wired into the Helm charts, so monitors keep publishing to the node-local socket — but three changes reach that existing path on upgrade.

Publisher metrics changed (#1900)

docs/METRICS.md gains a Health Event Publisher table (nvsentinel_health_events_publisher_*: sends, errors, skips, drops-by-reason, and retries, labelled by monitor). In the health-events-analyzer, the failure counter's per-attempt retryable_error and non_retryable_error values are removed — the shared client retries inside one call — and replaced by event_rejected and event_publishing_to_UDS_error. Update any dashboard or alert built on the old analyzer counter values.

A permanent rejection is no longer retried forever (#1900)

When the platform connector refuses a batch for good — an invalid event, or one naming a node the caller may not report on — the client now reports ErrPublishRejected and the monitor lets its state advance instead of requeuing the same event indefinitely. This applies on the socket path too, where such a rejection was previously retried or requeued. It removes a class of stuck-monitor loops, but it means an event the connector rejects is dropped rather than retried; the drop is counted.

PostgreSQL builds a new index on startup (#1896)

On PostgreSQL datastores, the table setup in fault-quarantine, fault-remediation, health-events-analyzer, and node-drainer now also creates the health-event idempotency index. It is built CONCURRENTLY so it does not block other components' inserts, and a setup lock serialises the four concurrent builds. No action is required; expect a one-time concurrent index build the first time each component starts on this version.

Major New Features

GPU Operator GPUCluster (DRA) Mode Support — In Progress (#1890, #1893, #1895)

GPU Operator GPUCluster (DRA) mode support is being added incrementally and is not yet complete. This release adds it to specific modules — the labeler's device-count classes and DCGM version detection, and DCGM service discovery — so those components work against both clusterpolicy (the default) and gpucluster (DRA) mode without per-mode source changes:

• Mode flag for labeler classes (#1890): documentation for using the gpu device-count class in clusterpolicy mode and the gpu-dra class in gpucluster mode.
• DCGM version detection in both modes (#1893, #1861): the labeler's DCGM app-label flag now accepts a comma-separated list (e.g. --dcgm-app-label=nvidia-dcgm,nvidia-dcgm-dra), so it finds the DCGM pod in either mode with no config change. The dcgmAppLabel, driverAppLabel, and gkeInstallerAppLabel args, previously hardcoded in Go, are now exposed as Helm values.
• DCGM service discovery in both modes (#1895, #1863): the gpu-health-monitor, the preflight dcgm-diag check, and the lifecycle-manager diag Job previously hardcoded nvidia-dcgm.gpu-operator.svc:5555. They now take a candidate list defaulting to the GPUCluster service name first and the ClusterPolicy name second, and use whichever resolves at runtime.

MaintenanceRequest Is Now Deployable (#1735)

Completes the MaintenanceRequest feature (controller and webhook landed in v1.22.0–v1.24.0) with the Helm chart that deploys it, gated behind controllers.maintenanceRequest.enabled (default false). The chart also configures the platform-connector authentication the feature requires: a MaintenanceRequest may name any node, but lifecycle-manager runs as a Deployment on one node, so platform-connector only accepts its cross-node health events when the publisher is on the cross-node allowlist and presents a ServiceAccount token minted for the configured audience. Without both halves the failure is silent — the MR is admitted, its HealthEventEmitted condition goes True, and the node never cordons.

Configurable PodMonitor Relabelings (#1916, #1915)

podMonitor.relabelings and podMonitor.metricRelabelings are now rendered on both the metrics and cleanup-metrics endpoints. Previously an operator needing a relabeling had to disable the chart's PodMonitor and maintain a private copy, which then silently stopped scraping any endpoint the chart added later. The motivating case: much of the DaemonSet-exported metric surface (including dcgm_health_active_events) carries no node identity, so a single target relabeling adds it for every metric at once rather than requiring a pod-to-node join at query time:

podMonitor:
  relabelings:
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node

Deployment Platform Connector — Landing Incrementally (#1858, #1900, #1911, #1896)

ADR-052 introduces an optional central platform-connector Deployment that monitors reach over the network with TLS and a token, as an alternative to the node-local DaemonSet socket. This release lands the pieces: the connector's deployment role (PC_MODE=deployment) with token-auth, idempotent writes, and index-gated readiness (#1858); direct publishing from every Go monitor, the analyzer, and the Python gpu-health-monitor (#1900); direct publishing from preflight checks running in tenant pods (#1911); and creation of the idempotency index from the datastore setup (#1896). The deployment mode is not yet wired into the Helm charts and cannot be enabled through the supported path in this release — every monitor continues to publish to the node-local socket. The socket-path and metric changes that do apply now are covered under Upgrade Notes.

Metrics

• NVSwitch faults recorded in dcgm_health_active_events (#1910, #1902): the gauge tracked GPU faults but not NVSwitch faults. The NVSwitch branch now sets the gauge on detection and zeroes it on recovery, and — matching the trap fixed for GPUs in v1.24.0 — a full recovery clears every code the pre-reset cache entry named, rather than only an assumed one. No remediation semantics change; the events remain STORE_ONLY.
• Dropped Kubernetes writes labelled by health direction (#1917, #1881): k8s_platform_connector_dropped_writes_total gains an is_healthy label so a dropped clear and a dropped set are distinguishable. They are not symmetric — a dropped set is re-reported by the next change, but a dropped clear on a change-only monitor may never be re-reported — so a fleet dropping writes during planned restarts previously looked identical to one latching stale faults.

Bug Fixes & Reliability

• Cordon labels removed on manual uncordon (#1873, #1872): a manual kubectl uncordon removed only the nvsentinel-state label and left cordon-by, cordon-reason, and cordon-timestamp behind, so the node returned to service permanently labelled cordon-by=NVSentinel and anything attributing cordons from that label mis-attributed the node's next cordon. The manual path now removes the same three labels the automatic path does — but only when NVSentinel set cordon-by, since some users reuse that key.
• NPD policies treat an Unknown condition as unhealthy (#1891, #1876): the NPD KOM policies added in v1.24.0 matched status == "True", and because the Kubernetes Object Monitor treats any non-match as the healthy edge, an NPD probe that timed out or crashed (publishing status: "Unknown") on an already-cordoned node produced a false recovery event. The predicates now match status != "False", so an Unknown status is not read as recovery.

Performance

• Health-events-analyzer query optimization (#1901): the four XID 74 rules use a $lookup self-join whose sub-query reads only the node name and GPU UUID, but it re-ran once per matching document and re-read the node's full 24-hour window each time, so cost grew with the square of the node's event rate. The available index also included the entitiesimpacted array, which multiplied index keys per document. A new index fitting the rule filters, plus a reworked lookup, make these queries cheaper.

Documentation

• Documentation consistency fixes (#1894): corrects inconsistencies across the documentation.

Acknowledgments

This release includes contributions from:

• @cbumb
• @HesandaLiyanage
• @iacker
• @KaivalyaMDabhadkar
• @lalitadithya
• @lfriedman-netllama
• @tanishagoyal2

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.25.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.24.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.25.0 \
  --namespace nvsentinel \
  --reuse-values