Release v1.19.0

Release v1.19.0

This release adds detection for a hung GPU driver — a failure that leaves nodes Ready with GPUs allocatable while no work can run — and makes the Kubernetes Object Monitor's cache-sync timeout configurable so it no longer crash-loops at 100k-node scale. It also removes a quadratic etcd read amplification in the platform connector that was measured at ~500x.

Major New Features

Detect an Unresponsive GPU Driver (#1574, #1576)

Adds a watchdog to the GPU health monitor that watches node-local DCGM operations for hard hangs — a failure mode where the driver stops responding but Kubernetes still reports the node Ready with its GPUs allocatable, so the scheduler keeps placing work that cannot run. When a probe stalls past its configured deadline in embedded mode, the monitor emits a persistent GpuDriverUnresponsive event; hangs against a remote DCGM endpoint are classified as connectivity failures instead. The check ships in STORE_ONLY mode by default (observe-only, no remediation), exposes probe and liveness/readiness timing through Helm values, publishes Prometheus metrics for stalled GPU probes, and comes with operator configuration and runbook documentation. Stored health events are also updated when their recommended action changes.

Configurable Cache-Sync Timeout for Kubernetes Object Monitor (#1603, #1589)

Exposes cacheSyncTimeout as a Helm value for the Kubernetes Object Monitor. controller-runtime's 2-minute default is too short for KOM at 100k-node scale, so the monitor crash-looped on every restart. KOM stores each watched object as unstructured.Unstructured — a map[string]interface{} with a separate heap allocation per field — and CPU profiling during cache sync showed roughly 80% of time in the JSON parser rather than network I/O (an equivalent typed informer decodes the same objects in ~5 seconds). Syncing 99k nodes takes ~56 seconds, and adding ~600k gpu-operator pods pushes the total well past the default. Because CacheSyncTimeout applies per informer source rather than globally, the node source fits inside 2 minutes while the pod source does not — making the timeout configurable is what unblocks restarts at this scale.

Bug Fixes & Reliability

• Node-event dedupe no longer triggers full-range etcd scans (#1582): K8sConnector.writeNodeEvent deduplicated node events by listing every event for the node with an involvedObject.name field selector. That selector is not index-served and core events bypass the apiserver watch cache, so each call made the apiserver read the entire default-namespace events range out of etcd and filter it in memory — once per unhealthy, non-fatal health event per node. On a healthy fleet this is nearly free, but during a fleet-wide health-event storm every affected node's connector issued full-range scans (plus conflict retries) exactly when etcd was already under incident load and the events range was largest, making the cost quadratic in cluster size. The connector now caches the name of the last event written per (node, type, reason, message) and resolves the dedupe read with a GET call.
• Guarantee delivery of DCGM connectivity failures before cleanup (#1573, #1575): The GPU health monitor could lose a connectivity-failure event when DCGM cleanup blocked inside the driver, so the very signal indicating the monitor was in trouble never reached the pipeline. Connectivity failures are now published synchronously before cleanup, bypassing the shared callback executor for critical delivery and bounding gRPC retries to the liveness budget, while ordinary health events keep their existing asynchronous, unbounded retry behavior. Delivery-failure detection is more reliable (covering callback errors and unavailable connections), failed notifications are retried on the next monitoring cycle, and the multiple-remediation detection window is widened from 2 to 10 minutes.
• Lambda CSP health monitor sends workspace ID (#1604): Follow-up to the Lambda support added in v1.18.0. Maintenance-event queries can now be scoped to a workspace, with the workspace ID accepted in dashed or undashed UUID form and applied consistently across paginated requests. Invalid workspace IDs are rejected at configuration time while an unset value preserves the previous behavior, and the configured workspace appears in log context.

Documentation

• Document assumeDriverInstalled for host-installed drivers (#1586, #1583): The labeler docs told operators to set nvsentinel.dgxc.nvidia.com/driver.installed=true by hand on nodes with no driver pod — but that label silently self-reverts, because the labeler computes an empty desired value for such a node and deletes the label as stale on the next reconciliation (any labeler restart, a driver/DCGM pod readiness transition, or a node event that changes compared labels). The guidance now points to the labeler.assumeDriverInstalled chart value, which the labeler applies persistently and which also covers nodes added later, and explains why a bare kubectl label cannot be relied on. The value is also now documented in the labeler Helm configuration reference.

Acknowledgments

This release includes contributions from:

• @alexander-demicev
• @HarshavardhanK
• @lalitadithya
• @natherz97
• @XRFXLP
• @YQ-Wang
• @yuanchen8911

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @HarshavardhanK and @YQ-Wang.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.19.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.18.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.19.0 \
  --namespace nvsentinel \
  --reuse-values