Release v1.20.0

Release v1.20.0

This release is dominated by a large-scale memory reduction effort: informer caches across janitor, fault-quarantine, node-drainer, labeler, and preflight now strip unused fields before caching, cutting settled working set by up to 7.6× on a 99k-node benchmark. It also authenticates health-event publishers and binds each event to its reporting node, adds Lambda workload-identity authentication, exposes client-go throughput limits as Helm values, and fixes node-condition recovery after a physical GPU replacement.

Major New Features

Informer Cache Memory Reduction Across Components (#1621, #1614, #1615, #1616, #1622)

Informer caches were the dominant memory consumer at scale, because each component cached full Node and Pod objects even though rule evaluation reads only a handful of fields. Every affected component now applies SetTransform (and, where applicable, cache.ByObject scoping) so objects are stripped before they enter the cache. Benchmarked on a KWOK-dominated cluster of ~99k nodes:

• Janitor (#1621, #1605) — the single largest consumer. With GPUReset enabled and ~397k gpu-operator pods, settled working set drops from 8,581 MiB to 1,134 MiB (7.6×). Nodes are stripped to identity, labels, taints, and the NodeReady condition; the pod watch is scoped and transformed for the GPUReset path only.
• Fault-quarantine (#1614, #1607) — node.Status is discarded before caching (no rule predicate reads it), taking working set from 3,909 MiB to 864 MiB (4.5×), or 38 MB → 7.2 MB per 1k nodes.
• Labeler (#1616, #1609) — unstripped node and pod caches occupied ~7.1 GiB at 99k nodes. The node transform is feature-aware: identity, labels, and the DCGMBootstrapCompleted annotation are always kept, and status.allocatable/status.capacity are retained only when a device-count class needs them.
• Node-drainer (#1615, #1606) — nodes keep only identity plus the quarantine annotation; system-namespace and DaemonSet-owned pods (never drain targets) are reduced to bare identity stubs, while drain-eligible pods retain just the fields drain decisions need.
• Preflight (#1622, #1608) — a two-level pod transform keeps gang-relevant fields for pods in preflight-enabled namespaces and reduces all other pods to identity stubs.

Authenticated Health-Event Publishers (#1636)

Health events are submitted to the platform connector over a per-node Unix domain socket, and each event carries a nodeName naming the machine it describes. The connector previously had no way to establish which component sent an event, so it took nodeName at face value — and because health events drive cordon, drain, and reboot, a wrong node name means disruptive work landing on the wrong machine. Every publisher now mounts a short-lived projected ServiceAccount token minted for a dedicated platform-connector.nvsentinel.nvidia.com audience and rotated by the kubelet, attached as a bearer credential on each publish. A server-side interceptor validates the token via TokenReview and then decides which nodes that caller may report on. The change is contained to the health-event path — how events are stored, evaluated, and remediated once accepted is unchanged.

Lambda Workload-Identity Authentication (#1624)

Adds workload-identity authentication to the Lambda CSP health monitor and janitor provider, exchanging signed JWTs for short-lived, scoped API credentials that refresh automatically. Static API keys remain supported, and the change adds workload-identity configuration for Kubernetes deployments plus secure Lambda endpoint validation and normalization.

Configurable client-go QPS and Burst Limits (#1632, #1600)

Fault-quarantine, node-drainer, labeler, and preflight now expose client-go QPS and burst limits as Helm values, so operators can tune Kubernetes API throughput without rebuilding binaries. The chart supplies global defaults (5 QPS, 10 burst) that each component may override, with presence-based precedence preserving explicit zero values. client-go semantics are followed: zero uses the client-go defaults and a negative QPS disables client-side throttling.

Fault-Remediation Pipeline Optimization (#1627, #1592)

Cold-start replay previously retained every fully decoded health-event document in the controller workqueue, so the whole pending backlog sat in memory as generic maps even though the discovery query itself was already batched. The pipeline now queues only each event's canonical document ID and lazily fetches one full document when a worker is ready to run the existing reconciliation path, bounding workqueue memory by backlog count rather than document size.

Bug Fixes & Reliability

• Node-condition recovery after a physical GPU replacement (#1587, #1585): Replacing a GPU changes its GPU_UUID, so a healthy event from the new hardware could never match — and therefore never clear — the node condition raised against the old GPU. Recovery matching now ignores GPU_UUID when a stable GPU index or PCI address is present, while preserving UUID-only matching where no stable identity exists and leaving non-GPU matching unchanged.
• Driver-pod detection under the NVIDIADriver CRD (#1588, #1565): Driver-pod detection in the labeler and log collector broke under GPU Operator's nvidiaDriverCRD: true mode. Both now locate driver pods via the stable app.kubernetes.io/component=nvidia-driver label; legacy driver pods keep their existing path and the CRD informer excludes them to avoid duplicate events.
• Labeler reconciles device counts on allocatable changes (#1602): nodeRequiresReconciliation only checked for node label changes, but the NIC count CEL expression reads node.status.allocatable. When a MOFED driver restart temporarily zeroed allocatable, the labeler wrote nic.count.current=0 and never corrected it after recovery because no labels had changed — leaving stale counts on 20+ nodes. Reconciliation is now also triggered by status.allocatable/status.capacity changes, but only for classes whose CEL expression references node.status, so label-only classes (e.g. GPU) pay no additional cost.
• Kata label update no longer short-circuits driver/DCGM labels (#1638): A needsUpdate = needsUpdate || l.updateDriverAndDCGMLabels(...) expression meant that once needsUpdate was already true, Go's short-circuit evaluation skipped the driver/DCGM label update entirely — so a genuine driver or DCGM label diff was silently dropped in the same pass. The call is now made unconditionally, with a regression test covering both labels applying in one pass.
• Serialize per-node label reconciliation (#1646): During labeler startup, reconcileAllNodes could read that a driver pod exists and then race with its deletion — the delete handler correctly removed driver.installed, but the startup sweep subsequently restored driver.installed=true from its stale earlier read, and because that output label is not a watched reconciliation input it could stay wrong indefinitely absent a replacement pod event. Reconciliation for a given node is now serialized across the startup sweep and the pod add/update/delete handlers using striped locks, so whichever path runs second recalculates from the latest informer state; different nodes still reconcile concurrently.
• XID 45 treated as non-fatal (#1629): XID 45 was classified as fatal — draining and rebooting the node — because MapActionStringToProto had no case for the WORKFLOW_XID_45 guidance class and fell through to CONTACT_SUPPORT, which determineFatality treats as fatal. The catalog asks for no node-level action for XID 45 (RESTART_APP/REPORT_ISSUE solo, IGNORE when not solo), and since both RESTART_APP and IGNORE already map to NONE, the bucket resolves correctly without needing solo/non-solo correlation.
• Thermal clocks-event mapping aligned to NONE (#1637): DCGM_FR_CLOCKS_EVENT_THERMAL (the DCGM 4.x name) and DCGM_FR_CLOCK_THROTTLE_THERMAL (the 3.x name) are the same underlying error code, but the two dcgmerrorsmapping.csv rows disagreed — one mapped to CONTACT_SUPPORT, the other to NONE. Since the GPU health monitor resolves the integer code to a single name, the rows are now aligned on NONE.
• Code-block-aware MDX sanitization in Fern docs CI (#1635, [#1634]...

Release v1.19.0

Release v1.19.0

This release adds detection for a hung GPU driver — a failure that leaves nodes Ready with GPUs allocatable while no work can run — and makes the Kubernetes Object Monitor's cache-sync timeout configurable so it no longer crash-loops at 100k-node scale. It also removes a quadratic etcd read amplification in the platform connector that was measured at ~500x.

Major New Features

Detect an Unresponsive GPU Driver (#1574, #1576)

Adds a watchdog to the GPU health monitor that watches node-local DCGM operations for hard hangs — a failure mode where the driver stops responding but Kubernetes still reports the node Ready with its GPUs allocatable, so the scheduler keeps placing work that cannot run. When a probe stalls past its configured deadline in embedded mode, the monitor emits a persistent GpuDriverUnresponsive event; hangs against a remote DCGM endpoint are classified as connectivity failures instead. The check ships in STORE_ONLY mode by default (observe-only, no remediation), exposes probe and liveness/readiness timing through Helm values, publishes Prometheus metrics for stalled GPU probes, and comes with operator configuration and runbook documentation. Stored health events are also updated when their recommended action changes.

Configurable Cache-Sync Timeout for Kubernetes Object Monitor (#1603, #1589)

Exposes cacheSyncTimeout as a Helm value for the Kubernetes Object Monitor. controller-runtime's 2-minute default is too short for KOM at 100k-node scale, so the monitor crash-looped on every restart. KOM stores each watched object as unstructured.Unstructured — a map[string]interface{} with a separate heap allocation per field — and CPU profiling during cache sync showed roughly 80% of time in the JSON parser rather than network I/O (an equivalent typed informer decodes the same objects in ~5 seconds). Syncing 99k nodes takes ~56 seconds, and adding ~600k gpu-operator pods pushes the total well past the default. Because CacheSyncTimeout applies per informer source rather than globally, the node source fits inside 2 minutes while the pod source does not — making the timeout configurable is what unblocks restarts at this scale.

Bug Fixes & Reliability

• Node-event dedupe no longer triggers full-range etcd scans (#1582): K8sConnector.writeNodeEvent deduplicated node events by listing every event for the node with an involvedObject.name field selector. That selector is not index-served and core events bypass the apiserver watch cache, so each call made the apiserver read the entire default-namespace events range out of etcd and filter it in memory — once per unhealthy, non-fatal health event per node. On a healthy fleet this is nearly free, but during a fleet-wide health-event storm every affected node's connector issued full-range scans (plus conflict retries) exactly when etcd was already under incident load and the events range was largest, making the cost quadratic in cluster size. The connector now caches the name of the last event written per (node, type, reason, message) and resolves the dedupe read with a GET call.
• Guarantee delivery of DCGM connectivity failures before cleanup (#1573, #1575): The GPU health monitor could lose a connectivity-failure event when DCGM cleanup blocked inside the driver, so the very signal indicating the monitor was in trouble never reached the pipeline. Connectivity failures are now published synchronously before cleanup, bypassing the shared callback executor for critical delivery and bounding gRPC retries to the liveness budget, while ordinary health events keep their existing asynchronous, unbounded retry behavior. Delivery-failure detection is more reliable (covering callback errors and unavailable connections), failed notifications are retried on the next monitoring cycle, and the multiple-remediation detection window is widened from 2 to 10 minutes.
• Lambda CSP health monitor sends workspace ID (#1604): Follow-up to the Lambda support added in v1.18.0. Maintenance-event queries can now be scoped to a workspace, with the workspace ID accepted in dashed or undashed UUID form and applied consistently across paginated requests. Invalid workspace IDs are rejected at configuration time while an unset value preserves the previous behavior, and the configured workspace appears in log context.

Documentation

• Document assumeDriverInstalled for host-installed drivers (#1586, #1583): The labeler docs told operators to set nvsentinel.dgxc.nvidia.com/driver.installed=true by hand on nodes with no driver pod — but that label silently self-reverts, because the labeler computes an empty desired value for such a node and deletes the label as stale on the next reconciliation (any labeler restart, a driver/DCGM pod readiness transition, or a node event that changes compared labels). The guidance now points to the labeler.assumeDriverInstalled chart value, which the labeler applies persistently and which also covers nodes added later, and explains why a bare kubectl label cannot be relied on. The value is also now documented in the labeler Helm configuration reference.

Acknowledgments

This release includes contributions from:

• @alexander-demicev
• @HarshavardhanK
• @lalitadithya
• @natherz97
• @XRFXLP
• @YQ-Wang
• @yuanchen8911

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @HarshavardhanK and @YQ-Wang.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.19.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.18.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.19.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.18.0

Release v1.18.0

This release adds Lambda Cloud as a supported CSP — both a csp-health-monitor provider and a janitor plugin — along with cross-CSP maintenance-event improvements that also close a long-standing gap for GCP. It also gates node-labeler detection labels on ExtRR ownership, fixes syslog-health-monitor silently missing XIDs emitted between boot and monitor startup, and documents the Percona MongoDB backend for ARM64.

Major New Features

Lambda Cloud Support (#1545, #1569)

Adds Lambda as a supported cloud provider across both detection and remediation:

• CSP health monitor (#1545): Chart wiring mirroring the existing [gcp]/[aws] providers, a paginated GET /api/v1/maintenance_events poller, an event normalizer, and a shared HTTP client with exponential-backoff retry under commons/pkg/lambda/ for reuse by future Lambda-facing components (there is currently no SDK for the Lambda APIs).
• Janitor plugin (#1569): A Lambda janitor provider supporting instance retrieval, power cycling, termination, action validation, and asynchronous readiness tracking, built on the same commons client. Includes authenticated JSON requests with safe retry handling and HTTPS redirect protection, plus Kubernetes configuration for Lambda endpoints and Secret-based API keys.

The health monitor PR also adds two cross-CSP trigger-engine paths driven off the internal event model, available to any CSP whose normalizer emits the right fields:

• Emergency-urgency quarantine bypass — DETECTED events with metadata.urgency=EMERGENCY fire QUARANTINE on the next poll with no scheduledStartTime window check.
• CANCELLED → HEALTHY dispatch — HEALTHY now fires as soon as an event is observed in CANCELLED. This closes the same gap for GCP, whose normalizer already produces StatusCancelled: previously, cancelling a maintenance event after it had been quarantined left nvidia.com/maintenance on the node forever because no HEALTHY was ever emitted. CANCELLED fires HEALTHY immediately (no maintenance actually happened), while MAINTENANCE_COMPLETE continues to wait for actualEndTime + postMaintenanceHealthyDelayMinutes.

Gate Node-Labeler Detection Labels on ExtRR Ownership (#1551)

Continues the ExternalRemediationRequest work (ADR-040) by gating the labeler and fault-quarantine on nvsentinel.dgxc.nvidia.com/managed=false, so a node under active external ownership stops having detection labels stamped — which would otherwise keep DaemonSet health monitors scheduled on it. The labeler strips the DCGMVersion, DriverInstalled, and KataEnabled detection labels when managed=false (monitors then self-evict via their existing nodeSelector, requiring no per-monitor changes) and skips pod-driven re-stamping for opted-out nodes. Both paths fail closed on lister error, so no mutations occur when the managed-label state is unknown, preventing accidental re-enablement of monitors on a released node. All six default fault-quarantine rulesets now include a CEL expression that skips opening a new quarantine session on such nodes, and a new labeler_node_labels_skipped_managed_total counter tracks the behavior. The existing k8saas.nvidia.com/ManagedByNVSentinel check is retained for backwards compatibility.

Bug Fixes & Reliability

• Syslog-health-monitor missed XIDs emitted before startup after a reboot (#1533, #1532): After a host reboot the monitor detects the boot-ID change and clears its persisted cursors; on the next poll the absent cursor triggered initializeJournalFromTail, which seeks to the journal's end — permanently skipping any XID emitted between boot and monitor startup. A postRebootInit flag now marks this case so the monitor instead seeks to the beginning of the current boot's journal (SeekHead() with an explicit _BOOT_ID=<current> match filter) and processes all entries forward, clearing the flag once all checks complete so subsequent polls resume normal cursor-based processing.
• Metadata-collector no longer exits on a deleted pod (#1578, #1568): UpdatePodDevicesAnnotations treated a NotFound response while patching pod device annotations as fatal, so an ordinary deletion race caused the collector to exit 1. NotFound is now treated as a benign deletion race: reconciliation continues over the remaining pods, deleted pods are not counted as updated, and other update errors are still reported.
• E2E flake from kwok host networking (#1570, #1567): Installing kwok with hostNetwork enabled made the kwok pod run on the node's IP and stamp that IP onto all 55 fake nodes; kindnetd, which identifies "is this node me" by IP, then repeatedly rewrote the real node's CNI config with fake pod CIDRs. Pods created afterwards received IPs already held by kwok's fake pods, and since kind v0.30 enforces NetworkPolicies, the policy engine resolved the shared IP to the wrong pod and silently dropped janitor → janitor-provider gRPC traffic — timing out for 10–25 minutes and stalling RebootNode CRs until tests ran out of time. kwok is now installed without hostNetwork (with both kwok charts pinned to 0.3.0). Test/CI reliability only; no runtime behavior change.

Documentation

• Percona MongoDB backend for ARM64 and Bitnami → Percona migration (#1566, #1497): The default MongoDB backend (Bitnami) ships amd64-only images, so the default NVSentinel configuration cannot run on ARM64 nodes; the Percona operator backend is multi-arch and does. Adds an ARM64 support section to the MongoDB store configuration docs (including the exact image-pull error users hit) and a full migration runbook covering the objects that survive helm uninstall and must be cleaned up (PVCs, the credentials secret, all cert-manager TLS secrets including per-replica server certs, and the runtime-created resume-control and circuit-breaker ConfigMaps), handling of in-flight quarantines and remediations, cloud volume size minimums, verification steps, troubleshooting, and rollback guidance. The chart default is intentionally unchanged.

Acknowledgments

This release includes contributions from:

• @aditighag
• @alexander-demicev
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @signalgap9-del
• @sravindra05

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @aditighag, @alexander-demicev, and @sravindra05.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.18.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.17.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.18.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.17.0

Release v1.17.0

This release adds detection for missing InfiniBand character devices — a failure mode that left RDMA workloads unable to start while every existing check reported the node healthy — and fixes three fault-remediation event-handling bugs that could drop actionable events, replay events indefinitely, or leave a recovered node marked remediation-failed.

Major New Features

Detect Missing InfiniBand Character Devices (#1546)

Adds InfiniBandCharDeviceCheck, a passive NIC health monitor state check that flags an InfiniBand device whose /dev/infiniband character nodes (issm/umad/uverbs) are missing while its ports still read ACTIVE/LinkUp. The existing port-state checks report this as healthy, yet RDMA workloads cannot start on the node — pods fail with lstat /dev/infiniband/issm9: no such file or directory. In the motivating case a GPU node came up with only 8 of ~12 expected issm* nodes, requiring manual triage and a spare migration.

Because udev creates /dev/infiniband/{issm,umad,uverbs}N from the sysfs class entries under /sys/class/infiniband_mad and /sys/class/infiniband_verbs, a missing class entry is an exact proxy for a missing /dev node — and those directories are already within the host /sys the DaemonSet mounts, so no /dev mount is required. The check is a per-device internal-consistency test rather than an absolute expected-count test (deliberately avoiding the false-positive class addressed in v1.15.0): for each in-scope device exposing at least one InfiniBand-mode port it expects one uverbs per device plus one umad and one issm per InfiniBand-mode port, with issm/umad gated on the InfiniBand link layer since RoCE/Ethernet-mode ports do not create them.

Bug Fixes & Reliability

• Retry events skipped while an equivalent maintenance CR is in progress (#1550, #1536): When a remediation-ready health event arrived while an equivalent maintenance CR was still InProgress, fault-remediation skipped the event, advanced the resume token, and returned success — nothing reconsidered the event once the CR completed, so it was dropped permanently. In production after a reboot, new fatal events (XID 154 and an NVLink failure) arrived a few seconds before the old RebootNode CR received its completion condition; all three were skipped, faultremediated stayed null, no new CR was created, and the node stayed cordoned with an actionable fault for over two hours. Fault-remediation now requeues such events (30s by default, configurable via ReconcilerConfig.InProgressRequeueDelay) without advancing the resume token or writing status, so the event stays live until the CR reaches a terminal state and the existing decision logic resolves it. Events waiting this way are visible via a new cr_status="waiting" value on fault_remediation_events_processed_total, and an event's status is re-read before a remediation starts since a requeued event may sit in the workqueue for a long time.
• Propagate per-event MongoDB resume tokens (#1549, #1513): MongoDB change-stream events were delivered without their per-event resume tokens, so AdaptedChangeStreamWatcher.Events() passed an empty token to fault-remediation and safeMarkProcessed() skipped checkpoint persistence entirely. Fault-remediation therefore replayed the same events after every restart, and large replay backlogs could fill the controller workqueue and cause OOM CrashLoops. Each event's resume token is now captured immediately after decoding (cloning the driver-owned bson.Raw so it stays valid after the cursor advances), propagated through the store-client stack, and persisted as an embedded BSON document in the shape SetResumeAfter expects. Existing stored tokens remain compatible.
• Prevent stale replay from re-applying remediation-failed (#1544, #1534): On resuming a change stream, fault-remediation could process an old remediation-ready event after the node had already recovered; for an unsupported action such as CONTACT_SUPPORT it trusted the historical event status and stamped dgxc.nvidia.com/nvsentinel-state=remediation-failed onto the healthy node, and a later replayed UnQuarantined/Cancelled event cleared the remediation annotation without removing the label. Before applying the label, fault-remediation now reads the node through the uncached API reader and verifies the event belongs to the node's current active quarantine session — a persisted NodeQuarantined value on a replayed event is no longer sufficient. With no matching active quarantine, the label is not applied, the event is marked terminal with FaultRemediated=false, the resume token advances, and no CR is created; live-node read errors are returned so the event is retried rather than incorrectly finalized. Cancellation processing now also conditionally clears stale fault-remediation labels.

Documentation

• HealthEvent data model reference (#1535): Adds a single authoritative reference for the HealthEvent protobuf message — the fundamental data unit flowing through NVSentinel's detection and remediation pipeline — consolidating semantics that were previously scattered across 45+ documents, proto comments, and source. Covers all 18 fields with the module that consumes each, the RecommendedAction enum and its remediation routing logic (GetEffectiveActionName), the ProcessingStrategy enum and its enforcement points, and BehaviourOverrides force/skip semantics.

Acknowledgments

This release includes contributions from:

• @KaivalyaMDabhadkar
• @lalitadithya
• @shaq918
• @signalgap9-del

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @signalgap9-del.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.17.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.16.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.17.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.16.0

Release v1.16.0

This release adds detection of wedged mlx5 NIC drivers (TX timeout followed by a NAPI soft lockup) via the syslog health monitor, automatic nvidia-bug-report collection when a GPU reset fails, node-label actions on fault-quarantine rule matches, and custom recommended actions for Kubernetes Object Monitor policies. It also suppresses false-positive NVLink-down remediations on PCIe GPUs and hardens the NIC monitor against silent misconfiguration.

Major New Features

Detect mlx5 NIC Driver TX Timeout and NAPI Soft Lockup (#1508)

Adds a multi-line detector to the syslog health monitor that catches a wedged mlx5 NIC driver from kernel logs and quarantines the node, closing a gap where a NAPI soft lockup left a node degraded for days with no NVSentinel signal. In the originating incident (#1334), an mlx5 TX timeout was followed ~11 seconds later by BUG: soft lockup - CPU#N stuck with the kernel stack pinned inside the mlx5 NAPI poll loop; the driver's built-in recovery never kicked in and the node slowly failed (CSI socket stalls, pods stuck Pending). Because the generic soft-lockup message and the mlx5 stack frame are separate journal entries, no single-line regex could connect them — the new detector uses a small state machine that arms a 150-line window on the soft-lockup header and confirms on a non-speculative mlx5e_poll_ico_cq / mlx5e_napi_poll stack frame.

Bug-Report Collection on GPU Reset Failure (#1499)

When the GPU reset script observes a failed nvidia-smi --gpu-reset, it now collects an nvidia-bug-report and, when UPLOAD_URL_BASE is configured, uploads it (best-effort) to the file-server endpoint for later diagnosis. The upload URL is wired through the janitor GPU-reset job config and Helm values, defaulting automatically to the in-cluster file server when it is enabled.

Node-Label Actions on Fault-Quarantine Rules (#1506)

Fault-quarantine rulesets can now apply optional Kubernetes node labels when a rule matches. Multiple matching rules for the same label key are resolved by priority (configuration order as tie-breaker), applied labels are recorded in the quarantineHealthEventAppliedLabels annotation and held while any tracked health event remains unhealthy, and only the recorded labels are removed once all tracked events recover (including cleanup on manual uncordon, manual untaint, and stale-state recovery). Metrics are exported for applied and removed labels.

Custom Recommended Actions for KOM Policies (#1521, #1530)

Kubernetes Object Monitor policies can now specify a customRecommendedAction alongside recommendedAction: CUSTOM (ADR-036), routing matched events to arbitrary CR types via fault-remediation's remediationActions config without any additional code — the operator just names the action key they defined in Helm values. The field is wired through to the published HealthEvent proto and validated at config load, so a CUSTOM action with an empty string is caught at startup rather than rejected later at the platform-connector gRPC boundary. The chart's policy TOML template now conditionally renders customRecommendedAction, fixing a startup crash when a policy set CUSTOM without the field appearing in the rendered config.

Bug Fixes & Reliability

• Suppress false-positive DCGM_FR_NVLINK_DOWN on PCIe GPUs (#1481): PCIe GPUs (A100 PCIe, A40, L40S, etc.) have no NVLink hardware, but DCGM's GpuNvlinkWatch reported all links as down with isFatal=true and recommendedAction=RESTART_VM, triggering spurious remediations. The metadata collector now records an nvlink_link_count (counting links where NVML GetNvLinkState returns SUCCESS, i.e. hardware present), and the GPU health monitor suppresses DCGM_FR_NVLINK_DOWN only for GPUs with zero NVLink hardware links — conclusive across SKUs since PCIe parts return NOT_SUPPORTED while SXM/NVL return SUCCESS even for broken links.
• NIC monitor hardening: config fail-fast, management-NIC scoping, reboot reconciliation (#1528, #1529): Three fixes so the NIC monitor is never silently wrong. (1) A typo in config.toml (e.g. thresholdType = "velocty") previously made the monitor quietly fall back to defaults — disabling all counter monitoring while still reporting healthy; a malformed config now fails pod startup with a clear error, while a genuinely missing file still falls back to defaults. (2) Counter (degradation) checks now respect management-NIC classification like the state checks do, so a link blip on a management RoCE NIC can no longer emit a fatal REPLACE_VM and cordon the node. (3) Reboots and hardware changes (e.g. NIC renames) can no longer orphan node conditions that nobody can clear.
• Embedded DCGM mode runs privileged (#1509): The embedded DCGM mode added in v1.13.0 could not find NVIDIA devices without elevated privileges; the gpu-health-monitor now runs privileged in embedded mode (DCGM 3.x and 4.x) so local DCGM connectivity works. Also updates the UAT fault-injection scripts to dynamically discover the monitored DCGM endpoint and work across all supported DCGM modes, and documents the embedded-mode privilege requirement in ADR-044.

Documentation

• Documentation fixes (#1526): Spelling and grammar corrections, consistent heading hierarchy across docs, fixes for <> rendering in Fern (replaced with {}), and consolidation of the NIC monitor design docs into the root design folder.

Acknowledgments

This release includes contributions from:

• @AnouarMohamed
• @deesharma24
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @shaq918
• @ymygogogo

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @AnouarMohamed and @ymygogogo.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.16.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.15.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.16.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.15.0

Release v1.15.0

This release completes the operator-triggered cold-start/change-stream reset begun in v1.14.0, adds kubectl get extrr printer columns, and delivers a set of high-impact reliability fixes — most importantly detecting frozen health-monitor loops via the liveness probe, hardening GPU reset job creation and partial drains, and correcting NIC monitor false positives and dropped failure events. It also expands operator documentation across the PostgreSQL/MongoDB stores, fault-quarantine rule sets, and the Kubernetes Object Monitor.

Major New Features

Cold-Start Reset via resume-control ConfigMap (#1476)

Completes the operator-triggered change-stream reset introduced in v1.14.0 (#1472). Previously, setting a module's key to CREATE in the resume-control ConfigMap only reset the change-stream resume token — a subsequent restart could still replay old persisted records because cold-start recovery queries the datastore directly. Modules that support cold-start recovery now record a component-managed <module>.coldStartAfter timestamp when they consume CREATE, and future cold-start queries only consider events created after that timestamp, making the reset durable. Operators still only set the module key to CREATE; the timestamp is internal status managed by the module.

ExternalRemediationRequest Printer Columns (#1495)

kubectl get extrr now shows NODE, CHECKNAME, and COMPLETIONTIME columns (with completion time rendered in a readable date format), so operators can see the state of ExternalRemediationRequest objects at a glance without inspecting each one.

Bug Fixes & Reliability

• Detect frozen health-monitor loops via /healthz (#1477): The gpu-, syslog-, and nic-health-monitor liveness probes targeted /metrics, served by an independent HTTP goroutine, so when a monitor's main polling loop froze the probe kept returning 200 and Kubernetes never restarted the pod. On one 370-GPU-node cluster, 13 gpu-health-monitor pods showed Running 1/1 with zero restarts while their reconciliation loops had been frozen for 7–24 days — those nodes had zero GPU health monitoring despite appearing healthy. A new reusable PollingHealthChecker tracks the last successful poll timestamp and makes /healthz return unhealthy once it exceeds a staleness threshold (3× the poll interval), so a stalled loop now trips the liveness probe and restarts the pod.
• GPUReset job creation failures and invalid partial drains (#1500): Two fixes for GPU reset. (1) Depending on the node name, the reset-job name truncation could produce an invalid RFC 1123 subdomain (e.g. a trailing .- sequence), so the controller failed to create the job indefinitely — and because this phase has no timeout, the CRD never reached a terminal state. Job names are now always valid RFC 1123 subdomains. (2) Full and partial drains skipped pods that were stuck terminating or NotReady, which may still hold running processes with GPU contexts and can cause a GPU reset to fail; partial drains now ensure targeted pods are removed or in a terminal state before the reset proceeds.
• NIC monitor no longer misses or drops failure events (#1492, #1488): Fixes six findings where the NIC health monitor could silently lose or never emit failure events or fabricate fatal disappearance events from transient sysfs read errors. Notably, fatal counter events were relabeled under the state check's name, so an ordinary ACTIVE/LinkUp recovery cleared a counter-originated condition while the breach latch kept suppressing further breaches — the link_downed safety net worked only once per boot; fatal counter events now keep their *DegradationCheck identity so a state recovery can no longer clear them. FATAL/recovery events were also persisted before publish (violating the healthpub caller contract) so a platform-connector outage permanently consumed the health boundary; checks are now transactional (Prepare/Commit).
• NIC monitor requires positive evidence for FATALs; adds inclusion override (#1462, #1361, #1379): A port the monitor had never seen healthy was reported FATAL unless proven expected-down, producing false alarms on ports that are down by design (e.g. unprovisioned Ethernet/RoCE aux ports on some cloud shapes). The rule is inverted: a first-sight DOWN port is now failed only when there is positive peer evidence it should be up (grouping cards by role and flagging those below the untied modal active-port count), otherwise it is suppressed; runtime ACTIVE → DOWN remains always fatal. Adds a nicInclusionRegexOverride to pin monitoring to an explicit NIC list, and fixes restart/config-change state bugs (fabricated "device disappeared" FATALs, lost counter latches, and card-level FATALs that could hold a node quarantined forever with no recovery event).
• Circuit breaker counts only GPU nodes (#1482, #1228): The fault-quarantine circuit breaker computed its thresholds, trip, and progress against all cluster nodes rather than GPU nodes, skewing the denominator on mixed clusters. It now counts only nodes labeled nvidia.com/gpu.present=true, with the label configurable via fault-quarantine.gpuNodeLabel (Helm) / --gpu-node-label (CLI).

Documentation

• PostgreSQL Store configuration guide (#1490): Adds a configuration guide for the PostgreSQL datastore.
• Percona Operator for MongoDB store (#1489): Adds Percona Operator documentation to the MongoDB store guide.
• Managing fault-quarantine rule sets (#1486): Documents how to add, modify, disable, apply, and verify fault-quarantine rule sets.
• Using the Kubernetes Object Monitor (#1483): Adds a guide for configuring and using the Kubernetes Object Monitor.
• Custom remediation plugin tutorial (#1493): Adds a tutorial for writing a custom remediation plugin.

Acknowledgments

This release includes contributions from:

• @jtschelling
• @KaivalyaMDabhadkar
• @natherz97
• @nitz2407
• @rupalis-nv
• @shaq918
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.14.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.14.0

Release v1.14.0

This release adds TTL-based health-event deduplication in the platform connector, re-lands GPU-reset-safe kernel filtering for the syslog health monitor, gives operators a one-shot change-stream resume reset, and garbage-collects stale preflight coordination ConfigMaps. It also lands a cluster of fault-management correctness fixes covering concurrent-event handling, entity matching, cancellation scoping, stuck state labels, and a PostgreSQL change-stream race.

Major New Features

Health Event Deduplication in Platform Connector (#1283)

Implements ADR-039: a TTL-based deduplication filter that runs once inside the platform-connector pipeline, after transformers and before fan-out to the enabled connectors, so duplicate health events are suppressed before they reach either the Kubernetes connector or the store connector. Events are deduplicated by a canonical key within a configurable suppression window; healthy events clear the matching cached unhealthy key so recovery is never dropped, per-check opt-out is supported, and cached keys expire on a cleanup ticker and on restart. This reduces redundant node-condition writes and datastore churn from health monitors that re-emit the same condition on every poll.

Kernel Filtering and GPU-Reset Acknowledgement for Syslog Health Monitor (#1475)

The XID, SXID, and GPU-fallen-off-bus checks now filter journald to kernel-origin entries (SYSLOG_FACILITY=0) so that, under journald retention pressure, unrelated high-volume userspace logs no longer push the read cursor past the retained window and cause XIDs to be missed (originally #1417). The earlier attempt at this in v1.13.0 (#1426) was reverted (#1473) because the kernel-only filter also dropped the GPU reset job's own logger messages, breaking GPU reset acknowledgement. This re-lands the filter with an added journald disjunction that matches the nvsentinel-gpu-reset message tag, so reset acknowledgements are captured alongside kernel XIDs. The Kata path continues filtering by -u containerd.service.

Operator-Triggered Change-Stream Resume Reset (#1472)

Adds a runtime-owned resume-control ConfigMap that lets operators perform a one-shot resume-token reset for watcher-based components. Each component reads its own key on startup: RESUME (the default) preserves normal resume behavior, while CREATE makes the component delete only its own stored resume token and start from the current stream head, then rewrite its key back to RESUME. The ConfigMap is not Helm/GitOps-managed — components self-populate their key as RESUME when it is missing — providing a recovery lever for a component wedged on a bad resume token without disturbing the others.

Garbage Collection of Preflight Coordination ConfigMaps (#1459)

Preflight gang-coordination ConfigMaps now carry an OwnerReference back to their scheduler gang owner object, so Kubernetes garbage-collects them automatically when the owning PodGroup/workload is deleted rather than leaving stale ConfigMaps behind. Owner-reference resolution covers native workloadRef (Kubernetes 1.35), native schedulingGroup/PodGroup (Kubernetes 1.36), and generic-scheduler PodGroup, and the reference is backfilled onto existing skeleton and provisional ConfigMaps without disturbing peer data.

Bug Fixes & Reliability

• Stuck remediation-failed label on partial recovery (#1421, #1416): When a node had multiple active failures and one with an unsupported action (e.g. CONTACT_SUPPORT) set the terminal dgxc.nvidia.com/nvsentinel-state=remediation-failed label, that label could remain stuck after the unsupported check recovered while other failures kept the node quarantined — cleanup only happened on full UnQuarantined/Cancelled. Fault-quarantine now propagates a partial recovery (a healthy event clearing a tracked failure while the node stays quarantined) via the existing AlreadyQuarantined status instead of dropping it, and node-drainer marks it AlreadyDrained so it reaches fault-remediation for label cleanup. Fault-quarantine stays domain-agnostic and does not interpret remediation-action support.
• Platform connector no longer clears unrelated conditions on partial entity match (#1468, #1360): A healthy event could clear an existing unhealthy node condition if the condition message matched any single impacted entity from the healthy event. For composite identities such as NIC + NICPort, mlx5_0/port 1 and mlx5_1/port 1 are different entities that share a port number, so OR-based matching silently cleared unrelated NIC failures. Matching now requires the full composite entity rather than any single component.
• Fault-quarantine preserves concurrent health events (#1464, #1463): When two unhealthy events for the same node arrived within ~90 ms, the informer cache had not yet observed the first annotation update, so both took the "fresh quarantine" path and the second was treated as a no-op — dropping it from quarantineHealthEvent. The live node-update path now merges the incoming event with the node's current annotation using the Kubernetes state fetched for the update rather than trusting informer state, so near-concurrent events are both preserved (the second correctly classified AlreadyQuarantined).
• Node-drainer cancellation scoped by event cutoff (#1461, #1451): cancelledNodes[node] was a temporary presence flag, so while it existed node-drainer cancelled non-UnQuarantined events for that node — including a fresh quarantine that arrived before cleanup of the previous session finished. It is now a cutoff timestamp set to UnQuarantined.CreatedAt, so only events created at or before the cutoff are cancelled and a new quarantine after recovery survives.
• PostgreSQL change-stream race skipping UnQuarantined events (#1469, #1466): In the PostgreSQL changelog-based change-stream emulation, the watcher could apply a partially converted server-side SQL filter, fetch a later changelog row, advance its lastEventID bookmark, and then skip an earlier relevant row when its NOTIFY arrived later — causing node-drainer to miss an UnQuarantined event and leave a node stuck in draining. Server-side filtering now fails closed: the generated SQL filter is used only when the entire pipeline converts successfully, otherwise the watcher fetches changelog rows in order and relies on the application-side filter.

Documentation

• Writing your own preflight check tutorial (#1470): Adds a tutorial walking through building a custom NVSentinel preflight check.
• Preflight guide: OSMO+KAI, Grove, and debugging (#1467): Adds OSMO + KAI Scheduler setup, Grove limitations, and preflight-failure debugging sections to the preflight configuration guide.
• Drain plugin tutorial (#1435): Adds an end-to-end tutorial for building a new NVSentinel drain plugin, from an empty directory to a deployed controller that drains a node on NVSentinel's behalf.

Acknowledgments

This release includes contributions from:

• @lalitadithya
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.13.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.13.1

Release v1.13.1

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel --version v1.13.1

Release v1.13.0

Release v1.13.0

This release broadens DCGM deployment flexibility with three explicit source modes and per-code event suppression in the GPU health monitor, teaches fault-quarantine to coexist with externally-applied cordons and taints, adds runtime per-namespace preflight gang-discovery configuration, and implements the ExternalRemediationRequest reconciler. It also lands several impactful reliability fixes — most notably stopping node-drainer cold-start from replaying stale quarantine events and stranding remediation-failed labels.

Major New Features

DCGM Source Modes for GPU Health Monitor (#1429)

The GPU health monitor now supports three explicit DCGM source modes (see ADR-044): operator-service (the current default — the GPU Operator's nvidia-dcgm service), external-hostengine (a node-local host-installed nv-hostengine, with the DCGM-major image selected from the externally-managed nvsentinel.dgxc.nvidia.com/dcgm.version node label), and embedded (an in-process DCGM hostengine started by the monitor itself with NVIDIA runtime GPU visibility). Helm values and templates were updated for mode-specific endpoint, image, host-networking, and runtime-class selection, and the labeler preserves a valid dcgm.version label in external-hostengine mode when no DCGM pod is present.

Suppressible DCGM Error Codes (#1450)

The GPU health monitor can now be configured to suppress specific DCGM error codes via a SuppressedErrorCodes setting in the [dcgmhealthcheck] config. Non-fatal, non-actionable events (for example DCGM_FR_CLOCK_THROTTLE_POWER) can be dropped before emission rather than persisted, reducing datastore noise and avoiding unintended downstream side effects from high-frequency events.

Preserve Pre-Existing Cordons and Taints in Fault-Quarantine (#1445)

Fault-quarantine now records whether a node's cordon or taints existed before NVSentinel quarantined it, so that unquarantining no longer removes a cordon or taint that some other operator or tool applied. A new quarantineHealthEventCordonPreExisting annotation tracks pre-existing cordons, and tracked taints gain a backwards-compatible PreExisting field; cordon-by / uncordon-by labels are only added when the cordon was not pre-existing. Also fixes a bug where re-quarantining a node removed only the manual-uncordon annotation and not the manual-untaint annotation.

Namespace-Scoped Preflight Gang Discovery (#1436)

Preflight gang discovery can now be configured per namespace via a new namespaced PreflightConfig CRD (preflight.nvsentinel.nvidia.com/v1alpha1). Creating one with a spec.gangDiscovery block makes pods in that namespace use that discoverer, while all other namespaces fall back to the cluster-wide gangDiscovery Helm value. It is reconciled at runtime — no Helm upgrade or controller restart — so a single preflight deployment can serve namespaces running different gang schedulers (e.g. Volcano in one, native Kubernetes or Run:ai/OSMO in another).

ExternalRemediationRequest Reconciler (#1392)

Builds on the ExternalRemediationRequest (ExtRR) CRD foundation from v1.10.0 (#1376) by adding the janitor controller that drives the node-coordination state machine: it applies a release taint (keyed to the ExtRR's own name) plus the managed=false label and reports NVSentinelOwnershipReleased=True, then removes them and drops its finalizer once an external system reports ExternalRemediationComplete=True. Per the ADR-040 contract, ExternalRemediationComplete=False is intentionally asymmetric and does not close the request, and kubectl delete triggers the same node cleanup so operators can reclaim stalled nodes. This remains preview: nothing creates ExtRR objects automatically yet (the fault-remediation wiring is a follow-up), so today the controller only acts on hand-applied objects.

Bug Fixes & Reliability

• Node-drainer cold-start no longer replays stale quarantine events (#1443, #1347): On restart, node-drainer's cold-start logic re-queried the datastore for quarantine events still needing a drain but never checked whether the quarantine session had already ended, so it replayed stale Quarantined/AlreadyQuarantined records from sessions fault-quarantine had already resolved. It marked them drain-succeeded, and fault-remediation then stamped dgxc.nvidia.com/nvsentinel-state=remediation-failed onto already-healthy, uncordoned nodes for unsupported actions like CONTACT_SUPPORT — with no cleanup path, the label persisted forever, and a single restart could do this to many nodes at once. Cold start now verifies the quarantine session is still active before re-queuing a candidate.
• Fixed node conditions wedged by non-canonical recovery messages (#1438): platform-connectors recognized the No Health Failures recovery sentinel only via an exact, case-sensitive comparison, so any stored message that wasn't byte-identical (different casing, a trailing ;, or a value written by an external tool) was parsed as a real fault line and re-asserted Status=True. Because such a phantom line carries no entity token, the recovery path could never remove it, wedging the condition as unhealthy while LastHeartbeatTime kept advancing. The sentinel is now matched case-insensitively so recovery is recognized correctly.
• Kernel-origin syslog checks default to SYSLOG_FACILITY=0 (#1426, #1417): The XID, SXID, and GPU-fallen-off-bus checks were built with empty Tags, so they scanned every journald facility. Under journald retention pressure, unrelated high-volume userspace logs could push the read cursor behind the retained window and the journal segment holding an XID could be vacuumed before the monitor processed it — silently dropping the event. The three kernel-origin checks now default to the -k (SYSLOG_FACILITY=0) filter so they consume only kernel entries and keep up; the Kata path continues filtering by -u containerd.service.
• Skip thermal-margin monitoring on DCGM 3.x (#1448, #1449): The GpuThermalMarginWatch check introduced in v1.10.0 depends on DCGM field 153, which is unavailable on DCGM 3.x and caused the GPU health monitor to fail initialization. The check is now capability-gated on field 153 availability — DCGM 3.x logs a warning and skips the unsupported monitor while continuing standard health monitoring, and DCGM 4.x behavior is unchanged.
• Handle malformed XID 154 lines gracefully (#1440): A crafted XID 154 log line where ) precedes ( could crash the syslog health monitor's CSV parser. Such lines are now logged as a warning and skipped instead of crashing the monitor.
• Fixed TestNICCounterIBDegradation flake (#1453, #1446): The test could fail with the shared worker node "still cordoned" because the preceding TestDCGMBootstrapCompletedAnnotation (added in #1425) deletes the nvidia-dcgm pod, correctly triggering a fatal GpuDcgmConnectivityFailure that cordoned the node; the test ended before the monitor reconnected. It now waits for the gpu-health-monitor to reconnect to DCGM before finishing. Test/CI reliability only; no runtime behavior change.

Documentation

• Per-namespace Node Drainer drain modes tutorial (#1442): Adds a tutorial documenting how to configure Immediate, AllowCompletion, and DeleteAfterTimeout eviction policies per namespace via Helm, with a multi-tier application example and validation steps.
• Writing a new health monitor tutorial (#1428): Adds a developer-facing, end-to-end guide for building, testing, containerizing, and deploying a new NVSentinel health monitor.

Acknowledgments

This release includes contributions from:

• @deesharma24
• @jackyliusohu
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @natherz97
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @jackyliusohu.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.12.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.12.0

Release v1.12.0

This release prevents the DCGM connectivity error that fired on every new node during GPU Operator bootstrapping, adds device-count labels from the labeler so downstream consumers can detect nodes reporting fewer GPUs or NICs than expected, adds an opt-in Magic SysRq reboot path for the generic bare-metal provider, and includes reliability fixes for the labeler, node-drainer, and the PostgreSQL store.

Major New Features

Prevent DCGM Connectivity Errors on Node Bootstrapping (#1425, #1423)

On a freshly launched node, the gpu-health-monitor pod could become ready before the GPU Operator's nvidia-dcgm pod finished its init-container startup sequence, producing a GpuDcgmConnectivityFailure unhealthy condition (DCGM_CONNECTIVITY_ERROR, CONTACT_SUPPORT) that only cleared minutes later once DCGM came up. The gpu-health-monitor is no longer scheduled until the nvidia-dcgm pod on the node is ready, so a normal node bootstrap no longer emits a false connectivity error. Node-deletion teardown behavior is unchanged.

Expected Device-Count Labels from Labeler (#1395)

The labeler can now write normalized current and expected device-count labels (e.g. nvsentinel.dgxc.nvidia.com/gpu.count.current / .expected) onto nodes, giving downstream modules a signal for detecting nodes that advertise fewer devices than their peers. Current count is derived from a configurable CEL expression supporting both device-plugin/GFD-style node labels and DRA ResourceSlice advertisements; expected count is either learned from peers in the same grouping-label partition or pinned via per-class overrides. Configured per device class (GPU, NIC) in a TOML ConfigMap and disabled by default. See ADR-043 for the design.

Opt-In SysRq Reboot for the Generic Bare-Metal Provider (#1418)

The generic bare-metal janitor provider now supports an opt-in Linux Magic SysRq reboot mode (janitor-provider.csp.generic.useSysrqReboot=true), which reboots a node by writing b to /proc/sysrq-trigger from a privileged Job rather than using the default chroot-based reboot. This is useful on hosts where the chroot path is unreliable. The existing chroot-based reboot remains the default, so existing deployments are unaffected.

Bug Fixes & Reliability

• Lazily initialize ResourceSlice informers in the labeler (#1422): The labeler eagerly started a DRA ResourceSlice informer even when device-count detection used only the device-plugin method, spamming failed to list *v1.ResourceSlice: the server could not find the requested resource errors on clusters without the resource.k8s.io API. The informer is now initialized lazily only when a class actually requires ResourceSlice data, and string digits are normalized to numbers during count evaluation. Follow-up to the device-count feature (#1395).
• Node-drainer ignores stale AlreadyQuarantined events (#1419, #1415): A stale AlreadyQuarantined event re-enqueued via a later change-stream update — after the node had already been unquarantined and its quarantineHealthEvent annotation removed — was treated as "not already drained" and fell through to normal drain evaluation. That marked the stale event Succeeded and mutated the node-state label (triggering an invalid none -> draining transition) despite there being no active quarantine context. The already-drained check now handles a missing annotation on a stale AlreadyQuarantined event correctly instead of proceeding to drain.
• Fixed PostgreSQL UpdateDocument placeholder collision (#1391): In the direct PostgreSQL store, UpdateDocument did not bind SET parameters before WHERE parameters, so combined update+filter statements could apply parameters in the wrong order. WHERE placeholders are now shifted after the update args (regex-based, so multi-digit placeholders such as $10 are not rewritten incorrectly) and executed with update args followed by filter args.

Acknowledgments

This release includes contributions from:

• @fallintoplace
• @natherz97
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.12.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.11.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.12.0 \
  --namespace nvsentinel \
  --reuse-values