Release v1.20.0

Release v1.20.0

This release is dominated by a large-scale memory reduction effort: informer caches across janitor, fault-quarantine, node-drainer, labeler, and preflight now strip unused fields before caching, cutting settled working set by up to 7.6× on a 99k-node benchmark. It also authenticates health-event publishers and binds each event to its reporting node, adds Lambda workload-identity authentication, exposes client-go throughput limits as Helm values, and fixes node-condition recovery after a physical GPU replacement.

Major New Features

Informer Cache Memory Reduction Across Components (#1621, #1614, #1615, #1616, #1622)

Informer caches were the dominant memory consumer at scale, because each component cached full Node and Pod objects even though rule evaluation reads only a handful of fields. Every affected component now applies SetTransform (and, where applicable, cache.ByObject scoping) so objects are stripped before they enter the cache. Benchmarked on a KWOK-dominated cluster of ~99k nodes:

• Janitor (#1621, #1605) — the single largest consumer. With GPUReset enabled and ~397k gpu-operator pods, settled working set drops from 8,581 MiB to 1,134 MiB (7.6×). Nodes are stripped to identity, labels, taints, and the NodeReady condition; the pod watch is scoped and transformed for the GPUReset path only.
• Fault-quarantine (#1614, #1607) — node.Status is discarded before caching (no rule predicate reads it), taking working set from 3,909 MiB to 864 MiB (4.5×), or 38 MB → 7.2 MB per 1k nodes.
• Labeler (#1616, #1609) — unstripped node and pod caches occupied ~7.1 GiB at 99k nodes. The node transform is feature-aware: identity, labels, and the DCGMBootstrapCompleted annotation are always kept, and status.allocatable/status.capacity are retained only when a device-count class needs them.
• Node-drainer (#1615, #1606) — nodes keep only identity plus the quarantine annotation; system-namespace and DaemonSet-owned pods (never drain targets) are reduced to bare identity stubs, while drain-eligible pods retain just the fields drain decisions need.
• Preflight (#1622, #1608) — a two-level pod transform keeps gang-relevant fields for pods in preflight-enabled namespaces and reduces all other pods to identity stubs.

Authenticated Health-Event Publishers (#1636)

Health events are submitted to the platform connector over a per-node Unix domain socket, and each event carries a nodeName naming the machine it describes. The connector previously had no way to establish which component sent an event, so it took nodeName at face value — and because health events drive cordon, drain, and reboot, a wrong node name means disruptive work landing on the wrong machine. Every publisher now mounts a short-lived projected ServiceAccount token minted for a dedicated platform-connector.nvsentinel.nvidia.com audience and rotated by the kubelet, attached as a bearer credential on each publish. A server-side interceptor validates the token via TokenReview and then decides which nodes that caller may report on. The change is contained to the health-event path — how events are stored, evaluated, and remediated once accepted is unchanged.

Lambda Workload-Identity Authentication (#1624)

Adds workload-identity authentication to the Lambda CSP health monitor and janitor provider, exchanging signed JWTs for short-lived, scoped API credentials that refresh automatically. Static API keys remain supported, and the change adds workload-identity configuration for Kubernetes deployments plus secure Lambda endpoint validation and normalization.

Configurable client-go QPS and Burst Limits (#1632, #1600)

Fault-quarantine, node-drainer, labeler, and preflight now expose client-go QPS and burst limits as Helm values, so operators can tune Kubernetes API throughput without rebuilding binaries. The chart supplies global defaults (5 QPS, 10 burst) that each component may override, with presence-based precedence preserving explicit zero values. client-go semantics are followed: zero uses the client-go defaults and a negative QPS disables client-side throttling.

Fault-Remediation Pipeline Optimization (#1627, #1592)

Cold-start replay previously retained every fully decoded health-event document in the controller workqueue, so the whole pending backlog sat in memory as generic maps even though the discovery query itself was already batched. The pipeline now queues only each event's canonical document ID and lazily fetches one full document when a worker is ready to run the existing reconciliation path, bounding workqueue memory by backlog count rather than document size.

Bug Fixes & Reliability

• Node-condition recovery after a physical GPU replacement (#1587, #1585): Replacing a GPU changes its GPU_UUID, so a healthy event from the new hardware could never match — and therefore never clear — the node condition raised against the old GPU. Recovery matching now ignores GPU_UUID when a stable GPU index or PCI address is present, while preserving UUID-only matching where no stable identity exists and leaving non-GPU matching unchanged.
• Driver-pod detection under the NVIDIADriver CRD (#1588, #1565): Driver-pod detection in the labeler and log collector broke under GPU Operator's nvidiaDriverCRD: true mode. Both now locate driver pods via the stable app.kubernetes.io/component=nvidia-driver label; legacy driver pods keep their existing path and the CRD informer excludes them to avoid duplicate events.
• Labeler reconciles device counts on allocatable changes (#1602): nodeRequiresReconciliation only checked for node label changes, but the NIC count CEL expression reads node.status.allocatable. When a MOFED driver restart temporarily zeroed allocatable, the labeler wrote nic.count.current=0 and never corrected it after recovery because no labels had changed — leaving stale counts on 20+ nodes. Reconciliation is now also triggered by status.allocatable/status.capacity changes, but only for classes whose CEL expression references node.status, so label-only classes (e.g. GPU) pay no additional cost.
• Kata label update no longer short-circuits driver/DCGM labels (#1638): A needsUpdate = needsUpdate || l.updateDriverAndDCGMLabels(...) expression meant that once needsUpdate was already true, Go's short-circuit evaluation skipped the driver/DCGM label update entirely — so a genuine driver or DCGM label diff was silently dropped in the same pass. The call is now made unconditionally, with a regression test covering both labels applying in one pass.
• Serialize per-node label reconciliation (#1646): During labeler startup, reconcileAllNodes could read that a driver pod exists and then race with its deletion — the delete handler correctly removed driver.installed, but the startup sweep subsequently restored driver.installed=true from its stale earlier read, and because that output label is not a watched reconciliation input it could stay wrong indefinitely absent a replacement pod event. Reconciliation for a given node is now serialized across the startup sweep and the pod add/update/delete handlers using striped locks, so whichever path runs second recalculates from the latest informer state; different nodes still reconcile concurrently.
• XID 45 treated as non-fatal (#1629): XID 45 was classified as fatal — draining and rebooting the node — because MapActionStringToProto had no case for the WORKFLOW_XID_45 guidance class and fell through to CONTACT_SUPPORT, which determineFatality treats as fatal. The catalog asks for no node-level action for XID 45 (RESTART_APP/REPORT_ISSUE solo, IGNORE when not solo), and since both RESTART_APP and IGNORE already map to NONE, the bucket resolves correctly without needing solo/non-solo correlation.
• Thermal clocks-event mapping aligned to NONE (#1637): DCGM_FR_CLOCKS_EVENT_THERMAL (the DCGM 4.x name) and DCGM_FR_CLOCK_THROTTLE_THERMAL (the 3.x name) are the same underlying error code, but the two dcgmerrorsmapping.csv rows disagreed — one mapped to CONTACT_SUPPORT, the other to NONE. Since the GPU health monitor resolves the integer code to a single name, the rows are now aligned on NONE.
• Code-block-aware MDX sanitization in Fern docs CI (#1635, #1634): Blind sed escaping of {, }, and < during the frozen-version checkout corrupted content inside fenced code blocks on the published docs site (visible as \{/\} in rendered protobuf and Go samples). Sanitization now uses awk that tracks fence state and skips inline code spans, escaping only in prose where MDX parsing actually applies.

Documentation

• Simplified README (#1628): Adds adoption stages for NVSentinel and removes redundant content from the README, redirecting readers to the documentation site.

Acknowledgments

This release includes contributions from:

• @alexander-demicev
• @cbumb
• @deesharma24
• @giranntu
• @iacker
• @KaivalyaMDabhadkar
• @lalitadithya
• @natherz97
• @pdmack
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @giranntu and @iacker.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.20.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.19.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.20.0 \
  --namespace nvsentinel \
  --reuse-values