Release v1.14.0

Release v1.14.0

This release adds TTL-based health-event deduplication in the platform connector, re-lands GPU-reset-safe kernel filtering for the syslog health monitor, gives operators a one-shot change-stream resume reset, and garbage-collects stale preflight coordination ConfigMaps. It also lands a cluster of fault-management correctness fixes covering concurrent-event handling, entity matching, cancellation scoping, stuck state labels, and a PostgreSQL change-stream race.

Major New Features

Health Event Deduplication in Platform Connector (#1283)

Implements ADR-039: a TTL-based deduplication filter that runs once inside the platform-connector pipeline, after transformers and before fan-out to the enabled connectors, so duplicate health events are suppressed before they reach either the Kubernetes connector or the store connector. Events are deduplicated by a canonical key within a configurable suppression window; healthy events clear the matching cached unhealthy key so recovery is never dropped, per-check opt-out is supported, and cached keys expire on a cleanup ticker and on restart. This reduces redundant node-condition writes and datastore churn from health monitors that re-emit the same condition on every poll.

Kernel Filtering and GPU-Reset Acknowledgement for Syslog Health Monitor (#1475)

The XID, SXID, and GPU-fallen-off-bus checks now filter journald to kernel-origin entries (SYSLOG_FACILITY=0) so that, under journald retention pressure, unrelated high-volume userspace logs no longer push the read cursor past the retained window and cause XIDs to be missed (originally #1417). The earlier attempt at this in v1.13.0 (#1426) was reverted (#1473) because the kernel-only filter also dropped the GPU reset job's own logger messages, breaking GPU reset acknowledgement. This re-lands the filter with an added journald disjunction that matches the nvsentinel-gpu-reset message tag, so reset acknowledgements are captured alongside kernel XIDs. The Kata path continues filtering by -u containerd.service.

Operator-Triggered Change-Stream Resume Reset (#1472)

Adds a runtime-owned resume-control ConfigMap that lets operators perform a one-shot resume-token reset for watcher-based components. Each component reads its own key on startup: RESUME (the default) preserves normal resume behavior, while CREATE makes the component delete only its own stored resume token and start from the current stream head, then rewrite its key back to RESUME. The ConfigMap is not Helm/GitOps-managed — components self-populate their key as RESUME when it is missing — providing a recovery lever for a component wedged on a bad resume token without disturbing the others.

Garbage Collection of Preflight Coordination ConfigMaps (#1459)

Preflight gang-coordination ConfigMaps now carry an OwnerReference back to their scheduler gang owner object, so Kubernetes garbage-collects them automatically when the owning PodGroup/workload is deleted rather than leaving stale ConfigMaps behind. Owner-reference resolution covers native workloadRef (Kubernetes 1.35), native schedulingGroup/PodGroup (Kubernetes 1.36), and generic-scheduler PodGroup, and the reference is backfilled onto existing skeleton and provisional ConfigMaps without disturbing peer data.

Bug Fixes & Reliability

• Stuck remediation-failed label on partial recovery (#1421, #1416): When a node had multiple active failures and one with an unsupported action (e.g. CONTACT_SUPPORT) set the terminal dgxc.nvidia.com/nvsentinel-state=remediation-failed label, that label could remain stuck after the unsupported check recovered while other failures kept the node quarantined — cleanup only happened on full UnQuarantined/Cancelled. Fault-quarantine now propagates a partial recovery (a healthy event clearing a tracked failure while the node stays quarantined) via the existing AlreadyQuarantined status instead of dropping it, and node-drainer marks it AlreadyDrained so it reaches fault-remediation for label cleanup. Fault-quarantine stays domain-agnostic and does not interpret remediation-action support.
• Platform connector no longer clears unrelated conditions on partial entity match (#1468, #1360): A healthy event could clear an existing unhealthy node condition if the condition message matched any single impacted entity from the healthy event. For composite identities such as NIC + NICPort, mlx5_0/port 1 and mlx5_1/port 1 are different entities that share a port number, so OR-based matching silently cleared unrelated NIC failures. Matching now requires the full composite entity rather than any single component.
• Fault-quarantine preserves concurrent health events (#1464, #1463): When two unhealthy events for the same node arrived within ~90 ms, the informer cache had not yet observed the first annotation update, so both took the "fresh quarantine" path and the second was treated as a no-op — dropping it from quarantineHealthEvent. The live node-update path now merges the incoming event with the node's current annotation using the Kubernetes state fetched for the update rather than trusting informer state, so near-concurrent events are both preserved (the second correctly classified AlreadyQuarantined).
• Node-drainer cancellation scoped by event cutoff (#1461, #1451): cancelledNodes[node] was a temporary presence flag, so while it existed node-drainer cancelled non-UnQuarantined events for that node — including a fresh quarantine that arrived before cleanup of the previous session finished. It is now a cutoff timestamp set to UnQuarantined.CreatedAt, so only events created at or before the cutoff are cancelled and a new quarantine after recovery survives.
• PostgreSQL change-stream race skipping UnQuarantined events (#1469, #1466): In the PostgreSQL changelog-based change-stream emulation, the watcher could apply a partially converted server-side SQL filter, fetch a later changelog row, advance its lastEventID bookmark, and then skip an earlier relevant row when its NOTIFY arrived later — causing node-drainer to miss an UnQuarantined event and leave a node stuck in draining. Server-side filtering now fails closed: the generated SQL filter is used only when the entire pipeline converts successfully, otherwise the watcher fetches changelog rows in order and relies on the application-side filter.

Documentation

• Writing your own preflight check tutorial (#1470): Adds a tutorial walking through building a custom NVSentinel preflight check.
• Preflight guide: OSMO+KAI, Grove, and debugging (#1467): Adds OSMO + KAI Scheduler setup, Grove limitations, and preflight-failure debugging sections to the preflight configuration guide.
• Drain plugin tutorial (#1435): Adds an end-to-end tutorial for building a new NVSentinel drain plugin, from an empty directory to a deployed controller that drains a node on NVSentinel's behalf.

Acknowledgments

This release includes contributions from:

• @lalitadithya
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.13.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --reuse-values