Release v1.17.0

Release v1.17.0

This release adds detection for missing InfiniBand character devices — a failure mode that left RDMA workloads unable to start while every existing check reported the node healthy — and fixes three fault-remediation event-handling bugs that could drop actionable events, replay events indefinitely, or leave a recovered node marked remediation-failed.

Major New Features

Detect Missing InfiniBand Character Devices (#1546)

Adds InfiniBandCharDeviceCheck, a passive NIC health monitor state check that flags an InfiniBand device whose /dev/infiniband character nodes (issm/umad/uverbs) are missing while its ports still read ACTIVE/LinkUp. The existing port-state checks report this as healthy, yet RDMA workloads cannot start on the node — pods fail with lstat /dev/infiniband/issm9: no such file or directory. In the motivating case a GPU node came up with only 8 of ~12 expected issm* nodes, requiring manual triage and a spare migration.

Because udev creates /dev/infiniband/{issm,umad,uverbs}N from the sysfs class entries under /sys/class/infiniband_mad and /sys/class/infiniband_verbs, a missing class entry is an exact proxy for a missing /dev node — and those directories are already within the host /sys the DaemonSet mounts, so no /dev mount is required. The check is a per-device internal-consistency test rather than an absolute expected-count test (deliberately avoiding the false-positive class addressed in v1.15.0): for each in-scope device exposing at least one InfiniBand-mode port it expects one uverbs per device plus one umad and one issm per InfiniBand-mode port, with issm/umad gated on the InfiniBand link layer since RoCE/Ethernet-mode ports do not create them.

Bug Fixes & Reliability

• Retry events skipped while an equivalent maintenance CR is in progress (#1550, #1536): When a remediation-ready health event arrived while an equivalent maintenance CR was still InProgress, fault-remediation skipped the event, advanced the resume token, and returned success — nothing reconsidered the event once the CR completed, so it was dropped permanently. In production after a reboot, new fatal events (XID 154 and an NVLink failure) arrived a few seconds before the old RebootNode CR received its completion condition; all three were skipped, faultremediated stayed null, no new CR was created, and the node stayed cordoned with an actionable fault for over two hours. Fault-remediation now requeues such events (30s by default, configurable via ReconcilerConfig.InProgressRequeueDelay) without advancing the resume token or writing status, so the event stays live until the CR reaches a terminal state and the existing decision logic resolves it. Events waiting this way are visible via a new cr_status="waiting" value on fault_remediation_events_processed_total, and an event's status is re-read before a remediation starts since a requeued event may sit in the workqueue for a long time.
• Propagate per-event MongoDB resume tokens (#1549, #1513): MongoDB change-stream events were delivered without their per-event resume tokens, so AdaptedChangeStreamWatcher.Events() passed an empty token to fault-remediation and safeMarkProcessed() skipped checkpoint persistence entirely. Fault-remediation therefore replayed the same events after every restart, and large replay backlogs could fill the controller workqueue and cause OOM CrashLoops. Each event's resume token is now captured immediately after decoding (cloning the driver-owned bson.Raw so it stays valid after the cursor advances), propagated through the store-client stack, and persisted as an embedded BSON document in the shape SetResumeAfter expects. Existing stored tokens remain compatible.
• Prevent stale replay from re-applying remediation-failed (#1544, #1534): On resuming a change stream, fault-remediation could process an old remediation-ready event after the node had already recovered; for an unsupported action such as CONTACT_SUPPORT it trusted the historical event status and stamped dgxc.nvidia.com/nvsentinel-state=remediation-failed onto the healthy node, and a later replayed UnQuarantined/Cancelled event cleared the remediation annotation without removing the label. Before applying the label, fault-remediation now reads the node through the uncached API reader and verifies the event belongs to the node's current active quarantine session — a persisted NodeQuarantined value on a replayed event is no longer sufficient. With no matching active quarantine, the label is not applied, the event is marked terminal with FaultRemediated=false, the resume token advances, and no CR is created; live-node read errors are returned so the event is retried rather than incorrectly finalized. Cancellation processing now also conditionally clears stale fault-remediation labels.

Documentation

• HealthEvent data model reference (#1535): Adds a single authoritative reference for the HealthEvent protobuf message — the fundamental data unit flowing through NVSentinel's detection and remediation pipeline — consolidating semantics that were previously scattered across 45+ documents, proto comments, and source. Covers all 18 fields with the module that consumes each, the RecommendedAction enum and its remediation routing logic (GetEffectiveActionName), the ProcessingStrategy enum and its enforcement points, and BehaviourOverrides force/skip semantics.

Acknowledgments

This release includes contributions from:

• @KaivalyaMDabhadkar
• @lalitadithya
• @shaq918
• @signalgap9-del

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @signalgap9-del.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.17.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.16.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.17.0 \
  --namespace nvsentinel \
  --reuse-values