Release v1.15.0

Release v1.15.0

This release completes the operator-triggered cold-start/change-stream reset begun in v1.14.0, adds kubectl get extrr printer columns, and delivers a set of high-impact reliability fixes — most importantly detecting frozen health-monitor loops via the liveness probe, hardening GPU reset job creation and partial drains, and correcting NIC monitor false positives and dropped failure events. It also expands operator documentation across the PostgreSQL/MongoDB stores, fault-quarantine rule sets, and the Kubernetes Object Monitor.

Major New Features

Cold-Start Reset via resume-control ConfigMap (#1476)

Completes the operator-triggered change-stream reset introduced in v1.14.0 (#1472). Previously, setting a module's key to CREATE in the resume-control ConfigMap only reset the change-stream resume token — a subsequent restart could still replay old persisted records because cold-start recovery queries the datastore directly. Modules that support cold-start recovery now record a component-managed <module>.coldStartAfter timestamp when they consume CREATE, and future cold-start queries only consider events created after that timestamp, making the reset durable. Operators still only set the module key to CREATE; the timestamp is internal status managed by the module.

ExternalRemediationRequest Printer Columns (#1495)

kubectl get extrr now shows NODE, CHECKNAME, and COMPLETIONTIME columns (with completion time rendered in a readable date format), so operators can see the state of ExternalRemediationRequest objects at a glance without inspecting each one.

Bug Fixes & Reliability

• Detect frozen health-monitor loops via /healthz (#1477): The gpu-, syslog-, and nic-health-monitor liveness probes targeted /metrics, served by an independent HTTP goroutine, so when a monitor's main polling loop froze the probe kept returning 200 and Kubernetes never restarted the pod. On one 370-GPU-node cluster, 13 gpu-health-monitor pods showed Running 1/1 with zero restarts while their reconciliation loops had been frozen for 7–24 days — those nodes had zero GPU health monitoring despite appearing healthy. A new reusable PollingHealthChecker tracks the last successful poll timestamp and makes /healthz return unhealthy once it exceeds a staleness threshold (3× the poll interval), so a stalled loop now trips the liveness probe and restarts the pod.
• GPUReset job creation failures and invalid partial drains (#1500): Two fixes for GPU reset. (1) Depending on the node name, the reset-job name truncation could produce an invalid RFC 1123 subdomain (e.g. a trailing .- sequence), so the controller failed to create the job indefinitely — and because this phase has no timeout, the CRD never reached a terminal state. Job names are now always valid RFC 1123 subdomains. (2) Full and partial drains skipped pods that were stuck terminating or NotReady, which may still hold running processes with GPU contexts and can cause a GPU reset to fail; partial drains now ensure targeted pods are removed or in a terminal state before the reset proceeds.
• NIC monitor no longer misses or drops failure events (#1492, #1488): Fixes six findings where the NIC health monitor could silently lose or never emit failure events or fabricate fatal disappearance events from transient sysfs read errors. Notably, fatal counter events were relabeled under the state check's name, so an ordinary ACTIVE/LinkUp recovery cleared a counter-originated condition while the breach latch kept suppressing further breaches — the link_downed safety net worked only once per boot; fatal counter events now keep their *DegradationCheck identity so a state recovery can no longer clear them. FATAL/recovery events were also persisted before publish (violating the healthpub caller contract) so a platform-connector outage permanently consumed the health boundary; checks are now transactional (Prepare/Commit).
• NIC monitor requires positive evidence for FATALs; adds inclusion override (#1462, #1361, #1379): A port the monitor had never seen healthy was reported FATAL unless proven expected-down, producing false alarms on ports that are down by design (e.g. unprovisioned Ethernet/RoCE aux ports on some cloud shapes). The rule is inverted: a first-sight DOWN port is now failed only when there is positive peer evidence it should be up (grouping cards by role and flagging those below the untied modal active-port count), otherwise it is suppressed; runtime ACTIVE → DOWN remains always fatal. Adds a nicInclusionRegexOverride to pin monitoring to an explicit NIC list, and fixes restart/config-change state bugs (fabricated "device disappeared" FATALs, lost counter latches, and card-level FATALs that could hold a node quarantined forever with no recovery event).
• Circuit breaker counts only GPU nodes (#1482, #1228): The fault-quarantine circuit breaker computed its thresholds, trip, and progress against all cluster nodes rather than GPU nodes, skewing the denominator on mixed clusters. It now counts only nodes labeled nvidia.com/gpu.present=true, with the label configurable via fault-quarantine.gpuNodeLabel (Helm) / --gpu-node-label (CLI).

Documentation

• PostgreSQL Store configuration guide (#1490): Adds a configuration guide for the PostgreSQL datastore.
• Percona Operator for MongoDB store (#1489): Adds Percona Operator documentation to the MongoDB store guide.
• Managing fault-quarantine rule sets (#1486): Documents how to add, modify, disable, apply, and verify fault-quarantine rule sets.
• Using the Kubernetes Object Monitor (#1483): Adds a guide for configuring and using the Kubernetes Object Monitor.
• Custom remediation plugin tutorial (#1493): Adds a tutorial for writing a custom remediation plugin.

Acknowledgments

This release includes contributions from:

• @jtschelling
• @KaivalyaMDabhadkar
• @natherz97
• @nitz2407
• @rupalis-nv
• @shaq918
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.14.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --reuse-values