Release v1.13.0

Release v1.13.0

This release broadens DCGM deployment flexibility with three explicit source modes and per-code event suppression in the GPU health monitor, teaches fault-quarantine to coexist with externally-applied cordons and taints, adds runtime per-namespace preflight gang-discovery configuration, and implements the ExternalRemediationRequest reconciler. It also lands several impactful reliability fixes — most notably stopping node-drainer cold-start from replaying stale quarantine events and stranding remediation-failed labels.

Major New Features

DCGM Source Modes for GPU Health Monitor (#1429)

The GPU health monitor now supports three explicit DCGM source modes (see ADR-044): operator-service (the current default — the GPU Operator's nvidia-dcgm service), external-hostengine (a node-local host-installed nv-hostengine, with the DCGM-major image selected from the externally-managed nvsentinel.dgxc.nvidia.com/dcgm.version node label), and embedded (an in-process DCGM hostengine started by the monitor itself with NVIDIA runtime GPU visibility). Helm values and templates were updated for mode-specific endpoint, image, host-networking, and runtime-class selection, and the labeler preserves a valid dcgm.version label in external-hostengine mode when no DCGM pod is present.

Suppressible DCGM Error Codes (#1450)

The GPU health monitor can now be configured to suppress specific DCGM error codes via a SuppressedErrorCodes setting in the [dcgmhealthcheck] config. Non-fatal, non-actionable events (for example DCGM_FR_CLOCK_THROTTLE_POWER) can be dropped before emission rather than persisted, reducing datastore noise and avoiding unintended downstream side effects from high-frequency events.

Preserve Pre-Existing Cordons and Taints in Fault-Quarantine (#1445)

Fault-quarantine now records whether a node's cordon or taints existed before NVSentinel quarantined it, so that unquarantining no longer removes a cordon or taint that some other operator or tool applied. A new quarantineHealthEventCordonPreExisting annotation tracks pre-existing cordons, and tracked taints gain a backwards-compatible PreExisting field; cordon-by / uncordon-by labels are only added when the cordon was not pre-existing. Also fixes a bug where re-quarantining a node removed only the manual-uncordon annotation and not the manual-untaint annotation.

Namespace-Scoped Preflight Gang Discovery (#1436)

Preflight gang discovery can now be configured per namespace via a new namespaced PreflightConfig CRD (preflight.nvsentinel.nvidia.com/v1alpha1). Creating one with a spec.gangDiscovery block makes pods in that namespace use that discoverer, while all other namespaces fall back to the cluster-wide gangDiscovery Helm value. It is reconciled at runtime — no Helm upgrade or controller restart — so a single preflight deployment can serve namespaces running different gang schedulers (e.g. Volcano in one, native Kubernetes or Run:ai/OSMO in another).

ExternalRemediationRequest Reconciler (#1392)

Builds on the ExternalRemediationRequest (ExtRR) CRD foundation from v1.10.0 (#1376) by adding the janitor controller that drives the node-coordination state machine: it applies a release taint (keyed to the ExtRR's own name) plus the managed=false label and reports NVSentinelOwnershipReleased=True, then removes them and drops its finalizer once an external system reports ExternalRemediationComplete=True. Per the ADR-040 contract, ExternalRemediationComplete=False is intentionally asymmetric and does not close the request, and kubectl delete triggers the same node cleanup so operators can reclaim stalled nodes. This remains preview: nothing creates ExtRR objects automatically yet (the fault-remediation wiring is a follow-up), so today the controller only acts on hand-applied objects.

Bug Fixes & Reliability

• Node-drainer cold-start no longer replays stale quarantine events (#1443, #1347): On restart, node-drainer's cold-start logic re-queried the datastore for quarantine events still needing a drain but never checked whether the quarantine session had already ended, so it replayed stale Quarantined/AlreadyQuarantined records from sessions fault-quarantine had already resolved. It marked them drain-succeeded, and fault-remediation then stamped dgxc.nvidia.com/nvsentinel-state=remediation-failed onto already-healthy, uncordoned nodes for unsupported actions like CONTACT_SUPPORT — with no cleanup path, the label persisted forever, and a single restart could do this to many nodes at once. Cold start now verifies the quarantine session is still active before re-queuing a candidate.
• Fixed node conditions wedged by non-canonical recovery messages (#1438): platform-connectors recognized the No Health Failures recovery sentinel only via an exact, case-sensitive comparison, so any stored message that wasn't byte-identical (different casing, a trailing ;, or a value written by an external tool) was parsed as a real fault line and re-asserted Status=True. Because such a phantom line carries no entity token, the recovery path could never remove it, wedging the condition as unhealthy while LastHeartbeatTime kept advancing. The sentinel is now matched case-insensitively so recovery is recognized correctly.
• Kernel-origin syslog checks default to SYSLOG_FACILITY=0 (#1426, #1417): The XID, SXID, and GPU-fallen-off-bus checks were built with empty Tags, so they scanned every journald facility. Under journald retention pressure, unrelated high-volume userspace logs could push the read cursor behind the retained window and the journal segment holding an XID could be vacuumed before the monitor processed it — silently dropping the event. The three kernel-origin checks now default to the -k (SYSLOG_FACILITY=0) filter so they consume only kernel entries and keep up; the Kata path continues filtering by -u containerd.service.
• Skip thermal-margin monitoring on DCGM 3.x (#1448, #1449): The GpuThermalMarginWatch check introduced in v1.10.0 depends on DCGM field 153, which is unavailable on DCGM 3.x and caused the GPU health monitor to fail initialization. The check is now capability-gated on field 153 availability — DCGM 3.x logs a warning and skips the unsupported monitor while continuing standard health monitoring, and DCGM 4.x behavior is unchanged.
• Handle malformed XID 154 lines gracefully (#1440): A crafted XID 154 log line where ) precedes ( could crash the syslog health monitor's CSV parser. Such lines are now logged as a warning and skipped instead of crashing the monitor.
• Fixed TestNICCounterIBDegradation flake (#1453, #1446): The test could fail with the shared worker node "still cordoned" because the preceding TestDCGMBootstrapCompletedAnnotation (added in #1425) deletes the nvidia-dcgm pod, correctly triggering a fatal GpuDcgmConnectivityFailure that cordoned the node; the test ended before the monitor reconnected. It now waits for the gpu-health-monitor to reconnect to DCGM before finishing. Test/CI reliability only; no runtime behavior change.

Documentation

• Per-namespace Node Drainer drain modes tutorial (#1442): Adds a tutorial documenting how to configure Immediate, AllowCompletion, and DeleteAfterTimeout eviction policies per namespace via Helm, with a multi-tier application example and validation steps.
• Writing a new health monitor tutorial (#1428): Adds a developer-facing, end-to-end guide for building, testing, containerizing, and deploying a new NVSentinel health monitor.

Acknowledgments

This release includes contributions from:

• @deesharma24
• @jackyliusohu
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @natherz97
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @jackyliusohu.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.12.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --reuse-values