Repository navigation
Release v1.26.0
Release v1.26.0
This release makes the deployment platform connector (ADR-052) deployable as an opt-in alternative to the node-local socket, runs NVCRE tests as part of new-node and post-remediation validation, adds a standing bound on how many nodes NVSentinel keeps cordoned, and reduces the health-events-analyzer's per-event query cost. GPU Operator GPUCluster (DRA) mode support is added to metadata-collector. The MongoDB setup Jobs change on upgrade — please read the Upgrade Notes below.
Upgrade Notes
MongoDB setup Jobs create the idempotency index; the external setup Job is no longer a Helm hook (#1927)
Both the bundled and the external MongoDB setup Jobs now create the unique idempotency index (healthevent_idempotency_key_unique), a partial index over events that carry an idempotency key. If existing documents already share a key, the Job prints an error with a query that lists them and completes the rest of the setup.
The external MongoDB setup Job (global.datastore.setupJob) is now a plain Job rather than a post-install,post-upgrade Helm hook, matching the bundled Job. A hook runs only once the release's pods are ready, while the deployment platform connector's replicas become ready only once this index exists, so the two would wait on each other. If you run external MongoDB with ArgoCD, the Job now syncs as a regular resource rather than at PostSync.
Major New Features
Deployment Platform Connector (#1927, #1934)
The platform connector can now run as a central Deployment that health monitors reach over the network with TLS and a projected ServiceAccount token, as an opt-in alternative to the node-local DaemonSet socket (ADR-052). The connector role, the publisher clients, and the datastore index landed in earlier releases; this release adds the Helm chart and documentation that make it deployable.
platformConnector.deployment.enabledrenders the Deployment on the system pool with a PodDisruptionBudget and anti-affinity, its Service, a cert-manager CA and leaf certificate, its ServiceAccount and RBAC, and a NetworkPolicy that admits only labelled health publishers.- Every publisher — gpu-, syslog-, nic-, kubernetes-object-, slurm-drain-, csp- and nvcre-certification monitors, the health-events-analyzer, lifecycle-manager, and preflight — gets a
publishToswitch:socket(the default) ordeployment. - The render refuses pairings that would lose events: a publisher on
deploymentwhile the Deployment is off, or onsocketwhile the DaemonSet is off. Bothenabledkeys must be real YAML booleans; a quoted"false"is refused.
Nothing changes for an existing install: the Deployment is off, every publishTo is socket, and a --reuse-values upgrade keeps today's behaviour. Node condition writes on the deployment connector are retried in place until the best-effort deadline (#1934), so faults for one node that reach different replicas at once no longer lose the losing writes.
NVCRE Tests in New-Node and Post-Remediation Validation (#1922)
Adds an nvcre-provider test provider that runs NVIDIA Cluster Readiness Engine tests — nccl-loopback-nvswitch and nccl-all-reduce — as part of new-node and post-remediation validation, alongside the existing k8s-job-provider and its dcgm-diag-test. It is opt-in. Install NVCRE, enable lifecycle-manager and post-remediation validation in fault-quarantine (both off by default), and apply the overrides in values-nvcre.yaml.
Validation-State Node Label (#1947)
Nodes targeted by a ValidationRequest now carry a nvsentinel.nvidia.com/validation-state label, derived from the existing active-validation-request and validation-session annotations:
| Label value | Meaning |
|---|---|
| (absent) | The node has no validation-session entries. |
validation-pending |
At least one entry is waiting, no request is running, and not every entry has failed. |
validating |
A request is running on the node. |
validation-failed |
No request is running and every entry has failed. |
The validation-controller and node-validation-controller also log every state change they make.
Standing Bound on Cordoned Nodes (#1926, #1924)
Every existing circuit-breaker bound counts cordon events inside a window, so none of them limits how many nodes end up cordoned in total — a rate that stays under the threshold can cordon a whole fleet given enough windows. The new circuitBreaker.maxCordonedNodes is a level rather than a rate: the breaker trips when that many nodes are quarantined by NVSentinel at once. It counts only NVSentinel's own cordons, so GPU Operator upgrade flows do not consume the budget, and it latches like the existing bounds. It is off by default (0), and it satisfies the requirement that at least one bound is set.
Health-Events-Analyzer Rule Evaluation (#1913, #1912)
The analyzer evaluates 22 rules per health event, each one an aggregation query sent to the datastore one at a time. Two changes reduce that cost:
- Rules can be skipped when they cannot match (#1913): each rule takes an optional
whenCEL expression over the incoming event, and the analyzer runs the rule's query only when it is true. Most rules apply to one XID or one check, yet MongoDB previously read the node's events in the rule's window before the rule's own gate discarded them. 21 of the 22 default rules now carry awhenthat copies their existing gate;MultipleRemediationshas no such gate and still runs for every event. - Opt-in concurrent rule evaluation (#1912):
--rule-concurrencysets how many rule queries run at once for a single event. It complements--workers, which cannot help when the backlog sits on one node because a node's events stay in order. The default is1, which keeps the current behaviour.
Custom Drain Scoped by Node Label Selector (#1871, #1857)
customDrain.enabled=true previously routed every node through the drain plugin, and validation rejected userNamespaces or podDrainPolicies alongside it — so a cluster running a Slurm partition next to ordinary Kubernetes workloads could not express both. The new customDrain.nodeSelector takes a standard label selector: matching nodes use the drain plugin, and all other nodes keep the built-in eviction path. The mutual-exclusion checks apply only when the selector is unset, and an empty selector keeps today's behaviour.
[customDrain]
enabled = true
nodeSelector = "scheduler=slurm"
[[userNamespaces]]
name = "*"
mode = "AllowCompletion"Configurable Kubelet Root Directory (#1951, #1936)
Kubelet creates its PodResources socket under --root-dir, but the chart mounted a fixed /var/lib/kubelet/pod-resources. On distributions that relocate kubelet, the mount was empty and metadata-collector crash-looped on every GPU node. The new global.kubeletRootDir (default /var/lib/kubelet) sets the host side of that mount. The value must be an absolute path, and a value ending in /pod-resources is refused at render time.
GPU Operator GPUCluster (DRA) Mode Support (#1923, #1930)
This release adds GPU Operator GPUCluster (DRA) mode support to metadata-collector:
- metadata-collector runs in GPUCluster mode (#1923): GPUCluster mode ships no Container Toolkit and no
nvidiaRuntimeClass, so the DaemonSet was rejected at admission. Withglobal.gpuOperator.mode: gpucluster, the chart dropsruntimeClassNameand requests GPU access through a DRAResourceClaimTemplate. - DRA GPU pods are mapped for partial drain (#1930, #1865): the pod-to-GPU mapper read only device-plugin allocations, so no GPU pod on a GPUCluster node received its device annotation and every partial drain degraded to a full node drain. The mapper now also reads DRA allocations and resolves device names to GPU UUIDs locally through NVML.
Bug Fixes & Reliability
- GPU conditions written by older releases now clear on recovery (#1969, #1940): GPU node conditions written by older releases name the GPU only as
GPU:<n>, while current recovery events carryGPU,PCI, andGPU_UUID. Matching still required thePCItoken, so after an upgrade those conditions stayedTruewith no way to clear them except a manual status patch. A GPU-only compatibility pass now clears such an entry when the recovered GPU index and every other non-PCI entity match exactly. - metadata-collector supports GPU Operator NRI plugin mode (#1949, #1948): with the NRI plugin enabled there is no
nvidiaRuntimeClass, so the default configuration failed admission and the host-path approach from v1.22.0 does not fit an operator-managed driver container. Settingmetadata-collector.nriPlugin.enabled: truerequests the GPU management CDI device through a pod annotation, the method GPU Operator documents for management containers. The NVSentinel namespace must be in GPU Operator'sNRI_MANAGEMENT_CDI_DEVICE_NAMESPACES; outside it, nothing is injected and the collector fails withERROR_LIBRARY_NOT_FOUND. - NIC topology parses on single-GPU nodes (#1956, #1954): the
nvidia-smi topo -mheader parser required two GPU columns, so single-GPU nodes with RDMA NICs published an empty NIC topology and no GPU NUMA node, and nic-health-monitor refused to start on them. The parser now accepts one GPU column. - Service hostnames follow
global.clusterDomain(#1942, #1941): the chart built in-cluster hostnames and certificate DNS names with a hardcodedsvc.cluster.local, so on clusters with another DNS domain the MongoDB URI, the deployment platform connector address, and those certificate names did not resolve or match. The newglobal.clusterDomain(defaultcluster.local) is used wherever the chart builds such a name. - gpu-health-monitor metrics bind address is configurable for IPv6-only clusters (#1487, #1407): the metrics server always bound to
0.0.0.0, so in IPv6-only clusters kubelet probes over IPv6 were refused and the pod crash-looped. The newglobal.metricsAddressHelm value sets the bind address. It defaults to0.0.0.0, so nothing changes unless you set it. IPv6-only clusters can set it to::, which binds dual-stack on standard Linux (net.ipv6.bindv6only=0). The DCGM client's connectivity over IPv6 hostnames is tracked upstream in NVIDIA/DCGM#301. - Change-gated counters report zero instead of being absent (#1931, #1929, #1933, #1932): a Prometheus
CounterVecwith no children exports nothing, so four platform-connector counters (dropped writes and batches, node condition updates, node event operations) and two nic-health-monitor counters were absent from/metricsuntil their first occurrence. An empty query could not distinguish "nothing happened" from "the component is not reporting". On one 287-node cluster, the nic-health-monitor counters were absent for 2.5 hours across 282 pods after a rollout. These counters are now pre-initialised at zero.
Documentation
- MaintenanceRequest and ExternalRemediationRequest tutorials (#1928, #1938): adds guides for requesting node maintenance with a MaintenanceRequest and for handing nodes to an external system with an ExternalRemediationRequest. The maintenance guide includes a step for clusters upgraded from v1.24.0 or earlier: Helm installs CRDs only on first install, so the
MaintenanceRequestandValidationRequestCRDs must be applied from the chart before the feature can be used.
Acknowledgments
This release includes contributions from:
- @abonillabeeche
- @avgvi
- @cbumb
- @changjjjjjjj
- @KaivalyaMDabhadkar
- @lalitadithya
- @lfriedman-netllama
- @mvanhorn
- @natherz97
- @ramessesii2
- @simoninr08
- @tanishagoyal2
Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @abonillabeeche, @avgvi, @changjjjjjjj, @mvanhorn, @ramessesii2, and @simoninr08.
Container Images
See versions.txt for the full list of container images and versions.
Helm Chart
Install with:
helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
--version v1.26.0 \
--namespace nvsentinel \
--create-namespaceTo upgrade from v1.25.0 — review the Upgrade Notes above first:
helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
--version v1.26.0 \
--namespace nvsentinel \
--reuse-values