Release v1.18.0

Release v1.18.0

This release adds Lambda Cloud as a supported CSP — both a csp-health-monitor provider and a janitor plugin — along with cross-CSP maintenance-event improvements that also close a long-standing gap for GCP. It also gates node-labeler detection labels on ExtRR ownership, fixes syslog-health-monitor silently missing XIDs emitted between boot and monitor startup, and documents the Percona MongoDB backend for ARM64.

Major New Features

Lambda Cloud Support (#1545, #1569)

Adds Lambda as a supported cloud provider across both detection and remediation:

• CSP health monitor (#1545): Chart wiring mirroring the existing [gcp]/[aws] providers, a paginated GET /api/v1/maintenance_events poller, an event normalizer, and a shared HTTP client with exponential-backoff retry under commons/pkg/lambda/ for reuse by future Lambda-facing components (there is currently no SDK for the Lambda APIs).
• Janitor plugin (#1569): A Lambda janitor provider supporting instance retrieval, power cycling, termination, action validation, and asynchronous readiness tracking, built on the same commons client. Includes authenticated JSON requests with safe retry handling and HTTPS redirect protection, plus Kubernetes configuration for Lambda endpoints and Secret-based API keys.

The health monitor PR also adds two cross-CSP trigger-engine paths driven off the internal event model, available to any CSP whose normalizer emits the right fields:

• Emergency-urgency quarantine bypass — DETECTED events with metadata.urgency=EMERGENCY fire QUARANTINE on the next poll with no scheduledStartTime window check.
• CANCELLED → HEALTHY dispatch — HEALTHY now fires as soon as an event is observed in CANCELLED. This closes the same gap for GCP, whose normalizer already produces StatusCancelled: previously, cancelling a maintenance event after it had been quarantined left nvidia.com/maintenance on the node forever because no HEALTHY was ever emitted. CANCELLED fires HEALTHY immediately (no maintenance actually happened), while MAINTENANCE_COMPLETE continues to wait for actualEndTime + postMaintenanceHealthyDelayMinutes.

Gate Node-Labeler Detection Labels on ExtRR Ownership (#1551)

Continues the ExternalRemediationRequest work (ADR-040) by gating the labeler and fault-quarantine on nvsentinel.dgxc.nvidia.com/managed=false, so a node under active external ownership stops having detection labels stamped — which would otherwise keep DaemonSet health monitors scheduled on it. The labeler strips the DCGMVersion, DriverInstalled, and KataEnabled detection labels when managed=false (monitors then self-evict via their existing nodeSelector, requiring no per-monitor changes) and skips pod-driven re-stamping for opted-out nodes. Both paths fail closed on lister error, so no mutations occur when the managed-label state is unknown, preventing accidental re-enablement of monitors on a released node. All six default fault-quarantine rulesets now include a CEL expression that skips opening a new quarantine session on such nodes, and a new labeler_node_labels_skipped_managed_total counter tracks the behavior. The existing k8saas.nvidia.com/ManagedByNVSentinel check is retained for backwards compatibility.

Bug Fixes & Reliability

• Syslog-health-monitor missed XIDs emitted before startup after a reboot (#1533, #1532): After a host reboot the monitor detects the boot-ID change and clears its persisted cursors; on the next poll the absent cursor triggered initializeJournalFromTail, which seeks to the journal's end — permanently skipping any XID emitted between boot and monitor startup. A postRebootInit flag now marks this case so the monitor instead seeks to the beginning of the current boot's journal (SeekHead() with an explicit _BOOT_ID=<current> match filter) and processes all entries forward, clearing the flag once all checks complete so subsequent polls resume normal cursor-based processing.
• Metadata-collector no longer exits on a deleted pod (#1578, #1568): UpdatePodDevicesAnnotations treated a NotFound response while patching pod device annotations as fatal, so an ordinary deletion race caused the collector to exit 1. NotFound is now treated as a benign deletion race: reconciliation continues over the remaining pods, deleted pods are not counted as updated, and other update errors are still reported.
• E2E flake from kwok host networking (#1570, #1567): Installing kwok with hostNetwork enabled made the kwok pod run on the node's IP and stamp that IP onto all 55 fake nodes; kindnetd, which identifies "is this node me" by IP, then repeatedly rewrote the real node's CNI config with fake pod CIDRs. Pods created afterwards received IPs already held by kwok's fake pods, and since kind v0.30 enforces NetworkPolicies, the policy engine resolved the shared IP to the wrong pod and silently dropped janitor → janitor-provider gRPC traffic — timing out for 10–25 minutes and stalling RebootNode CRs until tests ran out of time. kwok is now installed without hostNetwork (with both kwok charts pinned to 0.3.0). Test/CI reliability only; no runtime behavior change.

Documentation

• Percona MongoDB backend for ARM64 and Bitnami → Percona migration (#1566, #1497): The default MongoDB backend (Bitnami) ships amd64-only images, so the default NVSentinel configuration cannot run on ARM64 nodes; the Percona operator backend is multi-arch and does. Adds an ARM64 support section to the MongoDB store configuration docs (including the exact image-pull error users hit) and a full migration runbook covering the objects that survive helm uninstall and must be cleaned up (PVCs, the credentials secret, all cert-manager TLS secrets including per-replica server certs, and the runtime-created resume-control and circuit-breaker ConfigMaps), handling of in-flight quarantines and remediations, cloud volume size minimums, verification steps, troubleshooting, and rollback guidance. The chart default is intentionally unchanged.

Acknowledgments

This release includes contributions from:

• @aditighag
• @alexander-demicev
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @signalgap9-del
• @sravindra05

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @aditighag, @alexander-demicev, and @sravindra05.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.18.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.17.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.18.0 \
  --namespace nvsentinel \
  --reuse-values