Skip to content

v1.1.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 19 Oct 20:44
Immutable release. Only release title and notes can be modified.
v1.1.0
This tag was signed with the committer’s verified signature.
mchmarny Mark Chmarny
SSH Key Fingerprint: GvwYDxrKzpkXGzFjtmQ26NvM9ij3PeD/qnNvBmT67zE
Verified
Learn about vigilant mode.
5f8423c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release brings 12 new recipes with coverage for Nebius B300, GB300 on GKE, Amazon Linux 2023 on EKS, and topology-aware scheduling with Kueue, adds SLES support on RKE2, introduces Terraform and Rancher Fleet deployers, and adds graded opt-in NVSentinel remediation and least-privilege cluster access, alongside updated component pins.

Highlights

New Recipes and OS Support - 12 new recipes expand service, accelerator, OS, and scheduling coverage, with additional SLES support through a shared RKE2 base:

  • Nebius MK8s B300 - Nebius Managed Service for Kubernetes (--service mk8s) and B300 (--accelerator b300) join the criteria, with a B300 Ubuntu Dynamo inference recipe (NVIDIA/aicr#3278)
  • GB300 (A4X Max) on GKE - bare-metal COS recipes for training, Kubeflow training, inference, and Dynamo inference (NVIDIA/aicr#2811)
  • Amazon Linux 2023 on EKS - --os amazonlinux inference recipes for H100, GB200, GB300, and RTX PRO 6000 (NVIDIA/aicr#2743)
  • SUSE Linux Enterprise Server on RKE2 - --os sles, an os-sles mixin, and a shared rke2-sles base add SLES support with SUSE precompiled NVIDIA driver containers (NVIDIA/aicr#3040)
  • Topology-aware scheduling - opt-in platform: kueue recipes for H100 on Kind, GKE, and EKS install Topograph alongside Kueue, so Kueue has the node topology labels it needs to place pods (NVIDIA/aicr#3147)

NVSentinel Remediation Mixins - Remediation is now opt-in in graded steps: nvsentinel-observe (dry run), nvsentinel-quarantine (cordon and drain), and nvsentinel-remediation (repair, capped at three attempts) (NVIDIA/aicr#3137). Live remediation requires the health-events analyzer, which flags a node that keeps getting repaired without the fault going away (NVIDIA/aicr#3248). Every Slurm recipe now turns Slurm health-check drains into NVSentinel health events, and nvsentinel-slinky-drainer drains a faulty node through Slurm so running jobs finish first (NVIDIA/aicr#3211).

Updated Recipes

  • GPU Operator moves to v26.7.2, Network Operator to 26.7.0, NodeWright to v0.20.0, NVSentinel to v1.28.0 (NVIDIA/aicr#3312), and NVCRE to v0.6.0
  • Kueue moves to 0.19.6, Slinky to 1.2.2, and the MariaDB operator to 26.10.1, with the MariaDB database image used by Slurm accounting held at 11.8.8 by tag and digest (NVIDIA/aicr#2891). KAI Scheduler remains at v0.16.9 by default; the Nebius B300 recipe uses v0.17.1.
  • NicClusterPolicy, NodeFeatureRule, and NodeFeatureDiscovery CRs now update in place instead of being deleted and recreated as Helm hooks. Affected Flux installations need a one-time migration; see Upgrade Notes below (NVIDIA/aicr#3237).
  • NFD uninstall no longer prunes node labels, preserving labels used by GPU, network, and DRA workloads. Permanent NFD removal now requires explicit label cleanup (NVIDIA/aicr#3233).
  • Controller memory limits now cover clusters of about 2,500 nodes, and nfd-worker publishes labels instead of raw features to cut nfd-master memory (NVIDIA/aicr#3212)
  • OKE fabric recipes fail qualification when Oracle's managed NvidiaNetworkOperator add-on is present (NVIDIA/aicr#3216)
  • A new opt-in mixin wires a separately managed NVIDIADriver CR (NVIDIA/aicr#2942)

Terraform Deployer - aicr bundle --deployer terraform renders a recipe containing Helm components as a Terraform configuration, or as a module to call from your own. Each component becomes a module call and the recipe's dependency graph becomes depends_on, so charts install in parallel where the graph allows and terraform plan shows every release before anything changes. Recipes containing Kustomize components are rejected (NVIDIA/aicr#3217).

Rancher Fleet Deployer - aicr bundle --deployer fleet generates GitRepo bundles by default, or HelmOp resources with --fleet-mode helmop. Fleet reconciles the generated bundles in installation order, with each waiting for its predecessor to become ready (NVIDIA/aicr#3043).

Least-Privilege Operation - aicr validate --validator-service-account-name runs validator Jobs as an administrator-provisioned ServiceAccount, so the caller no longer needs permission to grant cluster-admin (NVIDIA/aicr#3226). aicr snapshot deploys its agent into an existing namespace without cluster-scoped namespace create (NVIDIA/aicr#3255), and --discover-network is now read-only, shrinking its ClusterRole from nine mutating rules to one read (NVIDIA/aicr#3242).

Cluster Upgrades - aicr recipe --inherit-from <prior-recipe-or-bundle> --inherit=namespace regenerates a recipe using component namespaces recorded in the prior artifact while taking other settings from the new release (NVIDIA/aicr#3114). aicr upgrade-check --from cluster reads only metadata in its at-risk scan and counts Helm-labeled pre-upgrade and post-upgrade hooks as owned (NVIDIA/aicr#3205). It also names Argo CD Applications it could not place (NVIDIA/aicr#3203).

Other Improvements

  • Recipe evidence can be signed with a KMS key (--signing-key) and checked with aicr evidence verify --key, for private use (NVIDIA/aicr#3241)
  • aicr bundle --vendor-charts refuses a pulled chart whose bytes differ from a digest pinned in the registry's new helm.defaultDigest field (NVIDIA/aicr#3221)
  • aicr validate --image-refs writes the pushed evidence digest to a file, and both validate and bundle can write it to a Job's termination-message file (NVIDIA/aicr#3232)
  • Five validator checks record redaction-safe outcomes in CTRF extra, and every in-pod skip names its reason, so a redacted evidence bundle still shows what was measured and why a check skipped (NVIDIA/aicr#3284)
  • The opt-in Cluster Readiness Engine check nccl-cre-all-reduce-bw is qualified on EKS H100. cre-training-goodput is registered but skips on every target because its workload does not run on pinned CRE v0.6.0 (NVIDIA/aicr#3133).
  • Shared DaemonSets, including NFD workers and EBS CSI nodes, receive the union of keyed system and accelerated-node tolerations, fixing scheduling across separately tainted node pools (NVIDIA/aicr#3141).
  • Dynamo performance validation matches runtime images to the recipe's dynamo-platform chart version, honors AICR_VALIDATOR_IMAGE_REGISTRY, propagates image-pull credentials to its workloads, and identifies the pod and image on image-pull timeouts (NVIDIA/aicr#3258).
  • The bundler skips the DRA migration probe when GPU Operator manages no driver, fixing a v1.0.0 regression (NVIDIA/aicr#3115), and NodeWright no longer hangs on reboot (NVIDIA/aicr#3101)
  • --set nv-sentinel:enabled=false works again on GKE and OKE bundles, and aicr snapshot --max-nodes-per-entry now takes effect in Job mode
  • Published OpenVEX statements document why aiperf-bench is not_affected by five libexpat1 HIGH findings, based on analysis of the affected library's reachability (a18cb56).

Upgrade Notes

  • Flux network-operator migration: affected releases originally installed client-side require Flux v2.8+ for the one-time migration, for both Git and OCI bundles. The general Flux OCI prerequisite remains v2.7+ (NVIDIA/aicr#3293); follow the network-operator migration procedure.
  • MariaDB operator and accounting database: when upgrading from 26.3.0 or 26.6.0, enable updateStrategy.autoUpdateDataPlane on HA MariaDB instances before upgrading CRDs and then the operator; disable it after the data plane updates. AICR's default non-HA accounting database needs no such flag change. The new image digest pin causes one restart of an existing accounting database and temporarily interrupts slurmdbd's database connection, so plan a quiet upgrade window (upgrade procedure).
  • Kueue with topology-aware scheduling disabled: also set TASPartialSlices=false alongside existing TAS subgate overrides, in the same change that upgrades Kueue to 0.19.6. Earlier controllers reject this new gate. Default configurations are unaffected (NVIDIA/aicr#2891).

Thanks to @abonillabeeche, @ArangoGutierrez, @atif1996, @awolde, @EronWright, @ezhang3333, @katepallewarprathmesh-sketch, @kky-fury, @La1itchauhan, @lockwobr, @mikecook, @mohityadav8, @ravisoundar, @rawsocket, @rorajani, @SammyTourani, @srao-nv, @varmesh, @yuanchen8911, and @mchmarny.

Changelog

New Features

Bug Fixes

Other Tasks

Contributors

mchmarny, mikecook, and 19 other contributors
Assets 22
Loading