Repository navigation
AICR's first stable release puts the v1 stability contract into effect, completes the apiVersion migration, extends component upgrade safety to live clusters and generated bundles, and adds validation that proves GPU access and fabric capability rather than assuming it.
Highlights
Community - AICR launched at KubeCon EU in March this year. Today, little over 6 months later, the AICR project reached its first stable release. In that short time over 85 people from more than 20 organizations have contributed, filing >1K issues and submitting >1.5K PRs. Every bug report, question, review, and fix helped decide what v1.0.0 commits to. Thank you!
v1 Stability Contract - The aicr CLI, the REST API, the Go SDK (pkg/client/v1), and the bundle layout and artifact schemas are now frozen. A breaking change on any of them waits for the next major release and is announced first under the deprecation policy.
Component Upgrade Safety - aicr upgrade-check --from cluster compares against what is actually installed, and warns about live objects an upgrade could destroy (NVIDIA/aicr#2840). Upgrade checks now also catch a change to a component's chart, source, path, type, manifests, or object names, and a bundle carries the upgrade steps for its own deployer whenever a pinned version needs them (NVIDIA/aicr#2986).
Validation Hardening
slinky-slurm-gpu-accessproves that a Slurm job given one GPU can use exactly that GPU, and that a job without one cannot open any (NVIDIA/aicr#3057)- GKE checks now verify that the GPUDirect-TCPXO fabric is usable (every a3 node exposes all 8 GPU NICs), not just that its Networks exist, and that
bundle-installerpools have driver auto-install turned off - Wider conformance coverage:
robust-controllerandsecure-accelerator-accesson the OKE, LKE, and RTX PRO 6000 bases, IMEX for recipe-supplied NCCL runtimes, and a fail-closed OKE device-plugin check
Bundler and Deployer Reliability - The Helm readiness gate now blocks the deploy and re-runs on upgrade, and its Jobs schedule on system nodes instead of hanging Pending on tainted clusters. Bundle scripts take the cluster connection from KUBE_CONTEXT and KUBECONFIG. A missing component dependency is reported as missing, not as a false cycle, before any deployer writes output.
Recipes and Coverage
- GB200 GKE Kubeflow gains
torch-distributed-rdma, a GPUDirect-RDMA training runtime a TrainJob selects by name (NVIDIA/aicr#2859) - GPU Operator moves to v26.7.1, dropping the ComputeDomain CRD workaround, with the OpenShift OLM channel on v26.7
- NVSentinel moves to v1.25.0, with the metadata-collector back on for VR200 and enabled on the k0s H200 leaf; Grove moves to v0.1.0-alpha.13
k8s-aibommoves to v1.5.1, andaicr recipe --runtime-inventory enabledcan now add it to any stock GKE recipe- VR200 recipes ship one rebooting NodeWright CR instead of two, replacing the standalone
rdma-netns-exclusivecomponent - EKS and AKS gain service-level Ubuntu overlays, matching GKE
- A
--dataoverlay can now add values to an embedded profile without forking the overlay that declares it (NVIDIA/aicr#2985)
Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @EronWright, @ezhang3333, @framsouza, @lalitadithya, @lockwobr, @mikecook, @mohityadav8, @njhensley, @rorajani, @SatyamPandey-07, @srao-nv, @varmesh, @yuanchen8911, and @mchmarny.
Changelog
New Features
- 71a1d47: feat(bundler): render upgrade guidance into bundles per deployer (#2986) (@lockwobr)
- 8c1d823: feat(ci): require /ok-to-test confirm-ci when a PR touches .github (#2938) (@mchmarny)
- 6a8cffd: feat(evidence): publish GB300 EKS evidence from v1.0.0-rc1 (#3098) (@yuanchen8911)
- 8e43f71: feat(evidence): surface aicr build commit on corroboration dashboard (#3063) (@mikecook)
- fa180fe: feat(gke): verify bundle-installer pool driver-install mode (#2943) (@mikecook)
- a9e4bb9: feat(header): retire alpha and empty apiVersions (ADR-022 N+2) (#2925) (@mchmarny)
- 8c53a74: feat(inventory): read installed components from a live cluster (#2840) (@lockwobr)
- 4947d95: feat(recipe): let --data add values to an embedded profile (#2985) (@mikecook)
- 78d52e4: feat(recipes): add GPUDirect-RDMA torch-distributed-rdma runtime for GB200 GKE Kubeflow (#2859) (@mikecook)
- 0743619: feat(recipes): add k8s-aibom GKE opt-in and upgrade to v1.5.1 (#2976) (@mchmarny)
- 91211b0: feat(release): support the previous minor, not just the latest (#2928) (@mchmarny)
- 9527a47: feat(uat): add a simulated-GPU Kind UAT lane for the Slurm stack (#2661) (@ArangoGutierrez)
- d48fc73: feat(uat): complete sxid fault-injection phase, Stages 1-3 (#2625) (#3028) (@mohityadav8)
- d22837f: feat(uat): move AWS daytime UAT to CT capacity with two slots (#2917) (@rorajani)
- 20e7e12: feat(upgrade): compare and inherit component object names (#2874) (@lockwobr)
- ac0c71a: feat(upgrade): guard chart, source, path, type and manifest identity (#2975) (@mikecook)
- 1bc0df7: feat(validation): verify GKE GPU NIC capability, not just Network existence (#3011) (@srao-nv)
- 5282788: feat(validator): verify GPU access and isolation on Slinky Slurm (#3057) (@ArangoGutierrez)
Bug Fixes
- 0cd2fa1: fix(bom): resync catalog with #2909 and drop a false lockstep claim (#2922) (@mchmarny)
- 37be501: fix(bundler): quote recipe-supplied name and namespace in install.sh (#2853) (@lockwobr)
- 08a8102: fix(bundler): recognize OCP component aliases in DRA rollout, annotation, and remedy paths (#2346) (@mohityadav8)
- 54a352d: fix(bundler): resolve cluster connection per binary in bundle scripts (#2858) (@lockwobr)
- 80e0789: fix(bundler): schedule readiness gate Jobs on system nodes (#2992) (@ayuskauskas)
- 44be0b0: fix(bundler): the helm readiness gate blocks and re-runs on upgrade (#2863) (@EronWright)
- aa1f2c6: fix(ci): authenticate license URL probes in make notices (#3056) (@mchmarny)
- ec06473: fix(ci): cache the Argo CD Redis image for the KWOK deployer lanes (#2959) (@mchmarny)
- 384655b: fix(ci): drop yq from the helm-diff checksum refresher (#2983) (@yuanchen8911)
- d0c6691: fix(ci): fail make bom-docs loudly on silent chart pull failures (#3007) (@mikecook)
- 793647a: fix(ci): fetch grype install.sh from the pinned tag, not main (#3023) (@mikecook)
- 59d2f83: fix(ci): re-pin grype install.sh checksum for v0.120.0 (@mchmarny)
- 8d73c92: fix(ci): stop one failed license probe from aborting make notices (@mchmarny)
- a1b1bbf: fix(ci): stream createCommitOnBranch payloads past the argv limit (#2918) (@mchmarny)
- 8ba4c43: fix(ci): surface every drift candidate, not just the largest step (#2965) (@mchmarny)
- 777293f: fix(cli): resolve symlinked parent for --image-refs (#2895) (#2958) (@mohityadav8)
- 0fc3645: fix(collector): fail closed on unlabeled OKE legacy device plugin (#3012) (@mikecook)
- 6db980d: fix(evidence): record opt-in-only service values as not-inferable (#2631) (#2894) (@atif1996)
- 7827c27: fix(recipe): let a Slurm bundle's own recipe.yaml load back (#2994) (@lockwobr)
- 10678e0: fix(recipe): normalize embedded and layered data provider paths (#2831) (#2832) (@SatyamPandey-07)
- e4a9461: fix(recipes): add robust-controller and secure-accelerator-access to … (#2946) (@mikecook)
- d983083: fix(recipes): add service-level OS overlays for EKS and AKS (#2730) (@ezhang3333)
- 8cc2c19: fix(recipes): bump nvsentinel to v1.25.0 (#2997) (@varmesh)
- 32c87dd: fix(recipes): drop temporary slinky-topograph image override (#2972) (@yuanchen8911)
- 2c3e13f: fix(recipes): enable nvsentinel metadata-collector on k0s H200 leaf (#2967) (@yuanchen8911)
- bb7c447: fix(recipes): fold VR200 RDMA netns mode into the tuning NodeWright CR (#2988) (@ayuskauskas)
- 82bccef: fix(recipes): mount only NVML for VR200 metadata-collector (#3104) (@yuanchen8911)
- 57f2b94: fix(recipes): nvsentinel v1.22.0 + re-enable VR200 metadata-collector (#2711) (@framsouza)
- 6f63bc4: fix(recipes): replace frozen os-shell init image with pinned busybox (@mchmarny)
- e158e52: fix(recipes): report dependency errors consistently across deployers (#2993) (@ezhang3333)
- 915aa78: fix(tests): read the full log window in object-monitor assertion (#2963) (@mchmarny)
- a9da6e7: fix(tools): install the pinned yq and verify its checksum (#2944) (@mchmarny)
- 43f83b6: fix(tools): verify every binary-release install through one helper (#2940) (@mchmarny)
- c346ec2: fix(uat): collect nodewright namespace pod logs in debug bundle (#3005) (@ayuskauskas)
- 34e35c1: fix(uat): drop trailing space that fails lint-yaml on main (@mchmarny)
- 2b78514: fix(uat): re-login to Azure before each teardown and janitor destroy (#3018) (@varmesh)
- 277d211: fix(uat): run AWS nightly UAT on the CT reservation (#3001) (@varmesh)
- bd30b1e: fix(validator): bump aiperf-bench base to distroless python 3.13-v4.1.5 (@mchmarny)
- 1a76fe7: fix(validator): find nodewright controller-manager by label, not name (#3037) (@mikecook)
- 3daf4ae: fix(validator): poll through nodewright CRD establishment (#3034) (@mikecook)
- 6366659: fix(validator): provision IMEX for supplied NCCL runtimes (#2974) (@yuanchen8911)
- c28466c: fix(validator): restore dra-node-labeler health check on the DRA-eviction opt-in path (#3045) (@atif1996)
- efc56fe: fix: bump gpu-operator to v26.7.1, drop ComputeDomain CRD workaround (#2966) (@yuanchen8911)
- 6380894: fix: normalize criteria serialization keys to lowercase with omitempty (#2961) (@mohityadav8)
Other Tasks
- 5c09ecc: chore(deps): Update dependency golangci/golangci-lint to v2.14.0 (#2957) (@github-actions[bot])
- 00c28ce: chore(deps): Update dependency micromark-core-commonmark to v2.0.4 (#2984) (@github-actions[bot])
- 2168df8: chore(deps): Update dependency orhun/git-cliff to v2.14.2 (#2953) (@github-actions[bot])
- 6821cbb: chore(deps): Update docker.io/alpine/kubectl Docker tag to v1.37.0 (#2909) (@github-actions[bot])
- 2b503b8: chore(deps): Update docker.io/alpine/kubectl Docker tag to v1.37.1 (#2981) (@github-actions[bot])
- 938716c: chore(deps): Update docker.io/library/busybox:1.38.0 Docker digest to cac8f90 (#2932) (@github-actions[bot])
- 86fa567: chore(deps): Update ministackorg/ministack Docker tag to v1.5.13 (#2889) (@github-actions[bot])
- f5fa504: chore(deps): Update testing-tools (#2950) (@github-actions[bot])
- 8d0a499: chore(deps): Update testing-tools (#2955) (@github-actions[bot])
- 2e4a12d: chore(deps): Update testing-tools (#2982) (@github-actions[bot])
- 10af7fe: chore(deps): Update testing-tools (#3051) (@github-actions[bot])
- 587953c: chore(deps): Update ubuntu:26.04 Docker digest to 3595d7f (#3050) (@github-actions[bot])
- 11258f6: chore(deps): Update ubuntu:26.04 Docker digest to da6fc2b (#2887) (@github-actions[bot])
- 2bd3f39: chore(deps): bump aiperf to 0.13.0 and fix license arch guard (#2987) (@mchmarny)
- 4e36a05: chore(deps): bump grype, tilt, helmfile, and zarf tool pins (@mchmarny)
- fd3ab5b: chore(deps): upgrade Go modules, holding grpc, kube-openapi, cel-go (@mchmarny)
- 73f9b6f: chore(deps): upgrade Go modules, holding kube-openapi and cel-go (@mchmarny)
- a57a10a: chore(fern): register v0.22.0 (#2898) (@github-actions[bot])
- caa38a6: chore(recipes): bump grove to v0.1.0-alpha.13 (#2979) (@yuanchen8911)
- 2810c76: chore(recipes): move OpenShift GPU Operator OLM channel to v26.7 (#2970) (@yuanchen8911)
- 556f9d3: chore(skills): record the 2026-09-28 component drift verdicts (@mchmarny)
- 390f888: chore: add nvsentinel runner role to access uat clusters (#3054) (@lalitadithya)
- 27b3d7a: chore: add nvsentinel runners IP (#3021) (@lalitadithya)
- 9dca992: chore: deps: bump actions/cache/restore from 5.0.3 to 6.1.0 (#3049) (@dependabot[bot])
- 5fc8235: chore: deps: bump actions/cache/save from 5.0.3 to 6.1.0 (#3047) (@dependabot[bot])
- 7eae380: chore: deps: bump renovatebot/github-action from 46.3.1 to 46.3.3 (#2947) (@dependabot[bot])
- b6b3ab8: chore: deps: bump renovatebot/github-action from 46.3.3 to 46.3.4 (#2952) (@dependabot[bot])
- 27f7e26: chore: deps: bump renovatebot/github-action from 46.3.4 to 46.3.5 (#3015) (@dependabot[bot])
- c96a59c: chore: deps: bump renovatebot/github-action from 46.3.5 to 46.3.6 (#3048) (@dependabot[bot])
- 64d0e5e: chore: deps: bump the codeql-action group with 3 updates (#2951) (@dependabot[bot])
- d6c5b95: chore: ensure correct communication in security group for nvsentinel (#3065) (@lalitadithya)
- 58ab9b3: chore: increase aws system node pool since mongodb fails to schedule (#3068) (@lalitadithya)
- 89ad0ce: chore: retire two superseded GB300 evidence pointers naming :edge val… (#2968) (@mikecook)
- bf2d622: chore: set UAT system node count so that it doesn't autoscale when nvsentinel installs (#3061) (@lalitadithya)
- 424f71c: ci(kwok): speed up Tier 1 cells and label KWOK as advisory (#2990) (@mikecook)
- 5fcd8de: ci(uat): gate release cells on a harness-compat floor (#2991) (@njhensley)
- 6d5973b: ci(uat): split readiness gate into its own phase and step (#2989) (@njhensley)
- 443fe43: ci: retry transient release downloads in setup-tools (#3009) (@mikecook)
- e629186: ci: watch component manifests/values in bom-freshness path filter (#2927) (@mikecook)
- e874f9f: docs(deployer): fix helmfile CRD claims and apply command (#2845) (@lockwobr)
- b4d7a3c: docs(evidence): add attestation evidence for GKE GB200 leaves (#2926) (@mikecook)
- c826c5a: docs(recipe): clarify recipe stewardship and composition (#3000) (@framsouza)
- 25a725b: docs(recipes): record k8s-aibom v1.5.1 rollback and API-server cost (#3002) (@mchmarny)
- 6c73d51: docs(skills): focus AICR project triage on ready work (@mchmarny)
- 9154535: docs(tests): name the real gate for the kind NVSentinel assert (#2920) (@ArangoGutierrez)
- c3b880a: docs(upgrading): correct what a namespace move actually does (#2899) (@lockwobr)
- 7757534: refactor(evidence): share sha256(issuer, identity) via HashIdentityPair (#2905) (@mikecook)
- 261234b: refactor: use attestation.HashBytesSHA256 in testgrid-publish (#2908) (@mikecook)
- d54c06a: test(recipes): enforce Prometheus Operator appVersion lockstep (#2929) (@ezhang3333)
- f812184: test(uat): run janitor azure re-login tests on macOS (@mchmarny)
- 1438a2e: test(validator): pin chart-shape constants to the pinned charts (#3008) (@ayuskauskas)