v1.2.0

This release adds NVKE GB200 recipes with CNCF AI conformance evidence and CKS training previews, updates GPU stack components, introduces the AICR Operator preview and core and ops bundles, and begins the NKTP telemetry pipeline, alongside opt-in NVSentinel features and new bundle output options. It also welcomes 7 new contributors.

Highlights

New and Updated Recipes

• CKS H100 and RTX PRO 6000 - training previews on CoreWeave Kubernetes Service (NVIDIA/aicr#3527, NVIDIA/aicr#3526)
• NVKE GB200 - training and Kubeflow training, details below
• SLES on RKE2 - nodes are now tuned with the nvidia-tuned generic profile (NVIDIA/aicr#3336)
• Rocky Linux - recognized as an OS criteria value (--os rocky) (NVIDIA/aicr#3332)
• nvidia-setup moves to 0.9.0 and the EKS setup chain is reordered, moving EKS nodes to the 7.0 kernel (NVIDIA/aicr#3519)
• nvidia-tuned 0.11.0, nvidia-dra-driver-gpu 0.5.1, nfd 0.20.1, and topograph 1.1.0
• k8s-aibom moves to v1.6.2 and ships in the GKE Dynamo recipes (NVIDIA/aicr#3373, NVIDIA/aicr#3375)
• gpu-operator operands tolerate NVSentinel's release taint (NVIDIA/aicr#3360)

NVKE GB200 and CNCF AI Conformance - A new NVKE GB200 Ubuntu training recipe uses SR-IOV InfiniBand and deploys Multus and the CNI plugins (NVIDIA/aicr#3295, NVIDIA/aicr#3468). A Kubeflow variant is included. AICR's NVKE v1.36 CNCF AI conformance submission is published with its evidence (NVIDIA/aicr#3455).

AICR Operator (Preview) - The opt-in operator runs a pinned AICR image in-cluster, validates each bundle's recipe, fingerprints inventory specs, and gates bundle rollout on approval (NVIDIA/aicr#3475, NVIDIA/aicr#3490). A default in-cluster registry removes the need for an external registry; operator images ship in the release train (NVIDIA/aicr#3533, NVIDIA/aicr#3491). The v1.2.0 operator defaults to AICR v1.1.2, the release with which it was tested. See the operator support policy to run another release.

Core and Ops Bundles - Per ADR-018, every registry component now declares a class (NVIDIA/aicr#3350):

• core covers what a plain GPU pod needs to run, with its fabric and install prerequisites.
• ops covers everything else, including workload platforms.

aicr bundle --split renders a class-partitioned bundle set, so the core GPU stack can be deployed and owned separately from the ops layer (NVIDIA/aicr#3420).

NKTP Telemetry (Initial Implementation) - ADR-026 adopts the NVKE Telemetry Pipeline (NKTP) as AICR's telemetry pipeline (NVIDIA/aicr#3390). This release lays the groundwork:

• NKTP joins the component registry (NVIDIA/aicr#3451).
• Components can declare the Prometheus metrics they read (NVIDIA/aicr#3450).
• Recipes scrape PodMonitors and enable chart-provided monitors (NVIDIA/aicr#3338).

Switching the stock recipes to NKTP comes in a later release.

NVSentinel - New opt-in mixins add new-node validation with a DCGM diagnostic (NVIDIA/aicr#3464), an event exporter (NVIDIA/aicr#3371), and a log collector (NVIDIA/aicr#3354). AICR now owns node-drainer drain scope and timing (NVIDIA/aicr#3415).

Other Improvements

• OCI output: aicr bundle --output oci-layout:<dir>[:tag] writes OCI layouts locally without pushing, with concurrent-writer protection, failure cleanup, and target tag/digest validation (NVIDIA/aicr#3538, NVIDIA/aicr#3557, NVIDIA/aicr#3563).
• Argo CD and deployers: --argocd-* settings cover project, name prefix, cascade delete, destination, and child sync retry. Deployer settings, child Application names, and retry backoff bounds are validated before any output is written (NVIDIA/aicr#3432, NVIDIA/aicr#3562, NVIDIA/aicr#3563, NVIDIA/aicr#3513).
• Evidence and upgrades: aicr recipe --require-evidence rejects recipes without published evidence; aicr upgrade-check follows the source's upgrade path and reports undefined-origin when no record covers it (NVIDIA/aicr#3342, NVIDIA/aicr#3481, NVIDIA/aicr#3394).
• Snapshot and cleanup: The snapshot agent bounds host reads and recovers written snapshots; Go-based tools/cleanup determines release ownership from inventory and bundle-info (NVIDIA/aicr#3430, NVIDIA/aicr#3325, NVIDIA/aicr#3433).
• Validation and readiness: Fully qualified test pod images support CRI-O; busy GPU nodes are skipped individually. Bundle readiness gates use component values, and network-operator bundles require the NFD NIC rule (NVIDIA/aicr#3359, NVIDIA/aicr#3362, NVIDIA/aicr#3346, NVIDIA/aicr#3446).
• CRE checks: Qualified on EKS H100 with NVCRE v0.8.2; nccl-cre-all-reduce-bw applies a floor fraction to its bandwidth target (NVIDIA/aicr#3251, NVIDIA/aicr#3418).

Security - The Go toolchain moves to 1.27.3 for the crypto/x509 CA distrust, sigstore/rekor to v1.5.5 for a go-chi CVE, and the Percona MongoDB and aiperf-bench images are refreshed (NVIDIA/aicr#3377, NVIDIA/aicr#3516, NVIDIA/aicr#3524).

Deprecations

• Argo CD options as deployer: value overrides - --set deployer:<key>=<value>, the matching POST /v1/bundle set value, and deployer: entries in spec.bundle.deployment.set are deprecated. Use the dedicated --argocd-* flags, spec.bundle.deployment.argocd config fields, or REST parameters instead. Removal is planned for v2.0.0 (deprecations).

New Contributors - Seven contributors landed their first change in this release, from new recipes and OS support to validator and attestation fixes:

• @csy20 - aicr recipe --require-evidence (NVIDIA/aicr#3342)
• @ErikJiang - attestation cancellation handling (NVIDIA/aicr#3489)
• @git-jxj - matching attestation for release candidates (NVIDIA/aicr#3326)
• @gusraggio - SLES RKE2 tuning with nvidia-tuned (NVIDIA/aicr#3336)
• @mlsorensen - CRI-O support in validator test pods (NVIDIA/aicr#3359)
• @robbert229 - CKS H100 and RTX PRO 6000 recipes (NVIDIA/aicr#3527, NVIDIA/aicr#3526)
• @vjsairam - Rocky Linux OS criteria (NVIDIA/aicr#3332)

Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @csy20, @depohmel, @ErikJiang, @git-jxj, @gusraggio, @La1itchauhan, @lockwobr, @mikecook, @mlsorensen, @mohityadav8, @njhensley, @ravisoundar, @robbert229, @rorajani, @srao-nv, @varmesh, @vineeth-bandi, @vjsairam, @yuanchen8911, and @mchmarny.

---

Changelog

New Features

• caf18dc: feat(bundle): write OCI image layout without pushing (#3538) (@njhensley)
• dbd6cd4: feat(bundler): --split renders a class-partitioned bundle set (#3420) (@yuanchen8911)
• a3046c2: feat(bundler): add --argocd-* settings and child sync retry (#3432) (@lockwobr)
• f57f796: feat(ci): add a GPU-free gpu-operator lane backed by Mokka (#2935) (@ArangoGutierrez)
• cf1a87d: feat(cleanup): decide release ownership from inventory and bundle-info (#3433) (@mikecook)
• b49eaaa: feat(nvsentinel): add opt-in event exporter mixin (#3371) (@varmesh)
• cbda002: feat(nvsentinel): add opt-in log collector mixin (#3354) (@varmesh)
• d4776f6: feat(nvsentinel): add opt-in new-node validation mixin (#3464) (@varmesh)
• 74a6650: feat(nvsentinel): own node-drainer drain scope and timing (#3415) (@varmesh)
• c44288f: feat(operator): add the AICR Operator and its kind CI lanes (#3475) (@mchmarny)
• 511bf93: feat(operator): run an in-cluster registry by default (#3533) (@mchmarny)
• de51e0b: feat(recipe): ADR-018 class field and class dependency boundaries (#3350) (@yuanchen8911)
• 864dd60: feat(recipe): add NVKE GB200 Ubuntu training recipe (SR-IOV IB) (#3295) (@atif1996)
• 634866e: feat(recipe): let components declare the Prometheus metrics they read (#3450) (@vineeth-bandi)
• 87c6d74: feat(recipe): recognize Rocky Linux as an OS criteria value (#3332) (@vjsairam)
• 4a1c859: feat(recipes): add CKS H100 training preview (#3527) (@robbert229)
• c7f2947: feat(recipes): add CKS RTX PRO 6000 training preview (#3526) (@robbert229)
• b0a00f0: feat(recipes): add NVKE Telemetry Pipeline (NKTP) to the registry (#3451) (@vineeth-bandi)
• 2e61c12: feat(recipes): bump nvidia-setup to 0.9.0 and reorder the EKS setup chain (#3519) (@ayuskauskas)
• 631e3ad: feat(recipes): requalify and re-pin k8s-aibom to v1.6.1 (#3373) (@mchmarny)
• 181d81f: feat(recipes): scrape PodMonitors and enable chart-provided monitors (#3338) (@vineeth-bandi)
• 04ea1e5: feat(recipes): ship k8s-aibom in the GKE Dynamo recipes (#3375) (@mchmarny)
• c7617e9: feat(recipes): tune SLES RKE2 nodes with the nvidia-tuned generic profile (#3336) (@gusraggio)
• b3df37f: feat(release): ship the AICR Operator preview in the release train (#3491) (@mchmarny)
• f5dbd79: feat(snapshot): bound agent host reads and recover a written snapshot (#3430) (@mikecook)
• 601f1f1: feat(validator): apply floor fraction in nccl-cre-all-reduce-bw (#3418) (@rorajani)
• 081be51: feat(validator): qualify CRE checks on EKS H100 with NVCRE v0.8.2 (#3251) (@rorajani)
• a2f7cb2: feat: add --require-evidence for recipe resolution (#3342) (@csy20)

Bug Fixes

• f9cf196: fix(attestation): classify context cancellation as ErrCodeCanceled (#3489) (@ErikJiang)
• d5b80a7: fix(bundle): close the OCI layout publish race and clean up on failure (#3557) (@njhensley)
• 78a07e6: fix(bundler): close idle connections in per-attempt HTTP clients (#3522) (@mohityadav8)
• e4aef5f: fix(bundler): place readiness gates from component values (#3346) (@mikecook)
• f4b4de3: fix(bundler): reject network-operator bundles missing the nfd NIC rule (#3446) (@yuanchen8911)
• 935e074: fix(bundler): tighten argocd retry and oci-layout target validation (#3563) (@lockwobr)
• 4d4e2e9: fix(bundler): validate argocd child names before writing output (#3562) (@lockwobr)
• 10f1e40: fix(bundler): validate deployer config before writing output (#3513) (@lockwobr)
• 3013757: fix(bundler): verify and copy binary attestation from one read (#3539) (@mohityadav8)
• 2eb300d: fix(ci): complete weekly scan of NCCL preflight images (#3431) (@lockwobr)
• 69ddbac: fix(ci): gate UAT reservation-id against cluster-config targets (#3388) (@mikecook)
• 4e02780: fix(client): reject a foreign result in RequirePublishedEvidence (#3364) (@yuanchen8911)
• df40bdc: fix(collector): skip sysctls whose read can block the agent (#3393) (@ayuskauskas)
• dd5b494: fix(constraints): identify GPU nodes by gpu.present at readiness (#3380) (@mikecook)
• bc08a3c: fix(deps): bump Go toolchain to 1.27.3 for crypto/x509 CA distrust (@mchmarny)
• 6f778c8: fix(docs): report docs link check stat failures and bare blob/main URLs (#3321) (@mikecook)
• a44f720: fix(kwok): skip the SR-IOV pre-delete hook in NVKE cleanup (#3453) (@yuanchen8911)
• 937be0d: fix(operator): resolve the post-merge review of the operator (#3501) (@mchmarny)
• 4629f6d: fix(operator): retry a push GHCR denies while it creates a package (#3518) (@mchmarny)
• 0d5c276: fix(operator): validate the bundle recipe, fingerprint inventory specs (#3490) (@mchmarny)
• 480c10d: fix(recipe): deploy multus and CNI plugins on NVKE GB200 (#3468) (@atif1996)
• d0e0bc2: fix(recipe): rebind health checks to inherited object names (#3424) (@mikecook)
• d682d92: fix(recipes): drop kubeflow-trainer values the chart never reads (#3541) (@La1itchauhan)
• 0d81295: fix(recipes): tolerate NVSentinel's release taint on gpu-operator operands (#3360) (@atif1996)
• 9e94071: fix(release): start stable release notes at the previous stable tag (#3313) (@La1itchauhan)
• e009d29: fix(security): VEX pillow 12.3.0 advisories on aiperf-bench (@mchmarny)
• a1b6eba: fix(security): bump Percona MongoDB server and exporter images (#3377) (@mchmarny)
• 3f13408: fix(security): bump aiperf-bench base to distroless python 3.13-v4.1.10 (#3524) (@rorajani)
• a5b8d4c: fix(security): bump aiperf-bench base to distroless python 3.13-v4.1.8 (@mchmarny)
• 456c454: fix(security): bump sigstore/rekor to v1.5.5 for the go-chi CVE (@mchmarny)
• ab579a4: fix(security): ship pillow 13.0.0 in aiperf-bench, drop pillow VEX (#3516) (@rorajani)
• f1a34c7: fix(tools): unblock cleanup when workloads outlive their operator (#3448) (@yuanchen8911)
• 1141a32: fix(trust): fall back to in-memory TUF metadata on a cache write race (#3512) (@mchmarny)
• 671ca4b: fix(upgrade): report undefined-origin for sources no record covers (#3394) (@ayuskauskas)
• f78bc89: fix(upgrade): select records along the source's upgrade path (#3481) (@mikecook)
• d1fd68a: fix(validator): fully qualify test pod images for CRI-O (#3359) (@mlsorensen)
• 5e6f523: fix(validator): skip busy GPU nodes per node; align coverage output (#3362) (@yuanchen8911)
• 2efc0fc: fix(vex): suppress aiperf-bench pillow findings; compact scan Slack (#3492) (@mchmarny)
• ce01619: fix: install matching attestation for release candidates (#3326) (@git-jxj)

Other Tasks

• 75230c2: chore(deps): ECR SDK, busybox and ubuntu digests, controller-gen v0.23 (#3504) (@mchmarny)
• 907a820: chore(deps): Update dependency fluxcd/flux2 to v2.10.0 (#3507) (@github-actions[bot])
• 68d3a8a: chore(deps): Update dependency golangci/golangci-lint to v2.15.0 (#3330) (@github-actions[bot])
• 13324cc: chore(deps): Update dependency oasdiff/oasdiff to v1.34.0 (#3370) (@github-actions[bot])
• 492aa11: chore(deps): Update dependency oras-project/oras to v1.3.5 (#3369) (@github-actions[bot])
• 3fe3bb4: chore(deps): Update docker.io/alpine/kubectl:1.37.1 Docker digest to a5d0fbe (#3407) (@github-actions[bot])
• 4525559: chore(deps): Update golang.org/x/exp digest to f0251eb (#3368) (@github-actions[bot])
• 75e0af2: chore(deps): Update kubernetes to v0.20.0 (#3331) (@github-actions[bot])
• 6bc8413: chore(deps): Update mariadb Docker tag to v11.8.9 (#3352) (@github-actions[bot])
• abc3698: chore(deps): Update mariadb:11.8.9 Docker digest to 6a4b3c2 (#3434) (@github-actions[bot])
• 426d506: chore(deps): Update mariadb:11.8.9 Docker digest to c4124d1 (#3458) (@github-actions[bot])
• b8ff128: chore(deps): Update ministackorg/ministack Docker tag to v1.5.29 (#3353) (@github-actions[bot])
• b5b0ed9: chore(deps): Update module github.com/aws/aws-sdk-go-v2/service/ecr to v1.67.0 (#3521) (@github-actions[bot])
• e136b05: chore(deps): Update supply-chain (#3381) (@github-actions[bot])
• 167d8ec: chore(deps): Update testing-tools (#3329) (@github-actions[bot])
• 7af8a64: chore(deps): Update testing-tools (#3436) (@github-actions[bot])
• 47d55fd: chore(deps): Update testing-tools (#3520) (@github-actions[bot])
• 16517b1: chore(deps): Update ubuntu:26.04 Docker digest to 0e73b71 (#3459) (@github-actions[bot])
• b4b57a5: chore(deps): Update ubuntu:26.04 Docker digest to 19fcd1b (#3435) (@github-actions[bot])
• 7a09666: chore(deps): bump Go modules via make upgrade (@mchmarny)
• 681e8a0: chore(deps): bump cosign, oras, kubelogin and unist-util-visit (#3465) (@mchmarny)
• 5fa5623: chore(deps): bump oras-go, oasdiff, goreleaser and test tools (#3414) (@mchmarny)
• 7481dae: chore(deps): sigstore-go 1.4.0, yaml 3.0.6, oasdiff 1.35.0, test tools (#3550) (@mchmarny)
• ccd7a45: chore(fern): register v1.1.0 (#3322) (@github-actions[bot])
• b1d84c0: chore(fern): register v1.1.1 (#3355) (@github-actions[bot])
• 845a00c: chore(fern): register v1.1.2 (#3356) (@github-actions[bot])
• f03b7fb: chore(recipes): bump k8s-aibom to v1.6.2 (#3403) (@mchmarny)
• 90bab48: chore(recipes): bump nfd to v0.20.1 (#3383) (@ArangoGutierrez)
• 5630777: chore(recipes): bump nvidia-dra-driver-gpu to 0.5.1 (#3505) (@atif1996)
• ae43972: chore(recipes): bump nvidia-tuned to 0.10.1 (#3334) (@gusraggio)
• b10a882: chore(recipes): bump nvidia-tuned to 0.11.0 (#3400) (@gusraggio)
• ffd8e28: chore(skills): resolve the pillow VEX release check, retire the nvsentinel exclusion (@mchmarny)
• 7ead90f: chore: Upgrade topograph to v1.1.0 (#3429) (@ravisoundar)
• d97a7eb: chore: add robbert229 as a copy-PR bot trustee (#3525) (@robbert229)
• bd694fd: chore: deps: bump actions/upload-artifact in / (#3405) (@dependabot[bot])
• 0a7478f: chore: deps: bump docker/login-action in /.github/actions/ghcr-login (#3406) (@dependabot[bot])
• f3f881a: chore: deps: bump renovatebot/github-action in / (#3367) (@dependabot[bot])
• 3b125f3: chore: deps: bump renovatebot/github-action in / (#3483) (@dependabot[bot])
• 161cdca: chore: deps: bump renovatebot/github-action in / (#3544) (@dependabot[bot])
• 8413898: chore: deps: bump the codeql-action group across 3 directories with 3 updates (#3404) (@dependabot[bot])
• daf97a3: chore: generalize instance-type refs and drop alternate NET fabric (#3386) (@yuanchen8911)
• aae0b28: ci(nvsentinel): run MongoDB Kind E2Es on ubuntu-22.04 (#3348) (@mikecook)
• 9cb9b5e: ci: bump codeql-action to v4.39.0 and keep its Dependabot group (#3344) (@yuanchen8911)
• 1d7aa21: ci: pause the Mokka lane's PR and nightly triggers (#3447) (@ArangoGutierrez)
• 1ec1063: docs(adr): ADR-018 class definitions: core by necessity, ops is the rest (#3341) (@yuanchen8911)
• 1171571: docs(adr): meet ADR-025 execution gates with AICR's own limits (#3419) (@rorajani)
• a797523: docs(adr): propose ADR-026 NKTP telemetry (#3390) (@vineeth-bandi)
• f47706e: docs(adr): propose ADR-027 AICR Operator (#3466) (@mchmarny)
• 0724334: docs(catalog): show how to opt a namespace into k8s-aibom (#3421) (@mchmarny)
• 3c02c24: docs(client): name the methods that check RecipeResult ownership (#3396) (@yuanchen8911)
• 248a66e: docs(deployer): correct CRD fallback rationale, test it on envtest (#3287) (@lockwobr)
• 8b5ca7a: docs(design): ADR-028 add fabric selection in recipes (#3506) (@yuanchen8911)
• 75480a2: docs(evidence): GB300 generic training on v1.1.0-rc2 (#3324) (@atif1996)
• 212fe58: docs(fern): publish only Latest until the next release (#3391) (@yuanchen8911)
• f250960: docs(nvcre): record release ownership of cluster-scoped objects (#3281) (@rorajani)
• 0a4d1a2: docs(operator): name what the approver roles do not grant (@mchmarny)
• bb27a7e: docs(recipes): note SUSE support in tuning-rke2 bootloader scope (#3438) (@gusraggio)
• 126cc7b: docs(release): bump supported versions to 1.1.x and 1.0.x (#3530) (@yuanchen8911)
• 02194cb: docs(validator): complete the CTRF extra disclosure lists (#3318) (@mikecook)
• 6f1da35: refactor(tools): rewrite tools/cleanup in Go (#3325) (@mikecook)
• fff2d1a: test(bundler): cover every registry component in the render golden (#3339) (@mikecook)