This release focuses on GPU stack profiles, broader recipe and platform coverage, signed release provenance, and validation hardening.
Highlights
GPU Stack Profiles (ADR-015) - As AICR covers more use-cases, one criteria combination increasingly maps to more than one valid infrastructure configuration. The only prior way to express the second was a bundle-time --set override that validation could not see, so aicr validate checked the stock configuration against a cluster running a different one. Recipes now declare a named gpuStack profile naming who owns a layer of the stack, the cloud service or the GPU Operator. One value is selected at generation time via aicr recipe --profile gpuStack=<value> or the declared default, hydrated into recipe.yaml, and its owned paths are locked against diverging overrides at bundle and mirror time. Selection is explicit intent, never inferred. So for example, the AKS pool reading supplied by aicr snapshot --aks-gpu-pools qualifies a selection and fails generation closed on mismatch. The mechanism landed in v0.18 with no adopter; AKS and the full GKE family have since converted. It is service-agnostic, so OKE addons and OCP operators extend it without new schema.
Supply Chain & Provenance - Release metadata and the aiperf-bench third-party source are published as signed OCI referrers alongside release artifacts, the latest release is re-verified daily, and Rekor identity monitoring gained a resumable scan that catches up large backlogs. aicr verify can be driven from AICRConfig (spec.verify), and POST /v1/bundle supports non-interactive attestation. A registry-inventory egress gate bounds which registries the supply chain may reach.
Validation Hardening - A pass now has to be earned. The validator fails closed on unmatched declared checks and on capability checks whose declared dependencies are missing, rolls back cluster-admin RBAC when prep fails, and re-establishes the Job watch after an apiserver idle-stream closure instead of reporting a false failure. Cordoned GPU and RDMA nodes are disclosed rather than silently skipped, gang-scheduling and webhook conformance tests are now behavioral, and six operator-owning health checks assert CRD Established=True. The evidence dashboard at validation.aicr.run gained multiple UX improvements, and validator outcomes carry redaction-safe CTRF extra data.
SDK Contract - The exported surface of pkg/client/v1 is pinned and gated by a compatibility check, alias and semver contracts are reconciled, and the selector API is context-aware with snapshot Jobs routed through the facade.
Recipes & Evidence
- AKS H100 Ubuntu training and inference (Dynamo) recipe evidence
- AKS training and inference Kubernetes floors aligned to the DRA-GA >= 1.34 rationale
- Slurm/Slinky gains accounting ownership modes and shared RWX storage for home and data
- Network Operator default bumped to v26.4.1; EKS GB200 enrolled in the nightly UAT
- OpenShift Support - OCP-compatible components for cert-manager, prometheus-adapter, nvidia-dra-driver-gpu, and k8s-nim-operator,
argocd-helmbundling now covers the OCP path with a pinned error contract, and recipes enabling bothgpu-operatorandgpu-operator-ocpare rejected at resolution
Other Improvements
aicr snapshotfails closed on unusable--snapshotinput; the collector fails loud when Kubernetes collection is canceled mid-propagation and emits lossless label/taint readingsGET /v1/queryrequires a selector, versionless legacy bundle recipes are accepted, and duplicateComponentRefnames are rejected- Constraint measurement paths are validated at recipe load time;
aicr validate --fail-on-errorscoping and its exit-code and CTRF-suite semantics are documented - The BOM extracts nested operator images; the aiperf-bench runtime moved to NVIDIA distroless Python with pip removed
- Readiness checks run in-process via
pkg/chainsaw(promoted fromvalidators/chainsaw), and the external chainsaw binary is gone from the validator images
Thanks to @andrewwhitecdw, @atif1996, @framsouza, @gat786, @kaynetu, @krtadev, @lockwobr, @mohityadav8, @njhensley, @rorajani, @rsd-darshan, @tjrasche, @varmesh, @yuanchen8911, and @mchmarny.
Changelog
New Features
- 76fb50c: feat(ci): add UAT orphan janitor (dry-run-first, all clouds) (#2068) (@njhensley)
- 9b180ac: feat(ci): add registry-inventory supply-chain egress gate (#2043) (@njhensley)
- 0716a37: feat(ci): enroll EKS GB200 into the nightly UAT batch (#1979) (@njhensley)
- 9a1064d: feat(cli): support aicr verify via AICRConfig (spec.verify) (#2046) (@lockwobr)
- 81e43cf: feat(corroborate): improve evidence dashboard UX (#1935) (@njhensley)
- 24183d9: feat(notices): add Python dependency licenses to third-party notices (#2004) (@lockwobr)
- c482c2e: feat(recipe): adopt the ADR-015 gpuStack profile on AKS (#1967) (@yuanchen8911)
- 01932ea: feat(recipe): convert GKE family to the gpuStack profile (ADR-015 PR 3) (#2044) (@yuanchen8911)
- 9a2002e: feat(recipe): implement the ADR-015 profile core (#1933) (@yuanchen8911)
- 018dd55: feat(recipes): backfill CRD Established=True for 6 operator-owning health checks (#1919) (@mohityadav8)
- 16a550c: feat(server): non-interactive bundle attestation for /v1/bundle (#1891) (@lockwobr)
- 1808312: feat(skills): add aicr-cross-review skill for multi-agent PR review (#1915) (@yuanchen8911)
- ff0301b: feat(skills): add aicr-triage skill for project board triage (#1911) (@yuanchen8911)
- 7e0f4a1: feat(skills): download UAT debug bundles for failure triage (#1913) (@njhensley)
- 580f516: feat(slinky): add shared RWX storage support for home and data (#2079) (@kaynetu)
- 0752ea1: feat(slurm): add accounting ownership modes (#1970) (@kaynetu)
- c0615e6: feat(uat): slot-aware daytime cluster-name convention (ADR-017) (#2081) (@njhensley)
- f2b54e5: feat(validator): carry redaction-safe CTRF extra outcome data (#1973) (@njhensley)
Bug Fixes
- 470397f: fix(api): accept versionless legacy bundle recipes (#1943) (@yuanchen8911)
- b8d410f: fix(api): drop unemitted metadata.created from spec (#2034) (@varmesh)
- f822560: fix(api): require selector on v1 query GET (#2013) (@tjrasche)
- 93328af: fix(bom): extract nested operator images (#1960) (@yuanchen8911)
- fd26a47: fix(bundler): create argocd-helm static/ only when populated (#1944) (@yuanchen8911)
- c7fa85a: fix(ci): break orphaned TF state lock before UAT teardown (#2049) (@njhensley)
- cb97658: fix(ci): complete UAT teardown on cancel, resize UAT budgets (#2080) (@njhensley)
- 00cb051: fix(ci): retry GKE UAT bringup on transient node-pool timeout (#2066) (@njhensley)
- 2fdcb78: fix(ci): run validators/ unit tests in make test, exclude from coverage gate (#1918) (@mohityadav8)
- d1b7800: fix(ci): size the GKE UAT timeout for the bringup retry (#2078) (@lockwobr)
- 052a18e: fix(ci): sync coverage docs to 80% and fail closed on bad threshold (#1937) (@lockwobr)
- 0f611ce: fix(ci): tear down daytime-up UAT cluster on failed/cancelled run (#2067) (@njhensley)
- 5e2304c: fix(collector): emit lossless label/taint readings (#2093) (@varmesh)
- a268a00: fix(collector): fail loud when K8s collection is canceled mid-propagation (#2036) (@mchmarny)
- 30ec323: fix(deps): bump network-operator default to v26.4.1 (#1938) (@atif1996)
- f2400a4: fix(docs): fail-closed MDX check for bare '<' + MDX-safe content (#2127) (@njhensley)
- d96d977: fix(evidence): bound on-disk bundle reads with FileReadTimeout ctx (#2071) (@njhensley)
- 19a4e3a: fix(evidence): bound residual on-disk reads on publish/verify paths (#2100) (@mchmarny)
- 6ed2486: fix(evidence): correct profile recipe name emphasis (#2099) (@njhensley)
- 5bfc5d9: fix(evidence): make gang and webhook conformance tests behavioral (#2097) (@yuanchen8911)
- e8acb9c: fix(gate): close residual fail-closed and execution-parity gaps (#2052) (@mchmarny)
- 73cd9ec: fix(kwok): fail closed on unmapped profiles; skip in batch (#2033) (@framsouza)
- 6cb3ab9: fix(kwok): harden profile-select tree-fault handling and test-harness coverage (#2110) (@framsouza)
- 9fafddd: fix(kwok): strict criteria reader in CI classify; document fail-closed contract (#2109) (@framsouza)
- a153567: fix(pod): guard Job watch resume against backoff/cancel race (#2012) (@njhensley)
- af321b6: fix(recipe): align AKS inference K8s floors with DRA-GA >= 1.34 rationale (#1984) (@mohityadav8)
- aa8733f: fix(recipe): align AKS training K8s floors with DRA-GA >= 1.34 rationale (#1908) (@mohityadav8)
- b172649: fix(recipe): validate constraint measurement paths at load time (#2126) (@mchmarny)
- 81268aa: fix(sdk): context-aware selector, route snapshot Jobs through facade (#2095) (@mchmarny)
- 21100b5: fix(sdk): reconcile alias and semver contracts (#2124) (@tjrasche)
- e6caec4: fix(security): VEX aiohttp C-parser OOB read (GHSA-cq5v-8q36-5273) (#2094) (@mchmarny)
- 200f03d: fix(serializer): bound ConfigMap read with its own timeout constant (#2062) (@framsouza)
- 39980b8: fix(skills): cross-review teardown resilience, posted-output format (#2134) (@yuanchen8911)
- 2abbd2b: fix(skills): harden UAT debug-bundle triage against malformed input (#1914) (@njhensley)
- affea42: fix(skills): raise cross-review Codex wait budget to three waits (#2101) (@yuanchen8911)
- 8c8ec35: fix(skills): recover cross-review lanes, harden consensus/fallback (#1932) (@yuanchen8911)
- 0dec317: fix(snapshot): fail closed on unusable --snapshot input (#1888) (#1892) (@yuanchen8911)
- 746014c: fix(tests): derive releasepolicy script deadlines from test timeout (#2088) (@yuanchen8911)
- be4580a: fix(uat): bug fix — restore GKE-default GPU provisioning; withdraw #1755 gate (#2047) (@yuanchen8911)
- 0a2f091: fix(uat): ephemeral daytime clusters, consistent cross-cloud names (#2042) (@njhensley)
- 8cf4179: fix(validator): disclose cordoned GPU nodes in check-nvidia-smi (#1936) (@rsd-darshan)
- bfe2c3b: fix(validator): disclose cordoned RDMA nodes in expected-resources (#1981) (@njhensley)
- b52aefb: fix(validator): fail capability checks when declared deps missing (#2130) (@njhensley)
- b0719a4: fix(validator): fail closed on unmatched declared checks (#2129) (@njhensley)
- 9d89ac6: fix(validator): re-establish Job watch after apiserver idle-stream closure (#1971) (@njhensley)
- c34be55: fix(validator): resolve deferred Job watch resume review feedback (#1980) (@njhensley)
- 602c8db: fix(validator): roll back cluster-admin RBAC on prep failure (#2128) (@njhensley)
- 34503dc: fix(validators): multi-stage aiperf-bench build; bump aiperf to 0.11.0 (#1912) (@njhensley)
- 27c24a4: fix(validators): pass aiperf model via --model flag (#1939) (@njhensley)
- a222923: fix: reject recipes where both gpu-operator and gpu-operator-ocp are enabled (#2037) (@gat786)
- 3d4f77d: fix: resolve goconst and errorlint warnings across OCI, coverage, validators (#1916) (@andrewwhitecdw)
Other Tasks
- a28449e: chore(deps): Update dependency anchore/grype to v0.116.1 (#2058) (@github-actions[bot])
- 0f1379f: chore(deps): Update dependency anchore/syft to v1.50.0 (#1993) (@github-actions[bot])
- bcb7eeb: chore(deps): Update dependency awscli to v1.45.52 (#1905) (@github-actions[bot])
- 19b9c2f: chore(deps): Update dependency awscli to v1.45.57 (#1975) (@github-actions[bot])
- d709e47: chore(deps): Update dependency databus23/helm-diff to v3.15.11 (#2059) (@github-actions[bot])
- 6be0a54: chore(deps): Update dependency google/go-containerregistry to v0.21.8 (#2141) (@github-actions[bot])
- 911cad1: chore(deps): Update dependency goreleaser/goreleaser to v2.17.1 (#1991) (@github-actions[bot])
- e15d642: chore(deps): Update dependency sigstore/cosign to v3.1.2 (#1924) (@github-actions[bot])
- 34035f4: chore(deps): Update docker.io/library/busybox:1.38.0 Docker digest to dc2d74b (#1990) (@github-actions[bot])
- 7f1d5ed: chore(deps): Update google.golang.org/protobuf digest to 644d026 (#2107) (@github-actions[bot])
- 963aa9f: chore(deps): Update ministackorg/ministack Docker tag to v1.4.7 (#1965) (@github-actions[bot])
- ef2c805: chore(deps): Update module github.com/sigstore/sigstore-go to v1.3.0 (#1994) (@github-actions[bot])
- c13c4f4: chore(deps): Update python Docker tag to v3.14 (#1906) (@github-actions[bot])
- 5a97405: chore(deps): Update testing-tools (#1934) (@github-actions[bot])
- 5bbc6c3: chore(deps): Update testing-tools (#1992) (@github-actions[bot])
- 8db8286: chore(deps): Update testing-tools (#2011) (@github-actions[bot])
- c557a6b: chore(deps): Update testing-tools (#2108) (@github-actions[bot])
- 2827437: chore(deps): Update testing-tools (#2143) (@github-actions[bot])
- 9f327d5: chore(deps): Update ubuntu:26.04 Docker digest to 678c655 (#2010) (@github-actions[bot])
- 1ae5e61: chore(deps): regenerate stale THIRD_PARTY_NOTICES.md (#1926) (@njhensley)
- 3b9f2a8: chore: add rorajani to copy-pr-bot.yaml (#2014) (@rorajani)
- 337fd84: chore: add varmesh to copy-pr-bot.yaml (#2030) (@varmesh)
- a1884fd: chore: deps: bump actions/checkout from 7.0.0 to 7.0.1 (#1930) (@dependabot[bot])
- eec7dfc: chore: deps: bump actions/stale from 10.4.0 to 11.0.0 (#1931) (@dependabot[bot])
- 5f8b7d3: chore: deps: bump azure/login from 3.0.0 to 3.0.1 (#2106) (@dependabot[bot])
- 1dd58ed: chore: deps: bump azure/login from 3.0.0 to 3.0.1 (#2139) (@dependabot[bot])
- 45d0107: chore: deps: bump dorny/paths-filter from 4.0.2 to 4.0.3 (#2140) (@dependabot[bot])
- 97e3f8b: chore: deps: bump github.com/prometheus/client_golang from 1.24.0 to 1.24.1 (#1922) (@dependabot[bot])
- dccd746: chore: deps: bump github.com/sigstore/sigstore from 1.10.8 to 1.10.9 (#2008) (@dependabot[bot])
- dbf0557: chore: deps: bump github.com/sigstore/sigstore-go from 1.2.2 to 1.3.0 (#1986) (@dependabot[bot])
- 29ec662: chore: deps: bump github.com/sigstore/sigstore/pkg/signature/kms/aws from 1.10.8 to 1.10.9 (#2006) (@dependabot[bot])
- cdbc823: chore: deps: bump github.com/sigstore/sigstore/pkg/signature/kms/azure from 1.10.8 to 1.10.9 (#2005) (@dependabot[bot])
- f0dcc7e: chore: deps: bump github.com/sigstore/sigstore/pkg/signature/kms/gcp from 1.10.8 to 1.10.9 (#2007) (@dependabot[bot])
- 0df192d: chore: deps: bump github.com/sigstore/sigstore/pkg/signature/kms/hashivault from 1.10.8 to 1.10.9 (#2009) (@dependabot[bot])
- 9c5062b: chore: deps: bump github/codeql-action/analyze from 4.37.3 to 4.37.4 (#1987) (@dependabot[bot])
- 90ef73c: chore: deps: bump github/codeql-action/analyze from 4.37.4 to 4.37.5 (#2089) (@dependabot[bot])
- 7569957: chore: deps: bump github/codeql-action/init from 4.37.3 to 4.37.4 (#1988) (@dependabot[bot])
- 5e1f312: chore: deps: bump github/codeql-action/upload-sarif from 4.37.3 to 4.37.4 (#1989) (@dependabot[bot])
- a688419: chore: deps: bump github/codeql-action/upload-sarif from 4.37.4 to 4.37.5 (#2091) (@dependabot[bot])
- fb34898: chore: deps: bump renovatebot/github-action from 46.1.20 to 46.1.21 (#1923) (@dependabot[bot])
- 8eee3c5: chore: deps: bump renovatebot/github-action from 46.1.21 to 46.2.0 (#1964) (@dependabot[bot])
- 245b71c: chore: deps: bump renovatebot/github-action from 46.2.0 to 46.2.1 (#2092) (@dependabot[bot])
- b9615b7: chore: deps: bump the codeql-action group across 1 directory with 3 updates (#2105) (@dependabot[bot])
- 9cd73fb: chore: deps: update hashicorp/azurerm requirement from ~> 4.0 to ~> 5.0 in /infra/uat-azure-account (#1945) (@dependabot[bot])
- 314c475: chore: handle re-occurring scan vulnerability alerts (@mchmarny)
- 0b392a5: chore: pausing scans (@mchmarny)
- 744d057: chore: promote validators/chainsaw to pkg/chainsaw (#2032) (@mchmarny)
- 905bec6: chore: remove stray MOFED debug artifacts from repo root (#1955) (@njhensley)
- f4a525a: chore: skills cleanup (@mchmarny)
- 77ab263: ci(docs): block merges on parser-level MDX validation (#2145) (@mchmarny)
- 91c28ca: ci(e2e): drop the stale chainsaw binary from validator images (#2048) (@mchmarny)
- 1acf128: ci(sdk): gate client API compatibility (#2060) (@tjrasche)
- 74a7fe2: ci(stale): drop phantom priority/critical exempt label (#1925) (@yuanchen8911)
- d0a48ac: ci(supply-chain): build aiperf-bench without the BuildKit layer cache (#2087) (@lockwobr)
- 75b7e94: ci(supply-chain): daily re-verification of the latest release (#2069) (@lockwobr)
- 1ce6fcb: ci(uat): gate readiness on GPU-node census to close late-join race (#2131) (@njhensley)
- 40b8962: ci(uat): re-pin GKE actuator to v0.5.17 (#2072) (@njhensley)
- 5cee4fd: constraints: reject bare bang expressions (#2015) (@krtadev)
- 488f8ef: docs(api): make the AKS v2 curl example fail on HTTP errors (#2055) (@yuanchen8911)
- db3d198: docs(supply-chain): document monitoring your own signing identity (#1959) (@lockwobr)
- 1344108: docs(user): document the GKE device-plugin ownership prerequisite (#1998) (@yuanchen8911)
- 4872c8f: docs(validate): scope --fail-on-error; fix exit-code & CTRF-suite docs (#1889) (#1893) (@yuanchen8911)
- c150546: evidence(aks): h100-aks-ubuntu training + inference-dynamo recipe evidence (#1896) (@yuanchen8911)
- d4f7bef: feat(rekor-monitor): resumable identity scan to catch up large backlogs (#1929) (@lockwobr)
- f385574: feat(supply-chain): publish aiperf-bench third-party source as OCI referrer (#2073) (@lockwobr)
- 9a0f656: feat(supply-chain): publish release metadata as signed OCI referrers (#1982) (@lockwobr)
- da5dd5b: fix(aicr-cross-review skill): consumer-less demotion; resumable Codex lane (#2102) (@yuanchen8911)
- 4cf6cf4: fix(aicr-cross-review skill): visible Codex dispatch in progress view (#2125) (@yuanchen8911)
- f92d203: fix(aiperf-bench): drop pip from the runtime image (#2104) (@lockwobr)
- da2fd3f: fix(aiperf-bench): move runtime to NVIDIA distroless python (#1999) (@mchmarny)
- 7aa34f2: fix(recipe,client): reject duplicate ComponentRef names, omit disable… (#1917) (@mohityadav8)
- f01acf2: fix(rekor-monitor): correlate identity via release-workflow run history (#1903) (@lockwobr)
- 5a671a9: fix(rekor-monitor): raise scan-window timeout for large backlogs (#1900) (@lockwobr)
- dc1d4b3: fix(rekor-monitor): separate infra noise from security alerts (#1897) (@lockwobr)
- 2f82665: fix(testgrid-publish): reject null and zero-timestamp predicates (#2082) (@yuanchen8911)
- b7c83bb: gate: run readiness checks in-process, drop chainsaw binary (#2045) (@mchmarny)
- 2c3f187: recipes: add OCP-compatible components for cert-manager, prometheus-adapter, nvidia-dra-driver-gpu, and k8s-nim-operator (#2057) (@mohityadav8)
- dcf58a5: refactor(recipe): rename GKE gpuStack profile values (#2098) (@yuanchen8911)
- 41a80bf: revert(validators): pin aiperf-bench base back to python:3.13-slim (#1909) (@njhensley)
- e8b492e: test(bundler): cover argocd-helm OCP bundling and pin its error contract (#1950) (@yuanchen8911)
- edb12ce: test(sdk): pin full exported stability surface (#2031) (@tjrasche)
- 2075b7b: validate: fail-closed GKE device-plugin ownership check (#2000) (@yuanchen8911)