v0.17.0

This release continued our focus on scaling AICR validation across new services, intents, and new accelerators (GB300, metal3, GB200), as well as a new recipe-driven GPU allocation policy.

Highlights

New Accelerators & Services

• GB300 accelerator and metal3 service support
• GB200 Slurm recipe with IMEX compute-domain and channel conformance
• slinky/slurm-operator upgraded to v1.2.0 with configurable operator and webhook placement
• Akamai Cloud / Linode LKE provider-ID mapping

Expanding UAT Validation - The automated User Acceptance Testing (UAT) pipeline added Azure AKS to the AICR fleet with OIDC federation, a phase runner, and per-intent configs. A new reservation broker registry now gates GPU-pool acquisition, ensuring that scheduling lands GPU nodes only when needed. You can view evidence of every AICR UAT validation with a responsive corroborate dashboard UI and version-aware consensus at validation.aicr.run.

GPU Allocation Policy - GPU allocation became a first-class, recipe-driven contract. Recipes now default to the device-plugin allocation strategy in production, the validator enforces the recipe-configured allocation policy, and new capability inspection hardens allocation against misconfiguration.

NCCL & Fabric Validation - NCCL validation matured substantially: a GKE NCCL preflight with launcher-failure mitigations, AKS H100 all-reduce performance validation, recipe-driven benchmark applicability, cross-node fabric resource homogeneity checks, and rotation-proof bandwidth capture via pod termination messages. NCCL launcher diagnostics now surface to stdout for faster root-causing.

OS Tuning - AKS moved off the ib-node-config DaemonSet to nodewright tuning, a tuning-status table is now auto-generated, and nvidia-setup (v0.4.0) and nvidia-tuned (v0.3.1) were bumped.

Bundler

• Multi-tenant Argo deploy options
• argocd-helm children-only render via includeRootApp
• repoURL default baked into values.yaml at bundle push time

Supply Chain & Evidence - Keyless and KMS signing now default to Rekor v2, with Rekor v2 identity monitoring for the release signer and restored SLSA Build L3 image provenance in the reusable workflow. The verifier now fails closed on invalid attestations under a unified allowlist. Evidence collection gained mode-aware CNCF DRA-support and secure-access phases, propagates CNCF section failures, and renders divergent recipe version pins as BOM variants.

Other Improvements

• NFD bumped to v0.19.0, nvidia-dra-driver-gpu to v0.4.1 GA, dynamo-platform/runtime to v1.2.1

Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @eljohnson92, @hogeheer499-commits, @kaynetu, @lockwobr, @mohityadav8, @njhensley, @srao-nv, @tjrasche, @yuanchen8911, and @mchmarny.

Changelog

New Features

• ddea27d: feat(bom): render divergent recipe version pins as BOM variants (#1634) (@yuanchen8911)
• 038ae9c: feat(bundle): multi-tenant Argo deploy options (#1656) (@mchmarny)
• eb8684f: feat(bundler): argocd-helm children-only render via includeRootApp (#1724) (@mchmarny)
• fe55152: feat(bundler): bake repoURL default into values.yaml at bundle push t… (#1562) (@mohityadav8)
• 4fe9dc8: feat(ci): Rekor v2 identity monitoring for the release signer (#1727) (@lockwobr)
• a363ce4: feat(ci): add UAT reservation registry and broker helper (#1274) (#1559) (@njhensley)
• bdd91fc: feat(ci): add testgrid-publish workflow (TG5) (#1477) (@srao-nv)
• 79de60e: feat(ci): daytime human-access deployment scheduler (#1281) (#1587) (@njhensley)
• 83d9825: feat(ci): nightly UAT version matrix + versioned install (#1274) (#1579) (@njhensley)
• 55fd14f: feat(ci): per-intent + daytime-lifecycle UAT cluster acquisition (#1586) (@njhensley)
• e4fdf16: feat(ci): route UAT through the reservation broker (#1274) (#1569) (@njhensley)
• 1eb9140: feat(ci): surface superseded UAT runs (#1274) (#1583) (@njhensley)
• 1a0c96f: feat(corroborate): responsive dashboard UI + version-aware consensus (#1571) (@njhensley)
• 8f198eb: feat(evidence): mode-aware CNCF dra-support and secure-access collection (#1694) (@yuanchen8911)
• a0f0f2d: feat(infra): grant classic-ELB sweep permissions to UAT AWS role (#1617) (#1622) (@njhensley)
• a98c845: feat(recipe): add gb300 accelerator and metal3 service support (#1608) (@atif1996)
• d1aaafc: feat(recipes): bump dynamo-platform and runtime to 1.2.1 (#1581) (@yuanchen8911)
• 1805af0: feat(recipes): device-plugin GPU allocation production default (#1671) (@yuanchen8911)
• 1329b7f: feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning (#1698) (@ayuskauskas)
• d331244: feat(tuning): auto-generate nodewright tuning-status table (#1594) (@ayuskauskas)
• 4a762d2: feat(uat): Azure AKS UAT phase runner + intent configs (2/2) (#1722) (@mchmarny)
• 7082dd7: feat(uat): Azure AKS UAT — OIDC federation, dispatch, pipeline (1/2) (#1709) (@mchmarny)
• f37bba2: feat(uat): enroll azure-h100 in the nightly batch with [training] (#1726) (@mchmarny)
• 70bc0b3: feat(uat): gate AWS GPU pool with skyhook runtime-required taint (#1614) (@mchmarny)
• 23e97a7: feat(validator): AKS H100 NCCL all-reduce performance validation (#1695) (@yuanchen8911)
• 4608e13: feat(validator): GKE NCCL preflight + launcher failure mitigations (#1631) (@njhensley)
• 8fa9893: feat(validator): GPU allocation capability inspection and hardening (#1620) (@yuanchen8911)
• 3af23fa: feat(validator): recipe-driven NCCL benchmark applicability (#1719) (@njhensley)
• 526893f: feat: enforce recipe-configured GPU allocation policy (#1664) (@yuanchen8911)

Bug Fixes

• e8450d5: fix(bundler): emit disableValidation on CRD-dependent -post wrappers (#1687) (@mchmarny)
• a4be19a: fix(bundler): honor bundlers query param; driver-managed cleanup (#1606) (@mchmarny)
• ca6799f: fix(ci): build+push aiperf-bench image in UAT so inference-perf runs (#1638) (@njhensley)
• 7e41661: fix(ci): guarantee helm-diff keyring cleanup on all paths (#1578) (@njhensley)
• 01a2f7a: fix(ci): make helm v4.2.2 pin effective, verify helm-diff provenance (#1575) (@njhensley)
• a90193b: fix(ci): point aws-h100 UAT reservation at cr-0e16ad417f9a5bf69 (#1659) (@njhensley)
• 82c752e: fix(ci): reap orphaned LB ELB/SG before UAT VPC teardown (#1617) (#1618) (@njhensley)
• 6205cf0: fix(ci): restore SLSA Build L3 image provenance (reusable workflow) (#1558) (@mchmarny)
• dc12250: fix(ci): target aws-h100 UAT nodes at cr-0e16ad417f9a5bf69 (#1660) (@njhensley)
• f465b92: fix(ci): trim UAT nightly matrix, quiet run-watch, run 2 GPU nodes (#1592) (@njhensley)
• 713f263: fix(evidence): propagate CNCF section failures (#1730) (@hogeheer499-commits)
• 079e173: fix(kwok): deadline-derived sync-gate budgets + 18m tier timeout (#1708) (@ArangoGutierrez)
• 7c7a12b: fix(kwok): mirror registry and gitea images off Docker Hub (#1707) (@ArangoGutierrez)
• a0befce: fix(lke): add provider ID mapping for Akamai Cloud / Linode LKE (#1560) (@eljohnson92)
• 94b05c5: fix(recipe): align H100 NCCL bandwidth gate across services (#1565) (@yuanchen8911)
• 1ee6eeb: fix(recipe): reject ComponentRefs with incoherent type/field combinations (#1585) (@yuanchen8911)
• fbe0e21: fix(recipe): reject enabled Helm refs lacking a deployable, versioned chart primary (#1621) (@yuanchen8911)
• b178e7a: fix(recipe): reject unapplied ComponentRef.Patches; fix coherence godoc (#1589) (@yuanchen8911)
• 13d47cf: fix(recipes): align AKS training with GPU Operator-managed default (#1680) (@yuanchen8911)
• f07962a: fix(recipes): guard overlay version pins against registry defaults (#1572) (@mchmarny)
• 2358d65: fix(recipes): pin toolkit containerd config source to file on AKS (#1689) (@mchmarny)
• 22651e2: fix(recipes): raise ebs-csi node sidecar memory limits for p5 nodes (#1693) (@njhensley)
• ef12565: fix(recipes): raise nvidia-setup-full resources for EFA DKMS build (#1738) (@ayuskauskas)
• 7708b26: fix(uat): bump AKS actuator to v0.1.7 (60m GPU-pool timeouts) (@mchmarny)
• 05b9b54: fix(uat): drop gVNIC network from GKE GPU pool (TCPXO RxDM 7/8) (#1662) (@njhensley)
• 38a9805: fix(uat): mount writable az-context copy into AKS actuator (@mchmarny)
• f770711: fix(uat): propagate az_federated_relogin step failures (#1725) (@mchmarny)
• 8f03d6f: fix(uat): refresh federated login before AKS cluster connect (@mchmarny)
• e1bd87a: fix(uat): tolerate skyhook runtime-required taint in AWS snapshot (#1630) (@njhensley)
• 880bc33: fix(validator): capture NCCL bandwidth via termination message (rotation-proof) (#1713) (@njhensley)
• 43eb2a1: fix(validator): harden NCCL preflight helpers (codes, cancel, SA token) (#1641) (@njhensley)
• d47ce33: fix(validator): poll GPU readiness signals to ride through reboots (#1598) (@njhensley)
• bda381b: fix(validator): raise GKE NCCL launcher debug to INFO for root-causing (#1647) (@njhensley)
• f834162: fix(validator): reliably capture NCCL launcher logs via termination message (#1651) (@njhensley)
• df37d6b: fix(validator): retry NCCL TrainJob create on Trainer webhook cache race (#1670) (@njhensley)
• 2033738: fix(validator): skip disabled componentRefs in platform-health check (#1681) (@yuanchen8911)
• 0eda712: fix(validator): skip orphan pods in negative health assertions (#1643) (@njhensley)
• 37ba978: fix(validator): surface + re-read NCCL launcher log on bandwidth parse miss (#1691) (@njhensley)
• 32d082a: fix(validator): surface NCCL launcher failure diagnostics to stdout (#1640) (@njhensley)
• 82bfe55: fix(validator): treat "other" phase status as failing in gates (#1639) (@njhensley)
• 083efb1: fix(validator): validate NCCL fabric resource homogeneity across nodes (#1714) (@yuanchen8911)
• 2f55a40: fix(validator): warn on unknown check names; fix omitted-checks doc (#1721) (@njhensley)
• 480b7ff: fix(verifier): fail closed on invalid attestation; unify allowlist (#1607) (@mchmarny)

Other Tasks

• 52e7b78: CUJ2 inference E2E in the UAT runner + nightly both-intent (#1276) (#1597) (@njhensley)
• 2f61724: Fix agentgateway release pin (#1476) (@danehans)
• 21921f7: chore(deps): Update dependency anchore/grype to v0.115.0 (#1557) (@github-actions[bot])
• bf1f758: chore(deps): Update dependency awscli to v1.45.37 (#1605) (@github-actions[bot])
• 47547fa: chore(deps): Update dependency goreleaser/goreleaser to v2.17.0 (#1654) (@github-actions[bot])
• e133734: chore(deps): Update dependency helmfile/helmfile to v1.7.0 (#1655) (@github-actions[bot])
• 620a669: chore(deps): Update dependency ko-build/ko to v0.19.1 (#1637) (@github-actions[bot])
• a942993: chore(deps): Update ministackorg/ministack Docker tag to v1.3.69 (#1570) (@github-actions[bot])
• a101c36: chore(deps): Update ministackorg/ministack Docker tag to v1.3.71 (#1682) (@github-actions[bot])
• 041656b: chore(deps): Update module golang.org/x/sync to v0.22.0 (#1683) (@github-actions[bot])
• 09cb46e: chore(deps): Update nvkind digest to c570504 (#1556) (@github-actions[bot])
• 9398a3a: chore(deps): Update testing-tools (#1610) (@github-actions[bot])
• c928981: chore(deps): Update testing-tools (#1702) (@github-actions[bot])
• dab26d7: chore(deps): Update testing-tools (#1734) (@github-actions[bot])
• 256d656: chore(deps): Update ubuntu:24.04 Docker digest to 4fbb8e6 (#1591) (@github-actions[bot])
• b76936e: chore(deps): consolidate GitHub Actions updates (#1705) (@ArangoGutierrez)
• 671f783: chore(gate): sync Dockerfile GO_VERSION doc-default to 1.26.4 (@mchmarny)
• 5788e8e: chore(license): header hand-authored files, skip generated evidence pointers (#1582) (@yuanchen8911)
• 2199c7b: chore(lint): queue golangci-lint runs instead of failing on the lock (#1635) (@yuanchen8911)
• a5a19a9: chore(recipes): bump NFD to v0.19.0 (#1711) (@ArangoGutierrez)
• 79a0ff2: chore(recipes): bump nvidia-dra-driver-gpu to 0.4.1 GA (#1648) (@yuanchen8911)
• ef3fe8f: chore(recipes): bump nvidia-tuned to 0.3.1 (#1663) (@ayuskauskas)
• 5132b5e: chore(scan): suppress unfixed oras-go tar-extraction CVE (GHSA-fxhp-mv3v-67qp) (#1593) (@mchmarny)
• 6ed3d7d: chore(tuning): bump nvidia-setup to 0.4.0 for EKS nodes (#1599) (@ayuskauskas)
• 7796d59: chore(vex): suppress chainsaw-binary HIGH CVEs on aicr-gate (@mchmarny)
• 527c015: chore: added tjrasche to copy-pr-bot.yaml (#1706) (@tjrasche)
• bbf3c32: chore: deps: bump actions/labeler from 6.1.0 to 6.2.0 (#1675) (@dependabot[bot])
• 1bf4f84: chore: deps: bump aws-actions/configure-aws-credentials from 6.2.1 to 6.2.2 (#1653) (@dependabot[bot])
• 86c4ec2: chore: deps: bump docker/build-push-action from 7.2.0 to 7.3.0 (#1590) (@dependabot[bot])
• 513136a: chore: deps: bump docker/setup-buildx-action from 4.1.0 to 4.2.0 (#1600) (@dependabot[bot])
• 9380913: chore: deps: bump dorny/paths-filter from 4.0.1 to 4.0.2 (#1604) (@dependabot[bot])
• 108c8bf: chore: deps: bump github.com/sigstore/sigstore-go from 1.2.1 to 1.2.2 (#1633) (@dependabot[bot])
• 68fee91: chore: deps: bump github/codeql-action/analyze from 4.36.2 to 4.36.3 (#1602) (@dependabot[bot])
• 2d55710: chore: deps: bump github/codeql-action/analyze from 4.36.3 to 4.37.0 (#1674) (@dependabot[bot])
• 4bad60b: chore: deps: bump github/codeql-action/init from 4.36.2 to 4.36.3 (#1603) (@dependabot[bot])
• 59a67ec: chore: deps: bump github/codeql-action/init from 4.36.3 to 4.37.0 (#1676) (@dependabot[bot])
• f262ac6: chore: deps: bump github/codeql-action/upload-sarif from 4.36.2 to 4.36.3 (#1601) (@dependabot[bot])
• 1be740f: chore: deps: bump github/codeql-action/upload-sarif from 4.36.3 to 4.37.0 (#1677) (@dependabot[bot])
• 8be1cfc: chore: deps: bump goreleaser/goreleaser-action from 7.2.2 to 7.2.3 (#1555) (@dependabot[bot])
• 8068cd4: chore: deps: bump oras.land/oras-go/v2 from 2.6.1 to 2.6.2 (#1732) (@dependabot[bot])
• 7ab862c: chore: deps: bump renovatebot/github-action from 46.1.17 to 46.1.18 (#1609) (@dependabot[bot])
• a5c5ec9: chore: deps: bump renovatebot/github-action from 46.1.18 to 46.1.19 (#1733) (@dependabot[bot])
• c710424: chore: deps: bump the golang-x group with 3 updates (#1673) (@dependabot[bot])
• 3086ab0: chore: deps: bump the kubernetes group with 2 updates (#1731) (@dependabot[bot])
• 4f2a07d: chore: global dep update (@mchmarny)
• bc3ea17: chore: global dep upgrade (@mchmarny)
• 471a57c: ci(evidence): make sign workflow commit-back GitHub-Verified (#1720) (@njhensley)
• 489ff0b: ci(uat): disable serve phase on GCP+AWS (exclude vLLM from UAT) (#1645) (@njhensley)
• 7e3b6cd: ci(uat): raise readiness gate budget to 60m for nodewright tuning (#1665) (@njhensley)
• d26c528: ci(uat-aws): refresh AWS creds per stage across the UAT phase chain (#1657) (@njhensley)
• 65a205e: docs(#1533): fix bundle-layout, validator, and GPU constraint-path drift (#1561) (@yuanchen8911)
• 58f616b: docs(demos): add dynamic-values and private-signing test walkthroughs (#1548) (@lockwobr)
• 9c7e782: docs(user): add AICRConfig configuration file reference (@mchmarny)
• 755c37a: evidence(aks): h100 training + inference-dynamo recipe evidence (#1699) (@yuanchen8911)
• 1d3c2f4: feat(slinky-slurm): add GB200 slurm recipe, IMEX compute domain and channel conformance (#1539) (@kaynetu)
• e98cdac: feat(slinky-slurm): upgrade slinky/slurm-operator to 1.2.0 and support operator and webhook placement (#1624) (@kaynetu)
• b39d4a0: feat(supply-chain): default keyless and KMS signing to Rekor v2 (#1679) (@lockwobr)
• cd83605: fix(supply-chain): VEX CVE-2026-15308 for aiperf-bench (#1736) (@mchmarny)
• bbe14ac: fix(supply-chain): match gate VEX to grype primary IDs; Go 1.26.5 (#1684) (@mchmarny)
• 3e8e64d: refactor(recipe): unify the two topological-sort paths (#1466) (#1728) (@lockwobr)
• ffe40ab: test(bom): harden BOM version-freshness gate (#1576) (#1580) (@yuanchen8911)
• 6188b9c: test(corroborate): lock version ordering with a semver comparator test (#1573) (@njhensley)