v0.16.0

This release focuses on portable recipe validation evidence, the API domain migration to aicr.run, network topology discovery, and broader conformance validation.

Highlights

Evidence & Trust - Evidence handling matured into an end-to-end signing and verification workflow. aicr evidence sign signs an already-pushed bundle, push gains a --no-sign mode, and pointers are auto-signed on push under a per-source pointer contract with a signer allowlist. A new fork-based signing workflow lets external contributors publish signed evidence, the evidence gate now partitions protected vs. other sources with a component-scoped cascade, and aicr verify --trust-root validates against a private Sigstore trust root. A Rekor identity + consistency monitor watches the transparency log, evidence bundles minimize sensitive content, and the updated ingest path automates bundle verification.

Evidence Dashboard & Corroboration - New corroborate consensus model rolls per-source results into a unified verdict (with PARTIAL handling for incomplete phase coverage) and generates a dashboard published to GitHub Pages at validation.aicr.run. This is initial work using only 1st party data to validate that flow. More to come here in next release.

aicr.run API Domain Migration - The artifact API domain migrated to aicr.run, bumping the contract to v1alpha2/v1beta2 (see ADR-013). aicr validate now warns on aicr version skew across inputs and enforces artifact apiVersion compatibility so mismatched recipes and snapshots fail closed.

Network Topology Discovery - A new NetworkTopology measurement type and collector integrate the l8k library based on NVIDIA k8s-launch-kit project to capture cluster network fabric, enabling cross-repo consumption via Subtype.Items.

New Validators & Recipes

• L40S accelerator support on OKE (Oracle Kubernetes Engine), with a new ol (Oracle Linux) OS value; OKE overlays now require explicit OS selection (no os: any fallback)
• slinky-slurm conformance validator with a Critical User Journey (CUJ) demo
• RoCE NET variant for nccl-all-reduce-bw
• Prometheus/Alertmanager StatefulSet readiness in the kube-prometheus-stack check
• OpenShift Container Platform (OCP) added as a supported --service value

Bundler

• Structured deployer output with color and step headers
• Private-by-default agentgateway inference-gateway exposure
• Dynamic toleration injection for pre-built OCI bundles

Other Improvements

• inference-perf frontend routing switched to least-loaded
• New kwok argocd-git/gitea lanes support in test flow
• Deployment ordering now honors enabled: false

Thanks to @almaslennikov, @atif1996, @haarchri, @kaynetu, @lockwobr, @mohityadav8, @njhensley, @rsd-darshan, @srao-nv, @yuanchen8911, and @mchmarny.

Changelog

New Features

• ddc4dde: feat(bundler): private-by-default agentgateway inference-gateway exposure (#1450) (@yuanchen8911)
• e9aba0d: feat(bundler): structured deploy.sh output with color and step headers (#1393) (@mohityadav8)
• f7545fd: feat(bundler): support dynamic toleration injection for pre-built OCI bundles (#1471) (@mohityadav8)
• 70b743c: feat(ci): GP5 GitHub Pages publish for evidence dashboard (#1504) (@njhensley)
• b3b4eee: feat(ci): enforce single-owner model for /assign and self-only /unassign (@mchmarny)
• 9e6374b: feat(ci): self-serve issue assignment via /assign and /unassign comments (@mchmarny)
• f144ddd: feat(corroborate): consensus model and dashboard generator (#1483) (@njhensley)
• e878161: feat(evidence): --no-sign push and pending/structured-cause verify (#1445) (@mchmarny)
• 14e347a: feat(evidence): GP2 ingest — verify bundles to source-keyed GCS (#1484) (@njhensley)
• a758de2: feat(evidence): aicr evidence sign — sign an existing pushed bundle (#1446) (@mchmarny)
• db3ad73: feat(evidence): auto-sign evidence pointers on push (#1454) (@mchmarny)
• 480b306: feat(evidence): fork-based signing workflow + publishing docs (#1451) (@mchmarny)
• 03b14d9: feat(evidence): minimize sensitive content in evidence bundles (#1414) (@njhensley)
• a2e0690: feat(evidence): per-source pointer contract + signer allowlist (#1401) (#1485) (@njhensley)
• 9be382b: feat(evidence): regenerate signed demo evidence under aicr.run (#1509) (@yuanchen8911)
• da21e22: feat(measurement): NetworkTopology type + Subtype.Items for cross-repo l8k integration (#1417) (@almaslennikov)
• dc23abf: feat(network): NetworkTopology collector + l8k library integration (#1474) (@almaslennikov)
• c7f4373: feat(recipe): add L40S accelerator support (#1510) (@atif1996)
• a1b6e20: feat(recipe): shared coordinate mapping + taxonomy spec (TG6) (#1409) (@mchmarny)
• caecdd0: feat(recipes): add Prometheus/Alertmanager StatefulSet readiness to kube-prometheus-stack check (#1497) (@mohityadav8)
• ffe7522: feat(testgrid): add testgrid-publish CLI tool (TG2) (#1447) (@srao-nv)
• 9a741e6: feat(validate): enforce artifact apiVersion compatibility (#1387) (@mchmarny)
• 68eee88: feat(validate): warn on aicr version skew across inputs (#1386) (@mchmarny)
• d57f7ac: feat(validation): switch inference-perf frontend routing to least-loaded (#1399) (@yuanchen8911)
• 381ba2a: feat(validators): RoCE NET variant for nccl-all-reduce-bw (#1428) (@yuanchen8911)
• f0ddcd0: feat(verify): private Sigstore trust root (aicr verify --trust-root) (#1449) (@lockwobr)
• 9be847e: feat: migrate API domain to aicr.run, bump v1alpha2/v1beta2 (#1499) (@mchmarny)

Bug Fixes

• 732a8dd: fix(attestation): retry ambient OIDC token acquisition (#1389) (@mchmarny)
• 24edd51: fix(ci): correct docs-only detection in merge gate (#1381) (@mchmarny)
• eebd348: fix(ci): link digest-pinned evidence ref in UAT summaries (#1498) (@njhensley)
• eea2a81: fix(cli): validate/recipe UX papercuts (#1383 items 1-7) (#1391) (@mchmarny)
• fc5e814: fix(corroborate): PARTIAL rollup for incomplete phase coverage (#1503) (@njhensley)
• ab022b2: fix(corroborate): escape boot() error + fail-closed meta schema gate (#1500) (@njhensley)
• fd8ba27: fix(evidence): key-constrain all redaction allowlist subtypes (#1419) (@njhensley)
• e19674c: fix(evidence): make unsigned-commit CI-sign flow coherent (#1530) (#1538) (@njhensley)
• 7839464: fix(evidence): relocate idempotency, strict digest, package docs (#1543) (@njhensley)
• 9cd3fb7: fix(recipe): fix deployment ordering to honor enabled:false (#1465) (@yuanchen8911)
• 7f7ae67: fix(test): argocd-sync gate: use all-semantics, handle health-gated apps (#1397) (@rsd-darshan)
• a0f2d09: fix(uat): bind Prometheus PVC to cluster default StorageClass (#1455) (@njhensley)
• 026804e: fix(uat): gate install on the deployment validation phase (#1439) (@njhensley)
• ebbd322: fix(uat): gate on Prometheus readiness; retry conformance metric APIs (#1452) (@njhensley)
• 6663eb5: fix(uat): gate validate on cluster convergence (readiness wait + failFast) (#1426) (@njhensley)
• c4f0872: fix(uat): run all validate phases to gate on GPU readiness (#1416) (@njhensley)
• 58721c5: fix(uat): wait for nodewright tuning before validate readiness gate (#1429) (@njhensley)
• 6ac2cfc: fix(validator): drop single-shot ClusterPolicy check from expected-resources (#1495) (@njhensley)
• 410e31a: fix(validator): pod-autoscaling passes on external-metric HPA path (DRA clusters) (#1408) (@yuanchen8911)
• 4b24b7d: fix(validator): wait for KAI deployments ready in gang-scheduling check (#1514) (@mchmarny)
• a74fe77: fix: harden timeout/false-PASS handling, evidence integrity, CI gaps (#1511) (@mchmarny)

Other Tasks

• bf294ed: Add OpenShift Container Platform (OCP) as a service type (#1380) (@kaponco)
• 3307169: Delete .github/workflows/devtrace.yaml (@mchmarny)
• e927f39: chore (validator): pin Trainer self-install JobSet image to promoted registry (#1440) (@yuanchen8911)
• b391a17: chore(ci): add kaynetu to copy-pr-bot trustees (#1412) (@kaynetu)
• 9c2d362: chore(deps): Update dependency awscli to v1.45.34 (#1415) (@github-actions[bot])
• 6011f49: chore(deps): Update dependency google/go-containerregistry to v0.21.7 (#1469) (@github-actions[bot])
• 44f912b: chore(deps): Update dependency hauler-dev/hauler to v2 (#1482) (@github-actions[bot])
• dbd698a: chore(deps): Update gitea/gitea Docker tag to v1.26.3 (#1444) (@github-actions[bot])
• cafb630: chore(deps): Update k8s.io/utils digest to a95e086 (#1468) (@github-actions[bot])
• fc24c42: chore(deps): Update testing-tools (#1378) (@github-actions[bot])
• 6d147e2: chore(deps): Update testing-tools (#1396) (@github-actions[bot])
• 71aea47: chore(deps): Update testing-tools (#1470) (@github-actions[bot])
• a5583f6: chore(deps): Update testing-tools (#1481) (@github-actions[bot])
• 0cc404c: chore(deps): Update testing-tools (#1522) (@github-actions[bot])
• 9eb3ce9: chore(evidence): refresh gb200-eks-ubuntu-training attestation (#1427) (@yuanchen8911)
• f330078: chore(fern): register v0.15.0 (#1370) (@github-actions[bot])
• 702f9ee: chore(fern): register v0.16.0 (#1546) (@github-actions[bot])
• 09039e8: chore(recipes): bump aws-efa to v0.5.29, align image tag to appVersion (#1418) (@yuanchen8911)
• ec48bbd: chore(recipes): update nodewright to v0.17.1 (fixes immutable selector) (#1502) (@lockwobr)
• 872f851: chore: deps: bump actions/cache from 5.0.5 to 6.0.0 (#1442) (@dependabot[bot])
• 0ef1bce: chore: deps: bump actions/cache from 6.0.0 to 6.1.0 (#1518) (@dependabot[bot])
• 822477e: chore: deps: bump actions/checkout from 6.0.3 to 7.0.0 (#1395) (@dependabot[bot])
• 8c4c46f: chore: deps: bump actions/setup-go from 6.4.0 to 6.5.0 (#1441) (@dependabot[bot])
• 932c794: chore: deps: bump actions/setup-go from 6.4.0 to 6.5.0 (#1521) (@dependabot[bot])
• 8faf89e: chore: deps: bump aws-actions/configure-aws-credentials from 6.2.0 to 6.2.1 (#1519) (@dependabot[bot])
• 3b285b0: chore: deps: bump azure/setup-helm from 5.0.0 to 5.0.1 (#1443) (@dependabot[bot])
• bbed4d4: chore: deps: bump renovatebot/github-action from 46.1.15 to 46.1.16 (#1398) (@dependabot[bot])
• 23fb5e5: chore: deps: bump renovatebot/github-action from 46.1.16 to 46.1.17 (#1520) (@dependabot[bot])
• b40188f: chore: deps: bump thingzio/devtrace-action from b35e96e42494859cfbab25d17499efb5cb30a59d to 167f053a160c2ddf5cf1fcd288bbb83e1da28930 (#1377) (@dependabot[bot])
• f2f8ed9: ci(kwok): add argocd-git lane with in-cluster gitea (#1390) (@haarchri)
• 9aeac26: ci(pages): pin custom domain via CNAME in Pages artifact (#1508) (@mchmarny)
• eac5eee: ci(supply-chain): add Rekor identity + consistency monitor (#1480) (@lockwobr)
• 551bc44: ci(uat): build snapshot-agent image from PR source (#1506) (@njhensley)
• 4e6824e: ci(uat): refresh cloud creds before teardown to prevent GPU leak (#1456) (@njhensley)
• 9b4a664: docs(adr): ADR-013 migrate artifact API domain to aicr.run (#1492) (@mchmarny)
• 77c6b6d: docs(agents): correct coverage floor to 75% (matches .settings.yaml) (#1552) (@yuanchen8911)
• 173e81c: docs(cli): explain recipe input modes (snapshot vs criteria) and why (#1411) (@yuanchen8911)
• b2364d4: docs(cli): list any as a supported --service value (#1501) (@njhensley)
• b2ca362: docs(contributing): single commit-signing rule (-s -S) for all contributors (#1494) (@mchmarny)
• 0c2ebbe: docs(contributor): document exposing scheduling knobs without new flags (#1379) (@mchmarny)
• 9bd2268: docs(evidence): add H100 GKE COS and GB200 EKS Ubuntu training attestations (#1368) (@atif1996)
• b0dd96e: docs(evidence): clarify pointer-contract gate verifies claimed signer (#1544) (@njhensley)
• 3eb13ff: docs(evidence): fix stale flat-pointer discovery claim in evidence-ingest (#1545) (@yuanchen8911)
• 6fc3444: docs(recipes): note b200-any-training retirement (#1053) in wildcard guide (#1475) (@yuanchen8911)
• 596ba97: docs(testgrid): public TestGrid page + Fern nav (TG6) (#1463) (@mchmarny)
• d25244b: feat(evidence-gate): protected/other partition + component-scoped cascade (#1448) (@mchmarny)
• 15c16f3: feat(slinky-slurm): add conformance validator and CUJ demo (#1394) (@kaynetu)
• 1391d84: fix(evidence,validator): correct catalog apiVersion examples, drop unverifiable demo evidence (#1499 follow-up) (#1507) (@yuanchen8911)
• c1d2468: refactor(client,cli): move CTRF merge + evidence emit behind facade (#1512) (@mchmarny)
• e19cb68: refactor(uat): stable digest-addressed OCI ref for evidence push (#1479) (@njhensley)
• 12b86df: test(uat): allow NATS ingress on UAT AWS/GCP clusters (#1376) (@mchmarny)