Recipe contribution workflow with CI evidence gate

[mchmarny]

Parent: #750
Depends on: aicr verify-evidence command (filed alongside)

Summary

Document the end-to-end community recipe contribution workflow and add a CI gate that runs aicr verify-evidence on every PR touching recipes/overlays/ (or other recipe-defining paths). The gate posts a Markdown verification summary as a structured review comment so a maintainer can approve a recipe for hardware they don't have.

Workflow (contributor view)

1. Author a new overlay or modify an existing one.
2. Apply it to a real cluster matching the recipe's criteria.
3. Run aicr snapshot on that cluster.
4. Run aicr validate --recipe <r> --snapshot <s> --emit-evidence ./evidence/.
5. Attach the resulting evidence-bundle.tar.gz to the PR (uploaded artifact, release asset, or large-file attachment — exact mechanism TBD per repo policy).
6. Open the PR. CI verifies the bundle and posts a summary. Maintainers review the bundle alongside the diff.

Workflow (maintainer / CI view)

1. CI detects a recipe-touching PR.
2. CI fetches the bundle (path TBD — likely evidence/<recipe>.tar.gz checked into the PR, or a separate signed-artifact link).
3. CI runs aicr verify-evidence <bundle> with expected issuer/identity constraints set to public OIDC providers AICR trusts (initially GitHub Actions OIDC).
4. CI posts the Markdown summary as a review comment. A failing bundle blocks merge via a required check.

Deliverables

• .github/PULL_REQUEST_TEMPLATE/recipe.md — recipe-specific PR template requiring evidence bundle attachment, contributor declaration of test environment, and link to relevant criteria documentation.
• .github/workflows/recipe-evidence.yaml — workflow that runs on pull_request events touching recipes/overlays/** (and other recipe paths). Runs aicr verify-evidence, posts a sticky review comment, sets a required check status.
• CONTRIBUTING.md update — new section: "Contributing a recipe for hardware AICR maintainers can't access" — walks through the contributor workflow with a worked example and an FAQ (what if I don't have an OIDC identity, what if my test failed but I think the failure is environmental, etc.).
• docs/integrator/recipe-development.md update — link from the existing recipe authoring docs to the contribution workflow page.

Acceptance criteria for the gate

• A PR adding a recipe without an evidence bundle fails the required check.
• A PR with an invalid bundle (bad signature, schema mismatch, fingerprint mismatch) fails the required check.
• A PR with a valid bundle whose evidence shows failing validator checks does not auto-fail the required check — it posts the summary and lets the maintainer decide. Failed-evidence PRs are sometimes intentional (e.g., known-issue documentation, work in progress).
• Required-check status, comment text, and exit codes are deterministic so contributors can self-debug.

Open questions

• Bundle delivery mechanism. Embedding tar.gz in the PR diff is awkward. Options: (a) GitHub Releases asset on the contributor's fork, (b) cosign-signed artifact in a registry referenced from the PR body, (c) checked-in under recipes/evidence/<recipe>/. We should pick one and document it. Starting with (c) is the simplest path for v1.
• Scope of "recipe-touching." Initially recipes/overlays/**. Should mixin or component changes also require evidence? Probably yes once the workflow is mature; out of scope for v1.
• Multiple bundles per PR. If a PR adds a recipe variant covering several cloud / accelerator combinations, must each be evidence-backed? Yes — one bundle per criteria combination — but the gate UX should handle the n-way case cleanly.

Success criteria

• A worked example PR exists in the docs (or as a real PR-as-fixture) that walks through contributor → CI → maintainer.
• A maintainer with no access to a particular hardware combination can confidently approve a PR for it on the strength of the bundle and the CI summary.
• The CI gate is robust: clear failure messages, no false negatives on bundle parsing, friendly to first-time contributors.

Out of scope

• Rewarding contributors with specific badges/recognition (different policy work).
• Hosting bundles outside the PR lifecycle for long-term retrieval (cosign + Rekor handle the durable record).

[yuanchen8911]

Related to #970 (CI guard for per-intent validation phase coverage on recipe overlays). The two are complementary, not duplicates:

• Add CI guard to enforce per-intent validation phase coverage on recipe overlays #970 is a static gate — walks each overlay's base: chain and asserts the resolved validation: block meets the per-intent phase floor (D + C, or D + P + C). Pure YAML lint, no hardware, runs on every PR.
• This issue (Recipe contribution workflow with CI evidence gate #751) is a runtime gate — requires the contributor to attach a signed evidence-bundle.tar.gz proving the declared validators were actually executed on real hardware matching the recipe's criteria.

#970 is a precondition for #751 to be meaningful: if an overlay only declares conformance:, then aicr validate --emit-evidence only runs conformance checks, and the resulting bundle is technically valid but proves nothing about deployment health or NCCL bandwidth. #970 forces the declaration to be complete; #751 forces it to actually run.