feat(ci): dispatch eval execution for fork PR secret access

[mrizzi]

Summary

• Split eval-pr.yml into a two-stage workflow_run dispatch pattern so fork PRs can run evals with secret access
• Stage 1 (eval-pr.yml) discovers changed skills and uploads metadata artifact — no secrets needed
• Stage 2 (eval-pr-run.yml) downloads artifact, checks collaborator trust via GitHub API, gates untrusted authors behind a protected environment, then runs evals and posts the PR review
• Security-hardened: PR identity derived from GitHub API (not artifact), skills validated against allowlist, no ${{ }} interpolation of untrusted values in shell, --verbose removed

Context

PR #90 from a fork failed because GitHub does not pass secrets to workflows triggered from forked repositories. This PR implements the officially documented workflow_run pattern to solve this.

Trust Model

• Trusted (write/admin collaborators): evals run automatically
• Untrusted (external contributors): require manual approval via eval-protected environment

Post-Merge Setup

Create the eval-protected environment in repo settings:

1. Settings > Environments > New environment: eval-protected
2. Add required reviewers

Test Plan

• Verify eval-pr.yml triggers on a PR that modifies skills/evals files
• Verify eval-pr-run.yml fires after eval-pr.yml completes
• Verify trusted collaborator PR auto-runs evals (gate skipped)
• Verify fork PR from non-collaborator triggers the approval gate
• Verify branches: [main] filter does not prevent Stage 2 from triggering — remove in follow-up if it does
• Verify eval results are posted as a PR review

🤖 Generated with Claude Code

Summary by Sourcery

Introduce a two-stage eval PR CI pipeline that supports forked PRs with secure secret access and collaborator-based trust gating.

Enhancements:

• Refactor eval-pr.yml into a discovery-only workflow that detects changed skills, captures PR metadata, and uploads it as an artifact without using secrets.
• Add a new eval-pr-run.yml workflow triggered via workflow_run that downloads metadata, derives PR identity from the GitHub API, enforces a protected environment gate for untrusted authors, and runs evals with secret-backed access before posting PR reviews.
• Strengthen CI security by validating skill names from artifacts, avoiding interpolation of untrusted data, and centralizing trust checks on GitHub collaborator permissions.

Documentation:

• Add a new design spec detailing the eval PR fork dispatch architecture and trust model, and cross-reference it from the existing eval skills CI workflow spec.

[sourcery-ai]

Reviewer's Guide

Splits the eval PR GitHub Actions workflow into a two-stage workflow_run dispatch flow so eval discovery runs safely on fork PRs without secrets, while a new workflow with trust-gating executes evals with secret access and posts reviews; also adds design docs describing the architecture and links them from the existing eval CI spec.

Sequence diagram for eval PR two-stage workflow_run dispatch

sequenceDiagram
    actor Contributor
    participant GitHub_PR as GitHub_PR
    participant Eval_PR as Eval_PR_Workflow
    participant Artifact as Artifact_Storage
    participant Eval_PR_Run as Eval_PR_Run_Workflow
    participant GitHub_API as GitHub_API
    participant Eval_Protected as Eval_Protected_Environment

    Contributor->>GitHub_PR: Open_or_update_pull_request
    GitHub_PR-->>Eval_PR: Trigger_pull_request_event

    Eval_PR->>Eval_PR: Checkout_repository
    Eval_PR->>Eval_PR: Discover_changed_skills
    Eval_PR->>Eval_PR: Write_PR_metadata_json
    Eval_PR->>Artifact: Upload_eval_pr_metadata_artifact
    Eval_PR-->>Eval_PR_Run: Emit_workflow_run_completed_event

    Eval_PR_Run->>Artifact: Download_eval_pr_metadata
    Eval_PR_Run->>GitHub_API: Get_pull_requests_for_head_sha
    GitHub_API-->>Eval_PR_Run: PR_identity_and_author
    Eval_PR_Run->>GitHub_API: Get_collaborator_permission_level
    GitHub_API-->>Eval_PR_Run: Permission_level

    alt Trusted_collaborator
        Eval_PR_Run->>Eval_PR_Run: Set_trusted_true
        Eval_PR_Run->>Eval_PR_Run: Skip_eval_protected_gate
    else Untrusted_contributor
        Eval_PR_Run->>Eval_Protected: Request_environment_approval
        Eval_Protected-->>Eval_PR_Run: Approval_granted
    end

    Eval_PR_Run->>Eval_PR_Run: Checkout_PR_merge_commit
    Eval_PR_Run->>Eval_PR_Run: Authenticate_to_Google_Cloud
    Eval_PR_Run->>Eval_PR_Run: Install_Claude_Code
    Eval_PR_Run->>Eval_PR_Run: Run_evals_for_each_skill
    Eval_PR_Run->>GitHub_PR: Post_or_update_eval_results_review

Loading

Flow diagram for collaborator trust gating logic in eval PR run workflow

flowchart TD
    Start[Start_discover_job] --> CheckMetadata[Load_eval_pr_metadata]
    CheckMetadata --> ValidateSkills[Validate_skills_format_regex]
    ValidateSkills -->|invalid| FailSkills[Set_workflow_failed_invalid_skills]
    ValidateSkills -->|valid| ResolvePR[Resolve_PR_by_head_sha_via_GitHub_API]

    ResolvePR -->|no_PR_found| FailPR[Set_workflow_failed_no_PR_for_commit]
    ResolvePR -->|PR_found| DeriveIdentity[Derive_PR_number_and_author]

    DeriveIdentity --> CheckPerm[Get_collaborator_permission_level]

    CheckPerm -->|admin_or_write| Trusted[Set_trusted_true]
    CheckPerm -->|other_or_not_collaborator| Untrusted[Set_trusted_false]

    Trusted --> SkillsCheckTrusted[Skills_non_empty]
    Untrusted --> SkillsCheckUntrusted[Skills_non_empty]

    SkillsCheckTrusted -->|skills_empty| EndNoSkillsTrusted[End_no_evals_run]
    SkillsCheckTrusted -->|skills_present| RunEvalsTrusted[Run_evals_without_gate]

    SkillsCheckUntrusted -->|skills_empty| EndNoSkillsUntrusted[End_no_evals_run]
    SkillsCheckUntrusted -->|skills_present| Gate[Require_eval_protected_environment_approval]

    Gate -->|approved| RunEvalsUntrusted[Run_evals_after_gate]

    RunEvalsTrusted --> EndTrusted[Post_eval_results_review]
    RunEvalsUntrusted --> EndUntrusted[Post_eval_results_review]

    FailSkills --> EndFailSkills[Workflow_failed]
    FailPR --> EndFailPR[Workflow_failed]

Loading

File-Level Changes

Change	Details	Files
Convert eval-pr.yml into a discovery-only workflow that emits PR metadata as an artifact for downstream execution.	Remove in-workflow eval execution, GCP authentication, Claude Code installation, and PR review posting steps from eval-pr.yml. Rename the main job to discover, restrict permissions to contents: read, and keep only the changed-skills detection logic. Add steps to serialize discovered skills and PR context into eval-pr-metadata.json and upload it as an eval-pr-metadata artifact using actions/upload-artifact@v4. Update file header comments and spec reference to point to the new fork-dispatch design document.	.github/workflows/eval-pr.yml
Introduce eval-pr-run.yml, a workflow_run-driven eval runner that downloads PR metadata, enforces a collaborator-based trust model, gates untrusted authors via a protected environment, and runs evals with secrets.	Define Eval PR Run workflow triggered by workflow_run of Eval PR with appropriate permissions (contents read, pull-requests write, actions read). Implement a discover job that downloads eval-pr-metadata, validates the skills field, resolves the PR number and author via GitHub APIs, and computes a trusted flag using collaborator permission levels. Add a gate job bound to eval-protected environment that only runs for untrusted authors with non-empty skills, requiring manual approval before eval execution. Implement a run-evals job that checks gating conditions, checks out the PR merge ref, authenticates to GCP, installs Claude Code, runs evals for each skill without interpolating untrusted data into shell, and posts/updates a structured PR review with eval results via actions/github-script@v7.	.github/workflows/eval-pr-run.yml
Document the fork dispatch design and cross-link it from the existing eval CI spec.	Add a cross-reference in the eval skills CI workflow spec to the new Eval PR Fork Dispatch design document for fork PR secret handling. Create a detailed Eval PR Fork Dispatch design spec describing the two-stage architecture, trust model, security rationale, and setup requirements. Add an implementation plan document under docs/superpowers/plans outlining the step-by-step changes for splitting eval-pr.yml and introducing eval-pr-run.yml.	docs/specs/2026-04-21-eval-skills-ci-workflow-design.md docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 2 issues, and left some high level feedback:

• In eval-pr-run.yml's discover job you currently trust the skills list from the artifact (with a regex check); for stronger defense-in-depth consider recomputing the changed skills from the checked-out repo in Stage 2 instead of propagating them via a user-controllable artifact.
• The workflow_run trigger in eval-pr-run.yml is additionally filtered by branches: [main]; since workflow_run already fires from runs of the Eval PR workflow, you may want to confirm this branch filter is actually needed and doesn’t prevent legitimate PR runs (e.g., if Eval PR is ever triggered from other base branches).

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In `eval-pr-run.yml`'s `discover` job you currently trust the `skills` list from the artifact (with a regex check); for stronger defense-in-depth consider recomputing the changed skills from the checked-out repo in Stage 2 instead of propagating them via a user-controllable artifact.
- The `workflow_run` trigger in `eval-pr-run.yml` is additionally filtered by `branches: [main]`; since `workflow_run` already fires from runs of the `Eval PR` workflow, you may want to confirm this branch filter is actually needed and doesn’t prevent legitimate PR runs (e.g., if `Eval PR` is ever triggered from other base branches).

## Individual Comments

### Comment 1
<location path=".github/workflows/eval-pr-run.yml" line_range="41-50" />
<code_context>
+      - name: Resolve PR identity and check trust
</code_context>
<issue_to_address>
**🚨 issue (security):** Relying on untrusted metadata.pr_number to pick a PR can mis-route reviews if a commit is shared

Here you prefer `metadata.pr_number` from an untrusted artifact over the authoritative PR list from `listPullRequestsAssociatedWithCommit`. If a commit is shared across multiple PRs (cherry-picks/backports), a malicious workflow could steer reviews to the wrong PR.

Given you already have `prs` from the GitHub API, consider either:
- Ignoring `metadata.pr_number` and selecting solely from `prs` (e.g., require `prs.length === 1` or pick based on expected base/head), or
- Validating `metadata.pr_number` against `prs` plus extra signals (base branch, author) and failing on mismatch.

This keeps the artifact informational-only and avoids relying on untrusted input for routing.
</issue_to_address>

### Comment 2
<location path=".github/workflows/eval-pr-run.yml" line_range="12-16" />
<code_context>
+
+name: Eval PR Run
+
+on:
+  workflow_run:
+    workflows: ["Eval PR"]
+    types: [completed]
+    branches: [main]
+
+permissions:
</code_context>
<issue_to_address>
**question (bug_risk):** The branches filter on workflow_run may unintentionally suppress runs for non-main PRs

Because `workflow_run` uses the *completed workflow’s* branch for `branches` filtering, `branches: [main]` here will typically not match PR runs (which run on the PR’s head branch, not `main`), so this runner may never fire for most PRs targeting `main`.

If you want evals for all PRs into `main`, consider either removing the `branches` filter and relying on the trust/approval checks, or checking the PR’s base branch in `discover` and exiting early when it isn’t `main` to avoid silently skipping valid PRs.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[mrizzi]

Addressing both items:

1. Skills from artifact: Fixed in 206a78d — PR identity (pr_number, author, head_sha) is no longer in the artifact at all. The artifact now carries only skills, which is validated against ^[a-z0-9-]+(,[a-z0-9-]+)*$ in Stage 2.

Recomputing skills in Stage 2 would require the base SHA for git diff, which isn't available from the workflow_run event context. The worst case with a spoofed skills list is running evals for skills that didn't change — wasted compute, not a security breach, since the trust check and PR selection are fully API-derived.

2. branches: [main]: Fixed in d3cc5f9. Removed. Confirmed broken by the PR's own CI (Stage 2 never triggered with the filter present).