Skip to content

Update code-review workflow and standardize AI disclosure#126110

Open
stephentoub wants to merge 6 commits intodotnet:mainfrom
stephentoub:updatecragent
Open

Update code-review workflow and standardize AI disclosure#126110
stephentoub wants to merge 6 commits intodotnet:mainfrom
stephentoub:updatecragent

Conversation

@stephentoub
Copy link
Copy Markdown
Member

@stephentoub stephentoub commented Mar 25, 2026

Update the code-review agentic workflow and standardize AI disclosure text across all skills.

Code-review workflow changes

  • slash_command — trigger the workflow using /code-review comment triggers rather than when pull requests are created or synchronized. This avoids the "Approve workflows to run" button on PRs from community contributors and relies on a human-in-the-loop trigger.
  • workflow_dispatch — add manual trigger with pr_number input
  • timeout-minutes: 45 — bump from default 20 minutes
  • concurrency — cancel in-progress reviews when new commits are pushed to the same PR
  • Empty-PR guard — skip review and don't post a comment if the PR has no code changes

AI disclosure updates (all skills + copilot-instructions)

  • Add github-actions[bot] and copilot as explicitly exempted accounts
  • Add "at the bottom of the content" placement guidance

Copilot AI review requested due to automatic review settings March 25, 2026 17:49
@stephentoub stephentoub requested a review from a team as a code owner March 25, 2026 17:49
@github-actions github-actions Bot added the area-skills Agent Skills label Mar 25, 2026
@stephentoub stephentoub requested a review from ericstj March 25, 2026 17:52
@stephentoub
Copy link
Copy Markdown
Member Author

@jeffhandley, can you help with the Code Review / activation (pull request) failure?

Error: None of the following secrets are set: COPILOT_GITHUB_TOKEN
The GitHub Copilot CLI engine requires either COPILOT_GITHUB_TOKEN secret to be configured.

Common causes if the secret appears to be configured:
  - Organization secrets must have repository access granted
  - Environment secrets require the job to specify that environment
  - Secret names are case-sensitive - verify exact spelling

Documentation: 'https://github.github.com/gh-aw/reference/engines/#github-copilot-default'
Error: Process completed with exit code 1.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the repository’s agentic code-review workflow to support additional trigger scenarios and standardizes the AI-generated content disclosure guidance across skills and Copilot instructions.

Changes:

  • Expanded code-review workflow triggering (ready_for_review, workflow_dispatch), added concurrency + increased timeout, and added an “empty PR” no-comment guard in the prompt.
  • Adjusted compiled workflow (code-review.lock.yml) to match the new triggering/activation behavior.
  • Standardized AI disclosure wording/placement guidance across skills and .github/copilot-instructions.md.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
.github/workflows/code-review.md Adds timeout/concurrency, new triggers (ready_for_review, workflow_dispatch), fork allowance, and “empty diff” guard text; updates prompt to use PR number/input.
.github/workflows/code-review.lock.yml Compiled workflow updated to match new triggers/timeout/concurrency and loosened activation gating.
.github/skills/vmr-codeflow-status/SKILL.md Updates AI disclosure guidance (bot exemptions + bottom placement).
.github/skills/performance-benchmark/SKILL.md Updates AI disclosure guidance (bot exemptions + bottom placement).
.github/skills/code-review/SKILL.md Updates AI disclosure guidance and adjusts the sample review heading format.
.github/skills/ci-analysis/SKILL.md Updates AI disclosure guidance (bot exemptions + bottom placement).
.github/skills/api-proposal/SKILL.md Updates AI disclosure guidance (bot exemptions + intended bottom placement).
.github/copilot-instructions.md Updates repo-wide AI disclosure guidance (bot exemptions + bottom placement).
Comments suppressed due to low confidence (5)

.github/workflows/code-review.lock.yml:243

  • github.event.inputs.pr-number is invalid for the workflow_dispatch input (hyphenated name) and will cause expression evaluation failures here too. Use github.event.inputs['pr-number'] or rename the input and update references.
        env:
          GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt
          GH_AW_EXPR_61AA0003: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
          GH_AW_GITHUB_ACTOR: ${{ github.actor }}

.github/workflows/code-review.md:111

  • github.event.inputs.pr-number is invalid for a hyphenated input name and will fail expression evaluation on workflow_dispatch. Use github.event.inputs['pr-number'] (or rename the input) here as well.
Follow the instructions in SKILL.md to perform a thorough code review of PR #${{ github.event.pull_request.number || github.event.inputs.pr-number }}.

.github/workflows/code-review.md:103

  • github.event.inputs.pr-number is invalid for a hyphenated input name and will fail expression evaluation on workflow_dispatch. Use github.event.inputs['pr-number'] (or rename the input) here.
You are an expert code reviewer for the dotnet/runtime repository. Your job is to review pull request #${{ github.event.pull_request.number || github.event.inputs.pr-number }} and post a thorough analysis as a comment.

.github/workflows/code-review.lock.yml:164

  • This workflow still uses github.event.inputs.pr-number, which is invalid for a hyphenated input name and will break workflow_dispatch runs. Use github.event.inputs['pr-number'] (or rename the input) when populating this env var.
          GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt
          GH_AW_SAFE_OUTPUTS: ${{ runner.temp }}/gh-aw/safeoutputs/outputs.jsonl
          GH_AW_EXPR_61AA0003: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
          GH_AW_GITHUB_ACTOR: ${{ github.actor }}

.github/workflows/code-review.lock.yml:232

  • github.event.inputs.pr-number is invalid for the workflow_dispatch input (hyphenated name) and will cause expression evaluation failures here. Use github.event.inputs['pr-number'] or rename the input and update references.
        env:
          GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt
          GH_AW_EXPR_61AA0003: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
        with:

Comment thread .github/workflows/code-review.lock.yml Outdated
Comment thread .github/workflows/code-review.md Outdated
Comment thread .github/workflows/code-review.md Outdated
Comment thread .github/skills/api-proposal/SKILL.md Outdated
Comment thread .github/workflows/code-review.md
Comment thread .github/workflows/code-review.lock.yml Outdated
Copilot AI review requested due to automatic review settings March 26, 2026 21:13
@stephentoub
Copy link
Copy Markdown
Member Author

@jeffhandley, for now I've removed the role and forks change, in order to get the rest of the changes in.

@stephentoub
Copy link
Copy Markdown
Member Author

/ba-g markdown only

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/code-review.md
Comment thread .github/workflows/code-review.lock.yml
Comment thread .github/workflows/code-review.lock.yml
stephentoub and others added 3 commits March 29, 2026 07:38
When triggered via workflow_dispatch, the PR branch is not automatically
checked out. Added Step 0 instructions for the agent to fetch/checkout
the PR branch and pass item_number to add-comment so the review targets
the correct PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 29, 2026 11:41
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

.github/workflows/code-review.md:124

  • The empty-PR guard is currently only enforced via prompt instructions (“Simply stop without producing any output”). If the goal is to guarantee no comment is posted when the effective diff is empty, consider adding an explicit workflow-level check (e.g., compute PR files/patch size via API and short-circuit before invoking the agent / before accepting add-comment).
Follow the instructions in SKILL.md to perform a thorough code review of PR #${{ github.event.pull_request.number || github.event.inputs.pr_number }}.

**Important:** Before performing any analysis, check whether the PR has any actual code changes (lines added, removed, or modified). If the diff is empty (e.g., a merge commit with no effective changes), do **not** post a review comment. Simply stop without producing any output.

When completed, post the review output as a regular comment on the PR using the `add-comment` safe output.

.github/workflows/code-review.md:110

  • For workflow_dispatch, this relies on the agent following manual git commands in the prompt to fetch/checkout the PR branch. To make the workflow deterministic (and avoid reviewing default-branch content if the agent deviates), consider doing the PR ref fetch/checkout as an explicit workflow step when github.event_name == 'workflow_dispatch' instead of only documenting it in the prompt.
## Step 0: Prepare Workspace (workflow_dispatch only)

When this workflow is triggered via `workflow_dispatch`, the PR branch is **not** automatically checked out — the workspace contains the default branch. Before reviewing, you **must** fetch and check out the PR branch so the workspace reflects the PR's code:

```bash
git fetch origin pull/${{ github.event.pull_request.number || github.event.inputs.pr_number }}/head:pr-branch
git checkout pr-branch
</details>

Comment thread .github/skills/code-review/SKILL.md
Comment thread .github/workflows/code-review.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 29, 2026 11:52
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comment thread .github/workflows/code-review.md
Comment thread .github/workflows/code-review.md Outdated
@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

Copilot Code Review

Holistic Assessment

Motivation: This PR switches the code-review workflow from auto-triggering on pull_request events to on-demand /code-review slash commands and manual workflow_dispatch, and standardizes AI disclosure text across all skills. Both changes are well-motivated — the slash-command approach avoids the "Approve workflows to run" friction on community PRs, and the disclosure standardization removes inconsistencies.

Approach: Sound overall. The workflow config changes use proper gh-aw features, the AI disclosure edits are mechanical and consistent, and the lock file recompilation is clean. Prior review feedback (hyphenated input names, contradictory "prepend at the bottom", fork/roles removal, disclosure placement placeholder) has been addressed.

Summary: ⚠️ Needs Human Review. There is one expression-level concern about PR number resolution for issue_comment-triggered slash commands that needs verification by someone familiar with the gh-aw framework's runtime behavior. The PR description is also stale relative to the current code. The rest of the changes are correct and well-structured.


Detailed Findings

Detailed Findings

⚠️ PR number expression may not resolve for issue_comment slash commands

The expression ${{ github.event.pull_request.number || github.event.inputs.pr_number }} is used in multiple places in code-review.md:

  1. safe-outputs.add-comment.target (line 25)
  2. concurrency.group (line 34)
  3. The prompt text (lines 103, 110, 114, 122)

For slash_command via pull_request_review_comment, github.event.pull_request.number IS populated — this path works correctly.

For slash_command via pull_request_comment (which compiles to issue_comment — the most common path, where someone types /code-review in a PR comment), github.event.pull_request is NOT set on issue_comment events. The PR number lives at github.event.issue.number instead. Since github.event.inputs.pr_number is also empty (not workflow_dispatch), the expression evaluates to empty string.

Potential impacts:

  • Concurrency group becomes code-review- (empty suffix) for all issue_comment triggers, meaning concurrent /code-review invocations on different PRs could cancel each other.
  • Safe-output target is empty — the framework captures GH_AW_GITHUB_EVENT_ISSUE_NUMBER (line 238 of the lock file) and may auto-resolve, but this needs verification.
  • Prompt text would say "PR #" with no number, and the git fetch command in Step 0 would fail.

Note: The gh-aw framework may handle this internally via GH_AW_GITHUB_EVENT_ISSUE_NUMBER and GH_AW_IS_PR_COMMENT detection for the safe-output target. However, the concurrency.group is evaluated at the GitHub Actions level before any framework code runs, so it cannot be rescued by framework logic. A human familiar with the framework should verify.

For comparison, the breaking-change-doc workflow uses github.run_id as a final fallback: ${{ github.event.pull_request.number || inputs.pr_number || github.run_id }} (line 101 of breaking-change-doc.lock.yml).

If this IS an issue, the fix would be:

${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}

⚠️ PR description is stale — lists features not in the PR

The PR description still lists:

  • roles: all — not present in the code
  • ready_for_review — not present in the code
  • forks: ["*"] — not present in the code
  • timeout-minutes: 30 — actually 45 in the code

The author noted in a comment that roles/forks were removed, but the description wasn't updated. The latest commit also introduced the slash_command trigger (replacing pull_request), which is a significant behavioral change not mentioned in the description. Consider updating the description for future reference.

✅ AI disclosure standardization — consistent and correct

All 6 files consistently:

  • Add explicit bot account examples: (e.g., \github-actions[bot]`, `copilot`)`
  • Change placement to "at the bottom of the content"
  • Use "add" as the verb (fixing the contradictory "prepend...at the bottom" from earlier revisions)

The code-review SKILL.md template also adds appropriate <!-- AI disclosure note --> placeholder comments and wraps Detailed Findings in <details> for collapsibility. The header was simplified from ## 🤖 Copilot Code Review — PR #<number> to ## Copilot Code Review, removing the emoji and per-PR numbering.

✅ Workflow config changes — well-structured

  • slash_command trigger with name: code-review and correct event types compiles correctly to issue_comment/pull_request_review_comment with proper body-matching conditions in the lock file.
  • workflow_dispatch with pr_number input (using underscore, not hyphen) is correct — the earlier hyphen issue was fixed.
  • Step 0 workspace preparation with git fetch + git checkout is appropriately scoped for workflow_dispatch.
  • Empty-PR guard in Step 2 is a sensible addition.
  • timeout-minutes: 45 is reasonable for a code review workflow.
  • report-failure-as-issue: false and allowed-reasons: [outdated] are appropriate for a review workflow.
  • network: defaults simplification is correct syntax.

✅ Lock files and actions-lock.json — clean recompilation

  • All 3 lock files consistently use gh-aw v0.68.1.
  • actions-lock.json properly consolidates from 4 stale gh-aw-actions/setup entries to 1 (v0.68.1).
  • actions/github-script upgraded v8→v9 with valid SHA.
  • New lock file format includes gh-aw-manifest header with secrets and actions audit trail — this is a good transparency improvement.

💡 min-integrity: none added to GitHub tools — verify intent

min-integrity: none was added to the tools.github configuration (line 15 of code-review.md). This is unique among the repository's workflows — breaking-change-doc and copilot-echo use the default ($GITHUB_MCP_GUARD_MIN_INTEGRITY). If intentional (e.g., needed for slash_command trigger context where integrity metadata may not be available), this is fine. Worth confirming this is deliberate rather than a leftover from testing.

Note

This review was created with the assistance of GitHub Copilot using Claude Opus 4.6.

Generated by Code Review (staging) · ● 7.5M ·

@danmoseley
Copy link
Copy Markdown
Member

Cc @mitchdenny

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-skills Agent Skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants