Add /tactics comment-triggered workflow for servicing PRs#53596
Add /tactics comment-triggered workflow for servicing PRs#53596
Conversation
Co-authored-by: marcpopMSFT <12663534+marcpopMSFT@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/sdk/sessions/e9426e50-d112-4958-82fc-c010e834b310
lbussell
left a comment
lbussell
left a comment
There was a problem hiding this comment.
@marcpopMSFT did you try this on your fork? what was the output?
Not great: https://github.com/marcpopMSFT/sdk/actions/runs/23564038929 |
Co-authored-by: marcpopMSFT <12663534+marcpopMSFT@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/sdk/sessions/769c23fd-4c04-415c-a470-007b89cbe31b
Co-authored-by: marcpopMSFT <12663534+marcpopMSFT@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/sdk/sessions/1d3c9fb1-4478-456b-97fd-cf11947df9db
|
@marcpopMSFT what is this trying to do? was this created with github agentic workflows, it looks different to the ones I'm familiar with? |
Cherry-picked changes from marcpopMSFT/sdk main branch: - Switch from GitHub Models to Copilot PAT pool with select-copilot-pat action - Upgrade AI model to gpt-4.1 and improve tactics prompt instructions - Add required Copilot-Integration-Id header to API call - Fix prompt variable reference (split system/user messages) - Update PR description with tactics when no linked issue found - Add pull-requests: write permission for PR body updates Co-authored-by: Marc Paine <marcpop@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
marcpopMSFT
force-pushed
the
copilot/create-workflow-for-tactics-command
branch
from
bf61f1c to
4e5aa57
Compare
@akoeplinger that was an older version. I went ahead and tried copying how @jeffhandley had done PAT token-based calls in the runtime repo. The latest version is the one using that method of calling AI. I updated the description to cover the goal here and included a few examples that I've tested over on my fork. It seems to be working pretty well over there. Open questions:
|
jeffhandley
left a comment
jeffhandley
left a comment
There was a problem hiding this comment.
I recommend converting this into an Agentic Workflow so we gain the benefits of the sandbox the agent runs in and defined safe_outputs.
Thanks for adopting the select-copilot-pat approach though; this is indeed what we're working to adopt. I will connect with you offline for us to coordinate getting the PAT pool set up/managed.
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
Convert update-tactics-on-comment from a traditional GitHub Actions workflow (.yml) to a GitHub Agentic Workflow (.md + .lock.yml). The agentic workflow uses Copilot to analyze PR context and generate tactics, replacing ~580 lines of JavaScript/YAML with a natural language prompt. Key changes: - Add .md source with YAML frontmatter and AI prompt - Add compiled .lock.yml (generated by gh aw compile) - Remove original .yml workflow - Set target: '*' on update-issue/update-pull-request safe-outputs for issue_comment trigger compatibility - Set add-comment max: 3 to allow status reporting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename .md workflow file and regenerate lock file via gh aw compile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
marcpopMSFT
force-pushed
the
copilot/create-workflow-for-tactics-command
branch
from
993cad0 to
19633cd
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new comment-triggered “/tactics” agentic workflow intended to gather PR + linked-issue context and write a standardized tactics section back to the issue (or PR body), using a pooled Copilot PAT selection mechanism.
Changes:
- Add an agentic workflow definition + compiled lockfile for
/tacticsonissue_comment. - Introduce a composite action to randomly select a Copilot PAT from
COPILOT_PAT_0..9plus onboarding docs for maintaining that pool. - Configure safe-output tools to allow commenting, labeling, and updating issue/PR bodies.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| .github/workflows/add-tactics-template-on-comment.md | Source definition/prompt and safe-output configuration for the /tactics workflow. |
| .github/workflows/add-tactics-template-on-comment.lock.yml | Generated, compiled workflow that runs the agent + safe-outputs pipeline. |
| .github/actions/select-copilot-pat/README.md | Documentation for onboarding/maintaining the Copilot PAT pool. |
| .github/actions/select-copilot-pat/action.yml | Composite action that selects a non-empty PAT secret index (0–9). |
|
|
||
| Use the GitHub API to check the repository collaborator permission level for `${{ github.actor }}`. | ||
|
|
||
| - If the actor does **not** have write or admin access, use the `add-comment` tool to post a comment on PR #`${{ github.event.issue.number }}` explaining that only collaborators with write access may trigger this workflow, then stop. |
There was a problem hiding this comment.
The prompt says unauthorized actors should receive a PR comment explaining they can’t trigger the workflow, but the generated workflow currently gates execution in pre_activation and skips downstream jobs entirely when the actor isn’t a team member—so that explanatory comment will never be posted. If you want that feedback behavior, emit it from pre_activation (or a dedicated lightweight job) before deactivating.
| - If the actor does **not** have write or admin access, use the `add-comment` tool to post a comment on PR #`${{ github.event.issue.number }}` explaining that only collaborators with write access may trigger this workflow, then stop. | |
| - If the actor does **not** have write or admin access, use the `add-comment` tool from `pre_activation` (or another lightweight job that runs before deactivation/skipping of downstream jobs) to post a comment on PR #`${{ github.event.issue.number }}` explaining that only collaborators with write access may trigger this workflow, then stop. |
jeffhandley
left a comment
jeffhandley
left a comment
There was a problem hiding this comment.
A few themes of feedback:
- Tighten up the frontmatter safe_outputs
- Try using https://github.github.com/gh-aw/guides/deterministic-agentic-patterns/#custom-trigger-filtering to move logic for collecting data and determining an activation gate to minimize the work done within the agent job and prevent the agent job from running when it's not applicable
- Try using
reaction:orstatus-comment:to apply the immediate feedback it's working, and then either move the other reaction output into a custom safe output, or just drop that functionality.
I've been building workflows up in forks and disposable repositories, following the TrialOps | GitHub Agentic Workflows approach and I recommend that. I've often gotten to where I have ~100 commits to main on that trial repo, I snapshot that over into a branch I push to my primary fork of the target repo, and then I ask Copilot to "rewrite the Git history for this feature to snow a short and natural progression toward completion without all the trial and error." 😄
Changes based on PR #53596 review feedback and end-to-end testing in marcpopMSFT/sdk: - Switch trigger from issue_comment to slash_command (name: tactics, events: [pull_request_comment]) - Framework handles command validation, auth/role checks, reactions, and status comments automatically - Remove manual Steps 1-3 (validation, auth, reaction) from agent prompt since slash_command handles them - Use sanitized trigger context (steps.sanitized.outputs.text) for safe input handling - Restrict safe-outputs to update-pull-request and noop only - Removed update-issue and add-labels: agent never modifies the original issue - Removed add-comment: framework auto-posts status comment with noop summary (avoids duplicates) - /tactics <issue-number> reads the issue as additional input context only, always writes tactics to the PR description - Fix tool name references in prompt (use underscored names: update_pull_request, noop) - Remove emoji reaction instructions from prompt (framework handles reactions) - Set noop report-as-issue: false - Upgrade gh-aw to v0.67.4 (pins Copilot CLI to v1.0.20, fixes MCP registry auth issue) - Set update-pull-request target to '*' (required for slash_command's issue_comment event context) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@jeffhandley I worked with copilot over in my fork taking your and Jan's feedback. I'm pretty happy with the current results. It updates the PR within a minute that it's processing and then updates the comment later with the result and correctly updates the description. Please take another look. |
The goal of this PR is to add a workflow where users with write permission to the repo can comment /tactics on the PR. This will trigger the workflow to grab an AI enabled PAT, review the PR as well as the originating issue (if there is one), update the description to add the .NET tactics template based on the information gathered, and mark the issue with the servicing-consider label.
Some examples:
marcpopMSFT#5
marcpopMSFT#4
marcpopMSFT#2
A zizmor audit has been run on both yml files with no issues identified.
models: readpermission to fix 401 from GitHub Models APIpull-requests: writetoupdatejob to fix 403 when posting comments on PRs (/issues/{pr_number}/commentsreturns "Resource not accessible by integration" without PR write permission)