ci(pr): inline annotations and changed-specs review#15755
Merged
dmcilvaney merged 3 commits intomicrosoft:tomls/base/mainfrom Feb 9, 2026
Merged
ci(pr): inline annotations and changed-specs review#15755dmcilvaney merged 3 commits intomicrosoft:tomls/base/mainfrom
dmcilvaney merged 3 commits intomicrosoft:tomls/base/mainfrom
Conversation
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
sameluch
reviewed
Feb 7, 2026
Add a github action which runs on pull request and adds inline annotations for changed specs based on upstream package standards. The check uses the Copilot CLI via a GitHub App user and requires the following secrets to be set in the repository: - COPILOT_TOKEN: The token for the GitHub App user. The action will run two parallel copies of the spec review agent with different models to review the changed specs and provide feedback. A third agent will then combine the feedback from both agents and create inline annotations on the pull request. Any annotation that is an error will cause the check to fail, while warnings and suggestions will be added as annotations without failing the check.
dmcilvaney
force-pushed
the
damcilva/spec_agent_workflow
branch
from
cd16099 to
642dd4c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new pull-request GitHub Actions workflow that runs a Copilot-based multi-model RPM spec review, validates the resulting JSON report, and surfaces findings as inline annotations and a PR comment.
Changes:
- Introduces
.github/workflows/spec-review.ymlto run on PRs touching**/*.spec(plusworkflow_dispatch) and publish annotations/comments/artifacts. - Adds a set of helper scripts to run single-/multi-model reviews and to validate/format/annotate the generated report.
- Adds Python dependency (
pydantic>=2.0) for report schema validation.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/spec-review.yml | New PR workflow orchestration for spec review, validation, annotations, commenting, and artifacts. |
| .github/workflows/scripts/spec_review.sh | Runs the Copilot agent for spec review (and synthesis), manages inputs/outputs, validates JSON. |
| .github/workflows/scripts/spec_review_multi.sh | Runs two reviewer passes + synthesis pass and produces intermediate artifacts. |
| .github/workflows/scripts/spec_review_schema.py | Pydantic schema validation + report comparison output. |
| .github/workflows/scripts/create_check_annotations.py | Converts report findings into GitHub workflow-command or Checks-style annotations. |
| .github/workflows/scripts/format_pr_comment.py | Formats report into a PR comment with links and a raw JSON section. |
| .github/workflows/scripts/_common.py | Shared path utilities for annotations/comment formatting. |
| .github/workflows/scripts/requirements.txt | Adds pydantic dependency for the scripts. |
| .github/workflows/scripts/README.md | Local developer guide for running the spec review scripts/CLI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+34
to
+38
| - name: Test gh auth | ||
| env: | ||
| GH_TOKEN: ${{ secrets.COPILOT_TOKEN }} | ||
| run: | | ||
| gh auth status |
There was a problem hiding this comment.
In fork-based PRs, repository secrets (including COPILOT_TOKEN) are not provided, so this step will fail and block the workflow. Consider gating all steps that require COPILOT_TOKEN behind a condition that only runs for same-repo PRs (or when the secret is present), and emit a clear notice when skipped.
Contributor
Author
There was a problem hiding this comment.
will deal with this later
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a github action which runs on pull request and adds inline annotations for changed specs based on upstream package standards.
The check uses the Copilot CLI via a GitHub App user and requires the following secrets to be set in the repository:
The action will run two sequential copies of the spec review agent with different models to review the changed specs and provide feedback.
A third agent will then combine the feedback from both agents and create inline annotations on the pull request.
Any annotation that is an error will cause the check to fail, while warnings and suggestions will be added as annotations without failing the check.
NOTE: This will only work for non-fork PRs right now, will need to setup an environment to handle that later.