Add Test Expert Reviewer workflow#7872
Merged
Evangelink merged 3 commits intomainfrom Apr 27, 2026
Merged
Conversation
New agentic workflow that reviews test code quality on all opened PRs, focusing on: - Test isolation (shared state, static field pollution, cleanup) - Assertion quality (weak assertions, wrong assertion methods) - Flakiness patterns (timing, ports, file system races) - Test completeness (missing edge cases, untested error paths) - Data-driven coverage ([DataRow] edge cases, boundary values) - Test structure (Arrange/Act/Assert, copy-paste tests) - Test performance (unnecessary I/O, heavy DI in unit tests) Also available on-demand via /test-review slash command. Explicitly scoped to test files only — production code review is handled by the Expert Code Reviewer. Designed to be portable to other MSTest repositories. Requires `gh aw compile` to generate the .lock.yml file.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new agentic workflow to provide a dedicated, test-focused PR reviewer for MSTest repositories, emphasizing test isolation, assertion quality, flakiness prevention, and coverage/structure.
Changes:
- Introduces
pr-test-expert-reviewer.mdworkflow definition with/test-reviewon-demand triggering pluspull_request: openedauto-triggering. - Defines a 7-lens review rubric (isolation, assertions, flakiness, completeness, data-driven coverage, structure, performance) and a max budget of 7 inline comments.
- Adds cache-memory based context loading and a deduplication step to avoid duplicate reviews.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/pr-test-expert-reviewer.md |
New agentic workflow prompt/config for a test-quality-focused PR reviewer. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 2
Add blank lines between bold cache-memory file headings and their associated list items in Step 6.
…ce, compile lock file
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Summary
Workflow: Expert Code Reviewer 🧠
Date: 2026-04-27
Repository: microsoft/testfx
Key Findings
This PR adds a new GitHub Actions agentic workflow (pr-test-expert-reviewer.md + compiled pr-test-expert-reviewer.lock.yml). No .cs or other production/test .NET code is changed, so the standard Expert Code Reviewer categories (algorithmic correctness, threading, performance, .NET API compatibility, cross-TFM, IDisposable management) do not apply.
Security & Workflow Configuration — No issues found:
- ✅ Top-level
permissions: {}(deny-all default) with minimal per-job grants (pull-requests: writeonly on the two jobs that actually post review comments) - ✅ No
bashtool access declared — the reviewer is read-only, which is the correct security posture - ✅ Fork PR protection: the
activationjob'sif:condition checksgithub.event.pull_request.head.repo.id == github.repository_id, preventing fork PRs from triggering with elevated permissions - ✅ All third-party action refs are pinned to SHA digests, not mutable tags
- ✅ Concurrency group (
cancel-in-progress: true) prevents parallel duplicate runs
Recommendations
None — no actionable issues found in the Expert Code Reviewer scope for this PR.
Generated by Expert Code Reviewer 🧠
🧠 Reviewed by Expert Code Reviewer 🧠
YuliiaKovalova
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a Test Expert Reviewer that focuses exclusively on test code quality — isolation, assertions, flakiness prevention, coverage, and structure.
Review categories (7 analysis lenses)
Assert.IsTrue(a == b)→AreEqual, wrong assertion methods, over-assertion on implementation detailsThread.Sleep, hard-coded ports, wall-clock assertions, order-dependent collection checks[DataRow]values, redundant test cases, missing negative casesTrigger model
pull_request: opened/test-reviewslash commandSeparation of concerns
src/filestest/filesThe test reviewer explicitly scopes to test files and defers production code review to the Expert Reviewer.
Portability
This reviewer is designed to be portable to other MSTest repositories — the analysis categories are framework-aware but not repo-specific. The only testfx-specific line is the context note about
TestFramework.ForTestingMSTest.Max comments: 7 (vs 5 for expert, 10 for nitpick)
Test PRs often touch more files than production PRs, and test quality issues are typically easier to fix, so a slightly higher budget is justified.
Before merging
gh aw compilemust be run to generatepr-test-expert-reviewer.lock.yml.