Trigger early pipeline test triage - #11016
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
✅ 22/22 dimensions clean — no findings.
This PR adds early/full analysis mode distinction to the pipeline-test-triage workflow and its agent definition. The changes are internally consistent:
- Escalation policy correctly splits the old single "pull-request-local" bullet into two mode-aware bullets (
early: comment-only, no issues;full: final comment, issues at durable thresholds). - Output quality checklist correctly requires the
early-mode disclaimer about incomplete evidence. - Workflow logic correctly promotes
early→fullwhen a delayed child-check event arrives after the build has already completed, preventing stale preliminary comments from overwriting authoritative ones. hide-older-comments: trueon theadd_commentsafe-output correctly implements the "supersedes" semantics described in the agent definition.- Concurrency group shares the same
head_sha-based key for both early and full triggers, socancel-in-progress: truecorrectly cancels an in-flight early run when the full aggregate check completes.
One NIT posted inline on the README trigger-column phrasing.
Comments that could not be inline-anchored
.github/workflows/README.md:67
[NIT] Documentation Accuracy
The trigger column packs three distinct trigger conditions into a single comma-heavy phrase that is harder to parse than the rest of the table:
> Failed Azure Pipelines microsoft.testfx (Build ...) checks, aggregate microsoft.testfx check completed, + manual
The stray + before "manual" and the mix of commas with it read awkwardly. Consider a semicolon-separated list or a bulleted sub-list for clarity, e.g.:
> Failed microsoft.testfx (Build ...) chi…
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Revision freshness and preliminary-comment cleanup must be addressed to prevent stale PR feedback.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
.github/agents/pipeline-test-triage-analyst.agent.md — A completed analysis can downgrade an early finding to an environmental one-off or insufficient… |
|
.github/workflows/pipeline-test-triage.md — The new PR-comment path does not verify that this Azure build still matches the PR's current… |
What changed in this PR
Enables early test-triage feedback from failed Azure Pipelines matrix legs before aggregate completion.
Changes:
- Adds early/full analysis modes and preliminary PR comments.
- Promotes delayed child events to full analysis.
- Regenerates the strict-mode workflow lock and updates documentation.
| File | Description |
|---|---|
.github/workflows/README.md |
Documents early and final triage behavior. |
.github/workflows/pipeline-test-triage.md |
Implements collection modes and PR comments. |
.github/workflows/pipeline-test-triage.lock.yml |
Regenerates the compiled workflow. |
.github/agents/pipeline-test-triage-analyst.agent.md |
Defines preliminary/final analysis policy. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Final feedback can remain stale or absent due to collector early exits and an unguarded post-collection analysis window.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The collector can prevent this final-resolution requirement from ever reaching the analyst: it… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The new PR-comment path does not verify that this Azure build still matches the PR's current… View resolved comment |
|
.github/agents/pipeline-test-triage-analyst.agent.md — A completed analysis can downgrade an early finding to an environmental one-off or insufficient… View resolved comment |
Suppressed comments (1)
.github/workflows/pipeline-test-triage.md:471
- This recheck still leaves the entire evidence upload and AI-analysis window unguarded. If a new commit is pushed after this block, the old-SHA run can post later; because concurrency is SHA-scoped and
hide-older-commentsis enabled, that stale comment can even hide feedback from the newer run. Export the expected head/merge SHAs and require the analyst to re-read the PR immediately before anyadd_commentorcreate_issuecall, no-oping on a mismatch.
if [[ -n "${PR_NUMBER}" ]]; then
if ! LATEST_PR=$(gh api "repos/${GH_REPOSITORY}/pulls/${PR_NUMBER}" 2>/dev/null); then
echo "::warning::Could not re-resolve PR #${PR_NUMBER} after evidence collection; skipping stale feedback."
emit_none
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Amaury Levé (Evangelink)
added
the
state/needs-review
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Evidence API failures can be reported as complete, potentially replacing valid preliminary feedback with an incorrect clearing comment.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — evidenceIncomplete remains false when either the timeline or artifact-list request fails: lines… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The collector can prevent this final-resolution requirement from ever reaching the analyst: it… View resolved comment |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Historical final comments permanently trigger redundant analysis and clearing comments on later successful builds.
Review tier: Balanced
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — evidenceIncomplete remains false when either the timeline or artifact-list request fails: lines… View resolved comment |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/pipeline-test-triage.md:188
- This treats any historical workflow comment as an unresolved preliminary comment. Because final comments carry the same workflow marker and minimized comments remain in the PR comment list,
HAS_PRIOR_TRIAGE_COMMENTstays true forever after the first comment; every later successful build on that PR will run AI analysis and post another clearing comment. Add distinct durable preliminary/final markers (ideally including the build ID) and only activate final resolution when the latest relevant marker is preliminary.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The triggering check name must be sanitized before being written to GitHub Actions command files.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
.github/workflows/pipeline-test-triage.md — CHECK_NAME is event-controlled, but this value is later written verbatim to both $GITHUB_OUTPUT… |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Cancelled builds and silently skipped malformed reports can leave misleading preliminary or final feedback.
Review tier: Balanced
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — CHECK_NAME is event-controlled, but this value is later written verbatim to both $GITHUB_OUTPUT… View resolved comment |
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/pipeline-test-triage.md:45
- A cancelled aggregate run is still filtered out here. If a child leg already posted preliminary feedback and the remaining Azure build is then cancelled, no full-analysis run can post the required final/inconclusive resolution, so the preliminary warning remains indefinitely. Let the completed aggregate
microsoft.testfxcheck through when its conclusion iscancelled(the collector already handles completed non-success results).
.github/workflows/pipeline-test-triage.md:543
evidenceIncompletestill reportsfalsewhen report parsing silently drops evidence. The collector ignores malformed CTRF files at lines 345–349, whilenormalize_reportsalso skips oversized or malformed TRX/JUnit files (.github/scripts/pipeline_test_triage.py:264-270) without making normalization fail. A full run can therefore publish a clean final resolution and hide a valid preliminary finding even though one or more completed-build reports were unreadable. Track skipped reports and include that count in this flag.
EVIDENCE_INCOMPLETE=false
if [[ "${NORMALIZATION_FAILED}" == "true" ]] ||
(( EVIDENCE_FETCH_FAILURES > 0 || DOWNLOAD_FAILURES > 0 )); then
EVIDENCE_INCOMPLETE=true
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Prior-comment detection trusts spoofable marker text without verifying the comment author.
Review tier: Balanced
Findings: None
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/pipeline-test-triage.md:191
- This query trusts the two HTML markers regardless of who authored the comment. Any PR participant can copy those markers into a comment, making a clean completed build consume an AI run as if preliminary feedback were pending; a forged newer
finalmarker can also suppress the clearing run for a real preliminary comment. Filter to comments created by the workflow identity (currentlygithub-actions[bot]) before derivingHAS_PENDING_PRELIMINARY_COMMENT.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Prior-feedback detection is not scoped by build ID, allowing overlapping analyses to suppress the required final resolution.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The prior-feedback lookup ignores the build ID embedded in each state marker. Because manual runs… |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Prior-comment detection fails when safe-output writes use the supported custom GitHub credential.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The prior-feedback lookup assumes every safe-output comment is authored by github-actions[bot],… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The prior-feedback lookup ignores the build ID embedded in each state marker. Because manual runs… View resolved comment |
Suppressed comments (1)
.github/workflows/pipeline-test-triage.md:215
- This fallback lookup has the same hard-coded author assumption. If safe outputs use the supported
GH_AW_GITHUB_TOKENcredential, its comments are not authored bygithub-actions[bot], so an unresolved preliminary marker from an older build is missed and a clean newer build remains silent instead of superseding it. Match the actual safe-output writer identity (or another spoof-resistant workflow provenance) for both lookups.
--jq '.[] | select(.user.login == "github-actions[bot]") |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The asynchronous cross-system workflow and automated PR/issue writes warrant final human validation.
Review tier: Balanced
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/pipeline-test-triage.md — The prior-feedback lookup assumes every safe-output comment is authored by github-actions[bot],… View resolved comment |
Failed Azure Pipelines matrix legs currently wait for the aggregate
microsoft.testfxcheck to finish before test triage starts. Long-running legs can therefore delay useful feedback even when another leg has already published actionable test results.This updates Pipeline Test Triage to:
microsoft.testfx (Build ...)check fails;The generated workflow lock file was rebuilt in strict mode, and action references passed the repository pin audit.
Related issue: N/A