Fix quarantine workflow skipping PR build failures#66546
Merged
Conversation
github-actions
Bot
added
the
area-infrastructure
wtgodbe
force-pushed
the
wtgodbe/PRFails
branch
from
b3828bc to
bd5e5b0
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the daily test-quarantine workflow instructions to ensure the AI agent processes merged PR build failures (Source B) instead of skipping them due to perceived scale, improving quarantine coverage for flakes that only appear in PR builds.
Changes:
- Adds an explicit “REQUIRED — do not skip” directive for Source B.
- Restructures Source B instructions to pre-filter PR builds by failed/partially-succeeded results before PR merge-status verification.
wtgodbe
force-pushed
the
wtgodbe/PRFails
branch
from
bd5e5b0 to
c97e524
Compare
The AI agent was lazily skipping Source B (merged PR failures) because querying all PR builds returned ~284 results and verifying each PR's merge status seemed 'too slow'. Changes: 1. Add explicit 'REQUIRED — do not skip' directive for Source B 2. Restructure steps: deduplicate PR numbers first, verify B2/B3 once per unique PR (not per build), then find failed builds within qualifying PRs. This keeps the retry-then-pass scenario while reducing API calls. 3. Reduce PR build window from 30 days to 7 days for a more focused and manageable dataset. 4. Fix B1 to compare against PR's head.sha from pull_request_read rather than inferring from the build list (which could be wrong if the final commit's build succeeded and was filtered out). 5. Add pagination reminder ( + continuationToken + minTime). 6. Remove duplicate 'get failed test results' sentence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
wtgodbe
force-pushed
the
wtgodbe/PRFails
branch
from
c97e524 to
5988f1d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The daily test quarantine workflow's AI agent was lazily skipping Source B (merged PR build failures) because querying all PR builds returned ~284 results and verifying each PR's merge status seemed 'too slow' to the agent. Evidence from the last two runs:
This means flaky tests that only fail in PR builds (not rolling builds) were never being detected for quarantine.
Changes
resultFilter=failedbefore grouping by PR — this reduces the working set from ~284 PRs to just the ones with failed builds, removing the agent's justification for skipping