Fix unquarantine threshold to use per-pipeline build counts#66285
Merged
Conversation
Quarantined tests only run in one of the two pipelines (84 or 87), so aggregating build counts across both pipelines dilutes the appearance rate and causes valid unquarantine candidates to be incorrectly excluded. Track counts per-pipeline and compare each test's appearance rate against the pipeline it actually runs in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
the
area-infrastructure
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the unquarantine candidate selection guidance to avoid incorrectly excluding quarantined tests by ensuring the “appeared in ≥66% of builds” threshold is evaluated against the correct per-pipeline build totals.
Changes:
- Update Step 2.1 to aggregate test result counts separately per pipeline (84 vs 87), instead of combining them.
- Update Step 2.2 to compare a test’s appearance rate against the total build count of the pipeline that actually runs it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The unquarantine step was likely skipping valid candidates because it aggregated build counts across both pipelines (84 and 87) and then checked if a test appeared in 66% of all builds. Since a quarantined test only runs in one of the two pipelines, its appearance rate would be ~50% at best when measured against the combined total — always failing the 66% threshold.
Changes