[POC] Conditional PR test execution - exploring test time reduction#54314
Draft
MichaelSimons wants to merge 5 commits into
Draft
[POC] Conditional PR test execution - exploring test time reduction#54314MichaelSimons wants to merge 5 commits into
MichaelSimons wants to merge 5 commits into
Conversation
Proof of concept for reducing PR validation time by only running tests relevant to the changed source paths. This is exploratory - not intended for merge in current form. Changes: - Add path-based test scope detection scripts (bash + PowerShell) - Wire detection into sdk-build.yml with /p: properties - Add conditional test removal to UnitTests.proj - Add runAllTests pipeline parameter for manual override - Add design document with open questions and future work Safety: CI builds always run all tests. Codeflow PRs (darc-* branches) and dependency updates (eng/Versions.props) also run all tests. Detection failures default to running everything (fail-open). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PowerShell with ErrorActionPreference=Stop treats git's stderr progress
output ('From https://...') as a terminating error. Fix by temporarily
setting ErrorActionPreference to Continue around git calls and piping
stderr through Out-Null.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced May 15, 2026
Open
- Use two-arg git diff (tree comparison) instead of three-dot syntax which requires merge-base computation that fails in shallow clones - Use SilentlyContinue for ErrorActionPreference around git commands in PowerShell to prevent stderr output from becoming terminating errors in PS 5.1 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is exploratory. The goal is to validate timing improvements from conditional test execution in PR builds.
What this does
Reduces PR validation time by only running tests relevant to changed source paths:
src/areas changed viagit diffSafety guarantees
runAllTestspipeline parameterExpected impact
For targeted PRs: ~43% reduction in Helix work items per leg (146 of 336 skippable)
Estimated time: 75 min → 40-50 min
Open questions (see docs/conditional-pr-tests-design.md)
Files changed
.vsts-pr.ymlrunAllTestsparametereng/pipelines/scripts/detect-test-scope.sheng/pipelines/scripts/detect-test-scope.ps1eng/pipelines/templates/jobs/sdk-build.ymltest/UnitTests.projdocs/conditional-pr-tests-design.md