Conditional test filtering framework for PR builds

[MichaelSimons]

Summary

Introduces a framework for conditionally running tests on PR builds based on what source files changed. Tests that are unrelated to the PR's changes can be skipped, reducing Helix compute consumption and improving PR validation times. All tests continue to run unconditionally on CI builds (main, release branches).

This is posted as a draft to seek feedback on the approach. The implementation is included to illustrate the direction — it is intentionally simple and small despite addressing a complex problem.

Received positive feedback on the approach — proceeding with the full implementation of conditional test filtering.

Design

See documentation/project-docs/pr-test-filtering.md for the full motivation, mechanisms, and instructions for adding new scopes.

What's included

• Design document with motivation, mechanisms, and instructions for adding scopes
• C# evaluation script
• Central config file (ConditionalTests.props) with TemplateEngine as the first scope
• MSBuild + pipeline integration
• Agent instructions update

Related to #55203

[nagilson]

Overall, I'm in favor of the proposal.

However, I want to poke on two points:

1. How will we define TriggerPaths?
   I think this should be more than the folder containing relevant source code for that repo - e.g. dotnet-watch project should not be conditioned on just dotnet-watch but also include the ProjectReference(s) such as ..\..\Cli\Microsoft.DotNet.Cli.Definitions\Microsoft.DotNet.Cli.Definitions.csproj. Changes to PackageReference(s) such as Microsoft.Build.Locator" may also result in a break.

dotnetup already uses conditional test filtering to a degree but we've seen more breaks than expected when running all of the tests. So, there is a risk involved in disabling these tests though on average I think it's the right approach. We should try to minimize that risk.

There have been times where we caught product bugs (e.g. system command line, roslyn, and others) because tests in other areas started failing even when roslyn's tests were all passing. But we shouldn't rely on test coverage of others like that.

2. When can we ignore 'CoditionalTests' - I think we should be able to ignore that. We can still get a lot of the time saving on codeflow or certain dev PRs... but I think for example in dotnet/dotnet flow we will still want to run all of the tests so we get full filtering.

I'd propose to have the conditional tests condition test runs out on code flow for specific repos - e.g. a change to dotnet/templating code flow shouldn't need to run the watch tests. If we are all encompassing on dependencies with 1 then maybe we can be looser on that?

[MichaelSimons]

@noahfalk, thanks for the insightful feedback.

1. How will we define TriggerPaths?

You are correct that defining just the src path for the affected area is often not sufficient. I did not elaborate on this in the documentation but should call it out in order to help devs and AI make better decisions when defining new test scopes.

So, there is a risk involved in disabling these tests though on average I think it's the right approach. We should try to minimize that risk.

Yes this is in ways a big hammer so it must be used correctly. Having good documentation calling these gotchas is imperative.

2. When can we ignore 'CoditionalTests'

I do see it valuable to be able to specify a RunAlways setting to be able to force tests to run on code flows. Given the vmr, it might not provide substantive value in providing fined grained support.

I updated the doc in d518adb in response to this feedback.

[noahfalk]

@noahfalk, thanks for the insightful feedback.

I do love free compliments but I think you meant to tag a different Noah (@nagilson) :)

[nagilson]

You are correct that defining just the src path for the affected area is often not sufficient. I did not elaborate on this in the documentation but should call it out in order to help devs and AI make better decisions when defining new test scopes.

@MichaelSimons Perhaps ambitious, but as a starting point would we want to consider a one-time build of the msbuild project dependency graph - and rebuild it whenever the project file changes? This would give us a better idea of what other TriggerPaths definitions to include without requiring the consumer to have to do that.

[MichaelSimons]

Perhaps ambitious, but as a starting point would we want to consider a one-time build of the msbuild project dependency graph - and rebuild it whenever the project file changes? This would give us a better idea of what other TriggerPaths definitions to include without requiring the consumer to have to do that.

@nagilson - Dependency graph resolution is an interesting idea. For now though, I'd like to get a few scopes defined and validate the realized filtering gains before investing in additional infrastructure.

It's also worth keeping in mind that this doesn't have to be 100% perfect — these tests still run on every CI build. We're only filtering on PR validation. If we start seeing that PR validation is frequently missing things that CI catches, that's the signal to adjust/invest in more tooling. But I don't think we need that up front.

[MichaelSimons]

This is now ready for a review of the implementation.

[Copilot]

Pull request overview

Adds infrastructure to conditionally skip selected test projects in PR validation based on the set of files changed, while keeping full test execution on non-PR CI builds. This aims to reduce Helix usage and shorten PR turnaround by filtering unrelated test suites.

Changes:

• Introduces a central scope configuration file (test/ConditionalTests.props) and integrates it into test/UnitTests.proj to remove skipped test projects before Helix submission.
• Adds a C# evaluation script (scripts/EvaluateConditionalTestScopes.cs) and wires it into the AzDO pipeline to set SkippedTestScopes.
• Adds design documentation and updates Copilot agent guidance to reference the new filtering system.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/UnitTests.proj	Imports conditional scope config and removes skipped test projects via a new MSBuild target.
test/ConditionalTests.props	Defines global trigger paths and the first conditional scope (TemplateEngine) with trigger/test project patterns.
scripts/EvaluateConditionalTestScopes.cs	Computes changed files vs PR target branch, evaluates scopes, and sets SkippedTestScopes pipeline variable.
eng/pipelines/templates/jobs/sdk-build.yml	Runs the evaluation script before queuing Helix tests and passes SkippedTestScopes into MSBuild.
documentation/project-docs/pr-test-filtering.md	Documents motivation, design, and instructions for adding scopes.
.github/copilot-instructions.md	Notes the existence of conditional test filtering and points contributors to the configuration/doc.