Add solution filtered parallel test run#4226
Add solution filtered parallel test run#4226westey-m merged 1 commit intomicrosoft:feature-xunit3-mtp-upgradefrom
Conversation
a39e324
into
microsoft:feature-xunit3-mtp-upgrade
There was a problem hiding this comment.
Pull request overview
This PR migrates the .NET test execution from sequential bash-based iteration to parallel execution using Microsoft Testing Platform (MTP). A new PowerShell script filters solution files by target framework, enabling dotnet test --solution to run all tests in parallel across supported projects.
Changes:
- Added a PowerShell script to dynamically filter .slnx solution files by target framework, removing test projects that don't support the specified framework
- Replaced bash-based sequential test execution (iterating through projects) with PowerShell-based parallel test execution using filtered solutions and
--filter-query - Maintained all existing test filtering, coverage collection, and environment configuration while improving performance through parallelization
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| .github/workflows/New-FrameworkFilteredSolution.ps1 | New PowerShell script that filters .slnx solution files to include only test projects supporting a specific target framework |
| .github/workflows/dotnet-build-and-test.yml | Migrated unit and integration test execution from bash to PowerShell, using filtered solutions for parallel test execution with --filter-query patterns |
| # Write the filtered solution | ||
| $slnx.Save($OutputPath) | ||
|
|
||
| # Report results to stderr so stdout is clean for piping |
There was a problem hiding this comment.
The comment on line 98 states "Report results to stderr so stdout is clean for piping", but Write-Host doesn't write to stderr. In PowerShell 5.0 and later, Write-Host writes to the information stream (stream 6), not stderr (stream 2). In earlier versions, it writes directly to the console host.
If the intent is to write to stderr to keep stdout clean, use Write-Error (for errors) or redirect to stderr explicitly using Write-Output "..." | Out-File -FilePath ([Console]::Error) -Append. However, since Write-Host output doesn't interfere with variable assignment or piping (as shown in the examples on lines 30-35), the current implementation works correctly for its intended purpose. The comment should be updated to clarify that Write-Host writes to the information stream, not stderr.
| # Report results to stderr so stdout is clean for piping | |
| # Report results via Write-Host (information stream) so stdout is clean for piping |
| } | ||
|
|
||
| # Query the project's target frameworks using MSBuild | ||
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null |
There was a problem hiding this comment.
The dotnet msbuild command's stderr output is being redirected to $null (2>$null), which could hide important error messages about project parsing issues. If the project file is malformed or has build errors, these errors would be silently suppressed.
Consider capturing and logging stderr output, or at least preserving warnings while suppressing only expected informational messages. This would help diagnose issues when the script doesn't work as expected.
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null | |
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo |
| $filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | ||
| dotnet test --solution $filtered --no-build -f net472 | ||
|
|
||
| .EXAMPLE | ||
| # Inline usage with dotnet test (PowerShell) | ||
| dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 |
There was a problem hiding this comment.
The example paths in the script documentation reference "./eng/New-FilteredSolution.ps1", but the script is actually located at ".github/workflows/New-FrameworkFilteredSolution.ps1". This inconsistency between the documentation and the actual file location could confuse users trying to use the script.
Update the example paths in lines 30 and 35 to reflect the actual location: ".github/workflows/New-FrameworkFilteredSolution.ps1".
| $filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | |
| dotnet test --solution $filtered --no-build -f net472 | |
| .EXAMPLE | |
| # Inline usage with dotnet test (PowerShell) | |
| dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 | |
| $filtered = .github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | |
| dotnet test --solution $filtered --no-build -f net472 | |
| .EXAMPLE | |
| # Inline usage with dotnet test (PowerShell) | |
| dotnet test --solution (.github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 |
| Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green | ||
| if ($removed.Count -gt 0) { | ||
| Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow | ||
| foreach ($r in $removed) { | ||
| Write-Host " - $r" -ForegroundColor Yellow | ||
| } | ||
| } | ||
| Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green |
There was a problem hiding this comment.
The script is called with the -Verbose flag on line 151, which will output verbose messages from the script. However, the script uses Write-Host (lines 99-106) for its main output instead of Write-Verbose. This means the verbose flag won't control whether these messages are displayed - they'll always be shown regardless of the -Verbose setting.
For proper verbose handling, informational messages about which projects are being kept or removed should use Write-Verbose instead of Write-Host, and the current Write-Verbose calls (lines 75, 85, 89) should remain as-is. This would allow the -Verbose flag to properly control the verbosity of the script's output.
| Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green | |
| if ($removed.Count -gt 0) { | |
| Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow | |
| foreach ($r in $removed) { | |
| Write-Host " - $r" -ForegroundColor Yellow | |
| } | |
| } | |
| Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green | |
| Write-Verbose "Filtered solution written to: $OutputPath" | |
| if ($removed.Count -gt 0) { | |
| Write-Verbose "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" | |
| foreach ($r in $removed) { | |
| Write-Verbose " - $r" | |
| } | |
| } | |
| Write-Verbose "Kept $($kept.Count) test project(s)." |
| $proj.ParentNode.RemoveChild($proj) | Out-Null | ||
| } | ||
| } | ||
|
|
||
| # Write the filtered solution | ||
| $slnx.Save($OutputPath) |
There was a problem hiding this comment.
The script sets $ErrorActionPreference = "Stop" on line 51, which will cause the script to terminate on any error. However, when the script removes projects from the XML (line 91), it pipes to Out-Null. If RemoveChild fails for any reason, the script will stop without saving the filtered solution or providing a clear error message about which project caused the issue.
Consider adding try-catch blocks around critical operations (especially the XML manipulation and Save operations) to provide more meaningful error messages and ensure cleanup happens even if errors occur.
| $proj.ParentNode.RemoveChild($proj) | Out-Null | |
| } | |
| } | |
| # Write the filtered solution | |
| $slnx.Save($OutputPath) | |
| try { | |
| $proj.ParentNode.RemoveChild($proj) | Out-Null | |
| } | |
| catch { | |
| Write-Error "Failed to remove project '$projRelPath' from solution XML. Error: $($_.Exception.Message)" | |
| } | |
| } | |
| } | |
| # Write the filtered solution | |
| try { | |
| $slnx.Save($OutputPath) | |
| } | |
| catch { | |
| Write-Error "Failed to save filtered solution to '$OutputPath'. Error: $($_.Exception.Message)" | |
| throw | |
| } |
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null | ||
| $targetFrameworks = $targetFrameworks.Trim() | ||
|
|
||
| if ($targetFrameworks -like "*$TargetFramework*") { |
There was a problem hiding this comment.
The script uses substring matching with the -like operator on line 84 to check if a target framework is supported. While this works correctly for the framework monikers used in this repository (net10.0, net9.0, net8.0, net472), it could cause issues with platform-specific target frameworks in the future.
For example, if a project targets "net8.0-windows" and you filter for "net8.0", the substring match would succeed even though "net8.0" doesn't match "net8.0-windows" exactly. Consider splitting TargetFrameworks by semicolons and checking for exact matches to make the script more robust for future use cases.
| if ($targetFrameworks -like "*$TargetFramework*") { | |
| # TargetFrameworks is a semicolon-separated list; match the requested framework exactly | |
| $frameworkList = $targetFrameworks -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne '' } | |
| if ($frameworkList -contains $TargetFramework) { |
Motivation and Context
Description
Contribution Checklist