Enable Microsoft Testing Platform (MTP) for xUnit v3 tests#53517
Draft
MichaelSimons wants to merge 31 commits intodotnet:mainfrom
Draft
Enable Microsoft Testing Platform (MTP) for xUnit v3 tests#53517MichaelSimons wants to merge 31 commits intodotnet:mainfrom
MichaelSimons wants to merge 31 commits intodotnet:mainfrom
Conversation
Switch all ~55 test projects from xUnit v2 (2.9.3) to xUnit v3 (3.1.0) using the arcade SDK's built-in XUnitV3.targets support. Key changes: - Enable TestRunnerName=XUnitV3 in test/Directory.Build.props - Switch from Microsoft.DotNet.XUnitExtensions to XUnitV3Extensions - Upgrade Xunit.Combinatorial 1.3.2 -> 2.0.24 (v3-compatible) - Switch Verify.Xunit -> Verify.XunitV3 - Rewrite test entry point (Program.cs) as ModuleInitializer - Update ITestOutputHelper implementations (new Write/Output members) - Fix IMessageSink namespace (Xunit.Abstractions -> Xunit.Sdk) - Fix DiagnosticMessage namespace (Xunit.Sdk -> Xunit.v3) - Add CallerFilePath/CallerLineNumber ctors for custom attributes (xUnit3003) - Rewrite dotnet-format custom discoverers using IBeforeAfterTestAttribute - Exclude transitive v2 assemblies that conflict with v3 types - Add TestContext alias to resolve Xunit.TestContext ambiguity - Use xunit.v3.assert for non-test projects (SDDLTests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…eues xUnit v3 runs tests out-of-process via a native AppHost, so when cross-compiling (e.g. ARM64 macOS build targeting x64 Helix queues) the publish step must set RuntimeIdentifier to produce the correct AppHost architecture. In CI, Arcade's centralized NuGet restore (via NuGet.targets) does not include RuntimeIdentifier, so the assets file lacks the RID target. An explicit Restore with RuntimeIdentifiers (plural) is added before PublishWithOutput to inject the RID target without clobbering multi-TFM transitive dependencies (e.g. BrowserRefresh which targets net6.0). ErrorOnDuplicatePublishOutputFiles=false suppresses NETSDK1152 from Exe project-references (e.g. dotnet.csproj) whose outputs appear in both the plain-TFM and RID-qualified directories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…etPackageAdd WarmUpNuGetCache: check exit code, add timeout handling, and verify ILLink analyzer DLLs exist after warm-up so failures are surfaced instead of silently causing CSC-only assertion failures. GivenDotnetPackageAdd file-based tests: use RestoreAdditionalProjectSources instead of RestoreSources to add local package paths without replacing NuGet.config feeds. With xUnit v3 hash-based ordering, the NuGet cache may not be pre-populated when these tests run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… dependency Both tests created a tool named MyFileBasedTool v1.0.0 with different source. With xUnit v3 hash-based ordering, Pack_CustomPath runs first and installs the tool; Pack then reuses the cached install instead of its own nupkg, missing the #if !DEBUG output. Rename to PackTool and PackCustomPathTool respectively. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WarmUpNuGetCache was using raw Process.Start which did not set the NUGET_PACKAGES environment variable that DotnetCommand sets via AddTestEnvironmentVariables. This caused packages to be restored to the default NuGet cache (~/.nuget/packages) while the actual tests resolved to the artifacts cache, so the ILLink analyzer DLLs were not found and GetBuildLevel() fell back to BuildLevel.All.
In xUnit v3, tests run out-of-process in a separate AppHost. When ItTerminatesWinExeAppWithCloseMainWindow calls GenerateConsoleCtrlEvent with dwProcessGroupId=0, the CTRL_C signal is sent to all processes sharing the console — including the xUnit v3 test host and the dotnet test process, crashing the entire test run. Fix by launching the dotnet run child process in a new process group via ProcessStartInfo.CreateNewProcessGroup. This isolates the child so that GenerateConsoleCtrlEvent(0, pid) targets only the child's group and does not propagate to the test host. This is the same pattern used by dotnet-watch (ProcessRunner.cs). Add CreateNewProcessGroup property to TestCommand/SdkCommandSpec so other tests that send console signals can use the same mechanism. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…thods Replace CancellationToken.None, default, and bare async calls with TestContext.Current.CancellationToken across all test projects so that test cancellation is responsive to the xUnit v3 runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Switch from VSTest adapter to MTP for test execution, bypassing the ~5s VSTest discovery overhead per invocation. Key changes: - Add eng/XUnitV3/XUnitV3.targets repo-local override using xunit.v3.mtp-v2 (MTP v2-compatible adapter, needed because arcade's built-in targets use xunit.v3 which only brings the MTP v1 adapter) - Add test.runner=Microsoft.Testing.Platform to global.json - Remove UseMicrosoftTestingPlatformRunner=false suppressions - Update Helix work items to run test exe directly with MTP args (--filter-class, --report-trx, --timeout) instead of dotnet test VSTest - Remove stale -wait arg from Build.Tasks.UnitTests launchSettings.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Mar 19, 2026
The AssemblyScheduler partitions test assemblies by class name using metadata heuristics that may include non-test utility classes. With MTP, exit code 8 (zero tests matched) is treated as failure. Use --ignore-exit-code 8 to suppress this, since it only affects the zero-tests case and does not mask real test failures (exit code 1/2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three RunAsync() calls were added by a merge from main after the bulk CancellationToken fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
xUnit v3 runs tests in a child AppHost process, so vstest's --blame-crash flag monitors the wrong process (testhost) and never captures the dump. Setting DOTNET_DbgEnableMiniDump as environment variables ensures they are inherited by the crashing AppHost process, and DOTNET_DbgMiniDumpName points to the Helix upload directory so dumps are preserved as artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Mar 19, 2026
Open
The NuGet.config references a local './binaries' source containing TemplateEngine packages (temp workaround). This directory was not being copied to the Helix payload, causing NU1301 errors on all Helix test work items. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace DOTNET_DbgEnableMiniDump environment variables with the MTP-native --crashdump and --hangdump extensions. These run as test host controllers that properly monitor the out-of-process test host, capturing full dumps to the results directory on crash or hang. - Add Microsoft.Testing.Extensions.CrashDump and HangDump packages - Pass --crashdump and --hangdump flags in Helix work item commands - Remove DOTNET_DbgEnableMiniDump env vars from RunTestsOnHelix scripts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AnsiExtensions.Url() checks TERM to decide whether to emit ANSI hyperlink escapes or raw URLs. Under the MTP runner, the test process inherits TERM=xterm from the CI agent, causing URL assertions to fail. Clearing TERM in the module initializer ensures consistent behavior regardless of the test runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NuGet.config ./binaries source needs to resolve in all test contexts: local builds, CI agent test execution, and Helix. Add MSBuild restore source as belt-and-suspenders, copy binaries to TestLayoutDir alongside NuGet.config so integration tests can find packages at runtime, and copy to Helix TestExecutionDirectory for Helix work items. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
.NET Framework tests are DLLs, not standalone MTP executables, so they must run via 'dotnet test' (vstest). The MTP CrashDump/HangDump extensions are also not compatible with .NET Framework. Use the vstest path with --blame-hang for netfx, and the standalone MTP exe path with --crashdump/--hangdump for .NET Core. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is WIP as it depends on #52930