Migrate to xunit.v3#13482
Conversation
fe653b5 to
65d17a8
Compare
|
fyi I'm working on this in #13215, but won't be mad if you beat me to it. Chasing what looks like some Windows hangs at the moment, and as a low-pri background thing. |
|
Oh, I didn't know you are already working on it. |
|
@rainersigwald I tried to pull in some changes you did in the other PR here to see if it gets me here in a better state. We can sync next Tuesday to align the efforts. Generally speaking, a big difference between the two PRs is that in this PR I'm running with MTP, while the other PR runs with VSTest. But I would think most test failures will be related to the xunit 2 to xunit 3 move and not VSTest/MTP differences. In addition, in this PR I'm keeping the tests wrapped in One thing to note is that xunit.v3 might be running the tests in a different order compared to xunit 2, which might (or might not) be the cause for some of the failures (i.e, some tests might have been modifying global state without resetting it - that was somehow unnoticed with xunit 2 but might become apparent now that the ordering changed) |
I believe this is not currently working--my interpretation of a bunch of failures in my attempt was "something is making this work now and failing a bunch of tests that should have been failing before but weren't". I'd love to have all those tests fixed of course . . . |
In the other PR, I commented on why the approach there won't work. But I think it might have been working on main already. |
ec93ec7 to
6f93f7e
Compare
cdf2641 to
9b1339b
Compare
There was a problem hiding this comment.
Pull request overview
This PR migrates the repo’s unit test infrastructure from xUnit v2 patterns to xUnit v3, aligning execution with Microsoft.Testing.Platform and updating shared test bootstrapping so test environment initialization remains consistent across the suite.
Changes:
- Switched test dependencies to xUnit v3 (and Verify.XunitV3), removed xUnit v2 console runner usage, and updated test runner/coverage configuration.
- Replaced the custom xUnit v2 “AssemblyFixtureSupport” framework with xUnit v3 pipeline startup + a custom test framework wrapper to create
TestEnvironmentper test case. - Updated many test projects/files to remove
Xunit.Abstractionsusages and stabilize a few test-global/static state interactions.
Reviewed changes
Copilot reviewed 160 out of 160 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Xunit.NetCore.Extensions/Xunit.NetCore.Extensions.csproj | Switch extension package reference to xUnit v3 extensibility core and adjust TFMs/settings. |
| src/Xunit.NetCore.Extensions/UseInvariantCultureAttribute.cs | Update BeforeAfterTestAttribute overrides to xUnit v3 signatures. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestMethodRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 assembly fixture runner implementation. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkWithAssemblyFixture.cs | Removed legacy xUnit v2 custom test framework. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkExecutorWithAssemblyFixture.cs | Removed legacy xUnit v2 executor wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestCollectionRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 collection runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestClassRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 class runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestAssemblyRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 assembly runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/README.md | Removed docs for the deleted xUnit v2 assembly fixture support. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/AssemblyFixtureAttribute.cs | Removed legacy assembly-level fixture attribute. |
| src/Utilities.UnitTests/ToolTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Utilities.UnitTests/ToolLocationHelper_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Utilities.UnitTests/ProcessExtensions_Tests.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/TestEnvironment.cs | Update default output helper implementation for xUnit v3 interface surface. |
| src/UnitTests.Shared/RunnerUtilities.cs | Update xUnit output helper import namespace. |
| src/UnitTests.Shared/ObjectModelHelpers.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/MockLogger.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/MockEngine.cs | Update xUnit output helper import namespace. |
| src/UnitTests.Shared/Microsoft.Build.UnitTests.Shared.csproj | Replace xUnit v2 packages with xUnit v3 assert/extensibility packages. |
| src/Tasks.UnitTests/XamlDataDrivenToolTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/WriteLinesToFile_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/VerifyFileHash_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Unzip_Tests.cs | Replace conditional attribute usage with runtime skip under xUnit v3. |
| src/Tasks.UnitTests/TestResources/TestBinary.cs | Update xUnit-related imports. |
| src/Tasks.UnitTests/RoslynCodeTaskFactory_Tests.cs | Remove Verify-specific attribute usage. |
| src/Tasks.UnitTests/ResourceHandling/ResGenDependencies_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/MSBuildResXReader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/GenerateResourceOutOfProc_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/GenerateResource_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResolveNonMSBuildProjectOutput_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResolveAssemblyReference_CustomCultureTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/RemoveDir_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/RegressionTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/PortableTasks_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/OutputPathTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/MSBuildInternalMessage_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/MSBuild_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Microsoft.Build.Tasks.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/Tasks.UnitTests/HintPathResolver_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetSDKReference_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetFileHash_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetCompatiblePlatform_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetAssembliesMetadata_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GenerateBindingRedirects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/FormatUrl_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Exec_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateVisualBasicManifestResourceName_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateItem_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateCSharpManifestResourceName_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Copy_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssignProjectConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/WinMDTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkHigherThanRedist.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkAttribute.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyIgnoreVersionForFrameworkReference.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/SuggestedRedirects.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAutoUnify.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAppConfig.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependency.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/SpecificVersionPrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/ResolveAssemblyReferenceTestFixture.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/ReferenceTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Perf.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/NonSpecificVersionStrictPrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Node/OutOfProcRarNode_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Miscellaneous.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/InstalledSDKResolverFixture.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/GlobalAssemblyCacheTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/FilePrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/AssemblyFoldersFromConfig_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Al_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AddToWin32Manifest_Tests.cs | Remove xUnit v2 abstractions import. |
| src/StringTools.UnitTests/StringTools.UnitTests.net35.csproj | Force OutputType Exe for net35 test project under new runner approach. |
| src/StringTools.UnitTests/InterningTestData.cs | Update xUnit-related imports. |
| src/Shared/UnitTests/xunit.runner.json | Remove xUnit v2 runner settings (shadow copy/app domain). |
| src/Shared/UnitTests/TypeLoader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Shared/UnitTests/TestProgram.cs | Removed custom xUnit v2 console entrypoint program. |
| src/Shared/UnitTests/TestAssemblyInfo.cs | Introduce xUnit v3 pipeline startup + custom test framework wrapper for TestEnvironment. |
| src/MSBuild.UnitTests/XMake_Tests.cs | Remove unused test platform + xUnit v2 abstractions imports. |
| src/MSBuild.UnitTests/XMake_BinlogSwitch_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/PerfLog_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/MSBuildServer_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/MSBuildMultithreaded_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/Microsoft.Build.CommandLine.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/MSBuild.UnitTests/CommandLineSwitches_Tests.cs | Stabilize change-wave state setup for a test. |
| src/Framework.UnitTests/FileUtilities_Tests.cs | Replace conditional fact usage with runtime skip under xUnit v3. |
| src/Framework.UnitTests/FileMatcher_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Directory.Build.targets | Update runner args/trait filtering, add xUnit v3 extensions + MTP coverage integration, remove xUnit console runner wiring. |
| src/BuildCheck.UnitTests/TaskInvocationAnalysisDataTests.cs | Remove xUnit v2 abstractions import. |
| src/BuildCheck.UnitTests/EndToEndTests.cs | Remove xUnit v2 abstractions import. |
| src/BuildCheck.UnitTests/BuildCheckManagerProviderTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/WarningsAsMessagesAndErrors_Tests.cs | Adjust assembly location resolution logic for task loading. |
| src/Build.UnitTests/Utilities_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TerminalLoggerConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TerminalLogger_Tests.cs | Remove Verify-specific attribute usage and xUnit v2 abstractions import. |
| src/Build.UnitTests/Telemetry/Telemetry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TaskHostFactoryLifecycle_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Resources_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ProjectCache/ProjectCacheTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/NodeStatus_Transition_Tests.cs | Remove Verify-specific attribute usage. |
| src/Build.UnitTests/NodeStatus_SizeChange_Tests.cs | Remove Verify-specific attribute usage. |
| src/Build.UnitTests/NetTaskHost_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/MSBuildTaskHostTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Microsoft.Build.Engine.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/Build.UnitTests/InvalidProjectFileException_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Instance/ProjectInstance_Internal_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/ResultCacheBasedBuilds_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/ProjectGraph_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/IsolateProjects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/GraphLoadedFromSolution_tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ExpressionTreeExpression_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/EvaluationProfiler_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/SdkResultEvaluation_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/ExpanderFunction_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/Expander_Tests.cs | Remove xUnit v2 abstractions import; reset change-wave state for stability. |
| src/Build.UnitTests/EscapingInProjects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/EndToEndCondition_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionProjectGenerator_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFilter_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFile_OldParser_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFile_NewParser_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/ProjectRootElement_Tests.cs | Clear cached read-only flags to prevent cross-test pollution in same process. |
| src/Build.UnitTests/ConsoleLogger_Tests.cs | Ensure Traits.LogAllEnvironmentVariables is restored after tests. |
| src/Build.UnitTests/CLR2TaskHost_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ChangeWaves_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BinaryLogger_Tests.cs | Stabilize test behavior around Traits.Instance state and environment variables. |
| src/Build.UnitTests/BackEnd/TaskRouter_IntegrationTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskRegistry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskHostFactory_Tests.cs | Adjust assembly location resolution logic for <UsingTask AssemblyFile=...>. |
| src/Build.UnitTests/BackEnd/TaskHostCallback_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskHost_MultiThreadableTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskBuilder_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TargetUpToDateChecker_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TargetEntry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/SdkResultOutOfProc_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/SdkResolverLoader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/MSBuild_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/LoggingContext_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/DebugUtils_tests.cs | Avoid Directory.GetFiles when debug dump dir doesn’t exist. |
| src/Build.UnitTests/BackEnd/BuildRequestConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/BuildManager_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/BuildManager_Logging_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/AssemblyTaskFactory_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/AppHostSupport_Tests.cs | Ensure DOTNET_ROOT_X86 is tracked for restoration in tests. |
| src/Build.OM.UnitTests/NugetRestoreTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Instance/ProjectInstance_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Definition/ProjectCollection_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Definition/Project_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Construction/ProjectMetadataElement_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Construction/ProjectFormatting_Tests.cs | Remove xUnit v2 abstractions import. |
| global.json | Configure test runner as Microsoft.Testing.Platform. |
| eng/Version.Details.xml | Rename Arcade dependency to Microsoft.DotNet.XUnitV3Extensions. |
| eng/Version.Details.props | Update Arcade xUnit v3 extensions version properties. |
| eng/dependabot/Directory.Packages.props | Add MTP coverage package; switch Verify dependency to Verify.XunitV3. |
| Directory.Packages.props | Switch xUnit extensions package and remove xunit.console version pin. |
| Directory.Build.props | Add xUnit v3/MTP-related properties (TestRunnerName, versions, OutputType for test projects, analyzer suppression). |
| CoverageWindowsFull.runsettings | Removed legacy VS CodeCoverage runsettings. |
| CoverageWindowsFull.config | Added new coverage configuration format for MTP coverage integration. |
| Coverage.runsettings | Removed legacy non-Windows runsettings. |
| Coverage.config | Added new non-Windows coverage configuration format. |
9b1b90b to
703e681
Compare
|
/review |
|
✅ Expert Code Review (command) completed successfully! |
|
Additional findings from deeper static analysis of the xUnit v3 migration ( 🔴
|
Pulling some changes from #13215
…auses some caches to be bad/unexpected for tests.
dfa3b19 to
15f93d6
Compare
|
Copilot concern for Reflection with on guard is also fine and pre-existing. |
|
Ping @rainersigwald @JanProvaznik @ViktorHofer for reviews please. |
No description provided.