Add end-to-end tests for Compatibility tooling and use Runtime=NET for apicompat and genapi tasks#54205
Open
ViktorHofer wants to merge 24 commits into
Open
Add end-to-end tests for Compatibility tooling and use Runtime=NET for apicompat and genapi tasks#54205ViktorHofer wants to merge 24 commits into
ViktorHofer wants to merge 24 commits into
Conversation
The ApiCompat and GenAPI MSBuild tasks no longer multi-target
.NETFramework. They now build only against $(NetMinimum) and the
<UsingTask> declarations use Runtime=NET` so MSBuild on .NET
Framework launches a .NET Core taskhost to run the tasks.
Changes:
* src/Compatibility projects (ApiCompat.Task, GenAPI.Task,
ApiCompatibility, ApiSymbolExtensions, PackageValidation, GenAPI):
switch from <TargetFrameworks> to a single <TargetFramework>
$(NetMinimum), drop the _AddBuildOutputToPackageDesktop pack
target and the ools/netframework/ package path, and remove
.NETFramework-conditional ItemGroups.
* build/Microsoft.DotNet.{ApiCompat,GenAPI}.Task.targets and the
in-tree Microsoft.NET.ApiCompat(.Common).targets: drop the
MSBuildRuntimeType conditional on the task assembly path, point at
the ools/net* location only, and add Runtime=NET` on every
<UsingTask>.
* Remove dead #if !NET / #if NET ... #else blocks that handled the
.NET Framework code paths (RoslynResolver, ValidateAssemblies,
CSharpFileBuilder, GenAPIApp, MetadataInformation,
TargetFrameworkFilter, CompatibleTFMValidator, DocIdSymbolFilter).
* Microsoft.NET.Build.Tasks: condition the ProjectReference to
ApiCompat.Task on '$(TargetFrameworkIdentifier)' == '.NETCoreApp'
so the net472 inner build no longer requires it (the targets only
reference ools/net11.0/).
* Tests: drop the net472 inner build from the affected test projects
and from the TestFullMSBuild=true list in test/UnitTests.proj.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restructure Microsoft.DotNet.ApiCompat.IntegrationTests so it exercises
only the customer-facing layer (CLI invocation and MSBuild-target
invocation against a real test project) and runs on every PR under both
.NET Core MSBuild and desktop MSBuild.
* Multi-target the IntegrationTests project to
$(SdkTargetFramework);$(NetFrameworkToolCurrent) so the net472
test runtime exercises the Runtime="NET" taskhost path.
* Drop the netcore-only ProjectReferences (PackageValidation,
ApiCompat.Task as compile reference) and the
Microsoft.CodeAnalysis.CSharp PackageReference. Add
ReferenceOutputAssembly="false" build-ordering deps on
ApiCompat.Task and ApiCompat.Tool with
SetTargetFramework=TargetFramework=$(NetMinimum) so the netcore
build of the netcore-only projects is consumed regardless of the
inner build.
* Move the in-process validator tests to
Microsoft.DotNet.PackageValidation.Tests:
- CompatibleFrameworkInPackageValidatorIntegrationTests.cs ->
CompatibleFrameworkInPackageValidatorTests.cs
- The five in-process methods of ValidatePackageTargetIntegrationTests
(ValidatePackageWithReferences, ValidateOnlyErrorWhenAReferenceIsRequired,
ValidateErrorWhenTypeForwardingReferences,
ValidateMissingReferencesIsOnlyLoggedWhenRunningWithReferences,
ValidateReferencesAreRespectedForPlatformSpecificTFMs) ->
ValidatePackageInProcessTests.cs
* Trim ValidatePackageTargetIntegrationTests.cs to keep only the
pack-the-asset/assert-stdout tests that exercise the SDK's
EnablePackageValidation pipeline end-to-end.
* Drop the dead CopyApiCompatFilesForTests target.
* Update test/UnitTests.proj: drop the
Condition="'$(TestFullMSBuild)' == 'true'" gate on the
IntegrationTests entry so it runs on every PR with both the default
$(SdkTargetFramework) row and an unconditional
TargetFramework=net472/RuntimeTargetFramework=net472 row that drives
desktop MSBuild. Pre-list the soon-to-be-added GenAPI.IntegrationTests
in the same desktop-MSBuild block.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fill the empty ValidateAssembliesTargetIntegrationTests stub and add a
new ApiCompatToolIntegrationTests file under
Microsoft.DotNet.ApiCompat.IntegrationTests so the customer-facing
behavior of both the ValidateAssemblies MSBuild target and the
`apicompat` CLI is exercised on every PR (under both .NET Core MSBuild
and desktop MSBuild for the MSBuild task tests).
* New static asset
test/TestAssets/TestProjects/ApiCompatValidateAssembliesTestProject/
- simple library with #if-gated `Goodbye` and `Welcome` members so a
single source can stand in for both a contract DLL and an
implementation DLL with controlled differences.
* New helpers in the IntegrationTests project:
- NuGetConfigHelper writes a NuGet.Config in a test working directory
that lists `SdkTestContext.Current.TestPackages` as a feed so the
test asset can `dotnet add package Microsoft.DotNet.ApiCompat.Task
--prerelease` and pick up the locally-built dev-versioned .nupkg.
- ToolPaths resolves the published `apicompat` (and later GenAPI /
ApiDiff) tool DLLs in artifacts/bin so the tests can invoke them
via `dotnet exec`.
* Route Microsoft.DotNet.ApiCompat.Task and
Microsoft.DotNet.GenAPI.Task through
test/Microsoft.NET.TestFramework/BuildTestPackages.targets so their
.nupkg files land in artifacts/tmp/<config>/testing/testpackages/,
mirroring how Microsoft.TemplateEngine.Authoring.Tasks is staged.
* ValidateAssembliesTargetIntegrationTests covers
NoBreakingChanges_Succeeds, BreakingChange_FailsWithCP0002,
StrictMode_FailsOnAddition, and GeneratesAndConsumesSuppressionFile.
* ApiCompatToolIntegrationTests covers AssembliesIdentical_ExitsZero,
BreakingChange_ReportsCP0002, SuppressionFile_RoundTrip, and
PackageMode_DetectsRemovedApi (reusing the existing
PackageValidationTestProject asset).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two new IntegrationTests projects mirroring
Microsoft.DotNet.ApiCompat.IntegrationTests, one per remaining tool.
* Microsoft.DotNet.GenAPI.IntegrationTests
- Multi-targets $(SdkTargetFramework);$(NetFrameworkToolCurrent) and
is listed in test/UnitTests.proj for both the default and the
net472 desktop-MSBuild row so the GenAPITask Runtime="NET" path is
exercised on every PR.
- Task/GenAPITaskIntegrationTests.cs: copies the new
GenAPITaskTestProject asset to a temp dir, writes a NuGet.Config
listing the local testpackages feed, and `dotnet add package
Microsoft.DotNet.GenAPI.Task --prerelease` before exercising
GenAPIGenerateReferenceAssemblySource on build, the
/t:GenAPIGenerateReferenceAssemblySource direct-target invocation
path, and the GenAPIRespectInternals=true flag.
- Tool/GenAPIToolIntegrationTests.cs: invokes
`dotnet exec Microsoft.DotNet.GenAPI.Tool.dll --assembly ...`
against an on-the-fly TestProject library and asserts on the
generated reference source for the default, --header-file, and
--respect-internals scenarios.
* Microsoft.DotNet.ApiDiff.IntegrationTests
- Single-targets $(SdkTargetFramework) since ApiDiff has no MSBuild
task to exercise from desktop MSBuild. Picked up only by the
default *IntegrationTests glob in test/UnitTests.proj.
- Tool/ApiDiffToolIntegrationTests.cs: invokes
`dotnet exec Microsoft.DotNet.ApiDiff.Tool.dll` against pairs of
built assembly folders and asserts on the produced markdown diff
for NoChanges, AddedMember, TableOfContentsTitle, and
MissingRequiredOption scenarios. CLI argument parsing is covered
end-to-end by these integration tests, so no separate
RootCommand-driving unit test is added.
* New static asset test/TestAssets/TestProjects/GenAPITaskTestProject/
- small library with public Calculator class and an internal member
so tests can assert filter behavior.
* New helpers per project (small, deliberate inline copies):
- NuGetConfigHelper writes a NuGet.Config that lists
SdkTestContext.Current.TestPackages as a feed.
- ToolPaths resolves the published tool DLL in
artifacts/bin/<Tool>/<Configuration>/<TFM>/<Tool>.dll.
* Add the new projects to sdk.slnx and
src/Compatibility/compatibility.slnf.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Local execution against the freshly built redist SDK uncovered three
issues. All test projects now pass on every supported runner.
1. ToolPaths in the three IntegrationTests projects hard-coded the tool
DLL location at `artifacts/bin/<Tool>/<config>/net<TFM>/<Tool>.dll`
using `ToolsetInfo.CurrentTargetFrameworkVersion`. The compatibility
tools target `$(NetMinimum)` (currently net10.0), not the SDK's
target framework (net11.0), so resolution failed. Replace the
hard-coded path with a recursive search under the tool's bin dir
for `<Tool>.dll`, skipping the `publish/` subdirectory. This also
makes the helpers resilient to NetMinimum bumps.
2. ApiCompatTool_BreakingChange_ReportsCP0002 and
ApiCompatTool_PackageMode_DetectsRemovedApi asserted on
`result.StdOut` but the apicompat CLI writes CP* errors to stderr
via `Reporter.Error`. Assert on the combined StdOut + StdErr so the
test verifies what the customer actually sees on the console.
3. ApiDiffTool_AddedMember_ProducesDiff expected the per-assembly
markdown at `<output>/MyLib.md`, but FileOutputDiffGenerator names
files `<tableOfContentsTitle>_<assemblyName>.md` (so the default is
`api_diff_MyLib.md`). Pass an explicit `--tableOfContentsTitle`
and assert on the matching name.
4. ApiDiffTool_MissingRequiredOption_FailsWithHelpfulError assumed a
non-zero exit code when a required option is omitted, but
System.CommandLine prints the error and the help text and exits 0.
Rename to *_ReportsHelpfulError and assert on the stderr message
text rather than the exit code.
Local verification (build.cmd then `dotnet test --no-build` after
sourcing `artifacts\sdk-build-env.bat`):
Microsoft.DotNet.ApiCompat.IntegrationTests net11.0 16/16 passed
Microsoft.DotNet.ApiCompat.IntegrationTests net472 16/16 passed
Microsoft.DotNet.GenAPI.IntegrationTests net11.0 6/6 passed
Microsoft.DotNet.GenAPI.IntegrationTests net472 6/6 passed
Microsoft.DotNet.ApiDiff.IntegrationTests net11.0 4/4 passed
Microsoft.DotNet.PackageValidation.Tests net11.0 46/46 passed
--------------
94/94 passed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `apidiff` CLI was written as
public static async Task Main(string[] args)
{
...
await rootCommand.Parse(args).InvokeAsync();
}
`Main` returned `Task` (not `Task<int>`) and the result of
`InvokeAsync()` (which returns the parser/handler exit code) was
discarded. As a consequence, when a required option such as `--output`
was missing, System.CommandLine printed `Option '--output' is required.`
and the help text but the process always exited with code 0 — so
nothing in CI or a calling script could tell that the invocation failed.
`apicompat` and `genapi` use the synchronous form
static int Main(string[] args) => rootCommand.Parse(args).Invoke();
and propagate the exit code correctly. Mirror that behavior in apidiff
by changing the return type to `Task<int>` and returning
`InvokeAsync()`'s result.
Also flip the corresponding integration test from
`*_ReportsHelpfulError` (which only asserted on the stderr text) back
to `*_FailsWithHelpfulError` so it now also asserts on the non-zero
exit code, locking in the fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ViktorHofer
changed the title
…tests
The RequiresMSBuildVersionFact / RequiresMSBuildVersionTheory attributes
in the compatibility test projects were either copy-pasted without a
real version requirement (e.g. "17.0.0.32901" - a 17.0 RC2 build
threshold that has been generally available for years and is always
satisfied today) or carried a stale "Needs System.Text.Json 8.0.5"
reason from a much earlier MSBuild bundle. None of the compatibility
tests actually exercise an MSBuild-specific feature that requires a
particular MSBuild version: the CLI tests invoke published tool DLLs
via `dotnet exec` (no MSBuild involvement) and the MSBuild target tests
run through the freshly built redist SDK, so whatever MSBuild ships
there is always used.
Replace every RequiresMSBuildVersion*(...) usage in the compatibility
test code with plain [Fact] / [Theory] so the tests just run on
whatever MSBuild the test toolset provides.
Files touched:
test/Microsoft.DotNet.ApiCompat.IntegrationTests/
Task/ValidateAssembliesTargetIntegrationTests.cs (4 -> Fact)
Task/ValidatePackageTargetIntegrationTests.cs (8 -> Fact)
Tool/ApiCompatToolIntegrationTests.cs (1 -> Fact)
test/Microsoft.DotNet.GenAPI.IntegrationTests/
Task/GenAPITaskIntegrationTests.cs (3 -> Fact)
test/Microsoft.DotNet.PackageValidation.Tests/
CompatibleFrameworkInPackageValidatorTests.cs (2 -> Fact)
ValidatePackageInProcessTests.cs (1 Fact + 3 Theory + 1 Fact -> Fact/Theory)
Local re-run after the change (build.cmd then `dotnet test --no-build`
under the redist SDK):
Microsoft.DotNet.ApiCompat.Tests net11.0 5/5
Microsoft.DotNet.ApiCompatibility.Tests net11.0 246/246
Microsoft.DotNet.ApiDiff.Tests net11.0 149/149 (3 skipped)
Microsoft.DotNet.ApiSymbolExtensions.Tests net11.0 24/24
Microsoft.DotNet.GenAPI.Tests net11.0 90/90 (2 skipped)
Microsoft.DotNet.PackageValidation.Tests net11.0 46/46
Microsoft.DotNet.ApiCompat.IntegrationTests net11.0 16/16
Microsoft.DotNet.ApiCompat.IntegrationTests net472 16/16
Microsoft.DotNet.GenAPI.IntegrationTests net11.0 6/6
Microsoft.DotNet.GenAPI.IntegrationTests net472 6/6
Microsoft.DotNet.ApiDiff.IntegrationTests net11.0 4/4
--------
608/608 passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
Q: any way to see what the size impact to the SDK tarballs would be after this change? Not a blocker in any way, I'm just interested in the overall 'cost' of supporting .NET Framework in our tools. |
Member
Author
|
@copilot investigate the size savings due to the removal of the .net framework apicompat assemblies in the .NET SDK |
Contributor
I measured this on Estimated SDK tarball impact from the removed Compatibility netfx payload is about 9.78 MB compressed (about 28.67 MB uncompressed, 165 files). If you look at ApiCompat alone, it’s about 1.53 MB compressed (about 4.22 MB uncompressed, 81 files).
Investigated on For ApiCompat netframework payload removal in the SDK path:
(If including GenAPI as part of the same Compatibility change, total estimated tar.gz impact is ~9.78 MB.) |
The first CI/Helix run on PR #54205 had every CLI integration test failing with: System.TypeInitializationException : The type initializer for 'ToolPaths' threw an exception. ---- System.InvalidOperationException : Could not locate the repo root from the test working directory. Root cause: ToolPaths used SdkTestContext.GetRepoRoot() to find the source repo and then probed artifacts/bin/<Tool>/<config>/<TFM>/ for the tool DLL. That works locally but the source tree is not present on Helix work item machines, and even if it were, the artifacts are elsewhere on the host. Fix: in each IntegrationTests csproj, add a _StageToolsUnderTest target (AfterTargets="Build") that copies the tool's bin output into $(OutputPath)tools\<ToolName>\, so the tool DLL closure rides along with the test assembly into the Helix work item payload. Then ToolPaths just resolves Path.Combine(AppContext.BaseDirectory, "tools", "<ToolName>", "<ToolName>.dll") which works in every environment. Files: test/Microsoft.DotNet.ApiCompat.IntegrationTests/ Microsoft.DotNet.ApiCompat.IntegrationTests.csproj (+ _StageToolsUnderTest) ToolPaths.cs (rewrite) test/Microsoft.DotNet.GenAPI.IntegrationTests/ Microsoft.DotNet.GenAPI.IntegrationTests.csproj (+ _StageToolsUnderTest) ToolPaths.cs (rewrite) test/Microsoft.DotNet.ApiDiff.IntegrationTests/ Microsoft.DotNet.ApiDiff.IntegrationTests.csproj (+ _StageToolsUnderTest) ToolPaths.cs (rewrite) Verified locally: build.cmd then `dotnet test --no-build --framework net11.0 --filter ApiCompatTool` reports 4/4 passing, and artifacts/bin/Microsoft.DotNet.ApiCompat.IntegrationTests/Debug/net11.0/tools/ contains the staged Microsoft.DotNet.ApiCompat.Tool/ closure. The GenAPI Task tests on the previous Helix run already passed (the testpackages feed plumbing for Microsoft.DotNet.GenAPI.Task installs correctly from /datadisks/.../Testpackages on Helix), so no other adjustments are needed for the work-item layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
@baronfel only apicompat is part of the .NET SDK so the answer to your question is the 1.53MB compressed number. |
The first attempt staged the tool DLL closure under the test's
\$(OutputPath)tools\<ToolName>\ via an AfterTargets="Build" Copy task.
That made `dotnet test --no-build` work locally but the Helix work
items still failed at type-init with FileNotFoundException pointing at
.../w/<workitem>/e/tools/Microsoft.DotNet.ApiCompat.Tool/Microsoft.DotNet.ApiCompat.Tool.dll
because the work item payload is built from each test project's
PublishDir, not OutputPath. (XUnitRunner.targets calls a custom
PublishWithOutput target on each test csproj and feeds the resulting
publish directory into SDKCustomCreateXUnitWorkItemsWithTestExclusion
as PayloadDirectory.)
Fix: switch each IntegrationTests csproj to dynamically add `<Content>`
items in a target that runs BeforeTargets="AssignTargetPaths", with
both CopyToOutputDirectory="PreserveNewest" and
CopyToPublishDirectory="PreserveNewest". The Content items use
Link="tools\<ToolName>\..." so the tool closure ends up under
tools/<ToolName>/ in both the bin output and the publish output.
Materialize the source files into an intermediate item first
(_<Tool>File) and reference them via @() so the per-item metadata
expressions for Link expand correctly inside the target (they don't
reliably expand on a Content item created from a wildcard inside a
target body).
Verified locally:
- artifacts/bin/.../IntegrationTests/Debug/net11.0/tools/<Tool>/<Tool>.dll exists
- artifacts/bin/.../IntegrationTests/Debug/net11.0/publish/tools/<Tool>/<Tool>.dll exists
- `dotnet test --no-build --framework net11.0 --filter ApiCompatTool`
reports 4/4 passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous round shipped them under an unconditional ItemGroup, which
sent the net472 work items to every test leg. On the Linux / macOS
TestBuild queues that fails immediately — the Helix runner's
`chmod +x Microsoft.DotNet.ApiCompat.IntegrationTests` step looks for
the test exe (without extension) and net472 doesn't produce one for
those platforms, so the work item exits 1 before xunit even starts.
Re-gate the row on TestFullMSBuild=true (the Windows FullFramework leg
sets it). FullFramework runs on every PR, so .NET Framework / desktop
MSBuild coverage is still part of every PR build, but the net472 work
items now only land on the Windows queue where they can actually run.
The default *IntegrationTests glob continues to schedule the
.NETCoreApp run on every leg, so cross-platform .NETCoreApp coverage is
unchanged.
Linux/macOS TestBuild legs in the prior CI run actually showed the
.NETCoreApp work items (.dll, no .netfx) PASSING; only the .netfx
work items failed there with the chmod issue. This change removes
those incorrectly-scheduled .netfx work items.
Note: a separate FullFramework Helix failure remains
("MSBUILD : error : The task failed to load because it requires the
MSBuild .NET Runtime Task Host, but the .NET Runtime Task Host could
not be found for the specified version" — see aka.ms/nettaskhost).
That's the Runtime="NET" taskhost path failing to locate the .NET
Core taskhost in the FullFramework Helix environment and is the next
thing to investigate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ask host
The FullFramework Windows Helix leg failed every test that triggered a
PackageValidationTask invocation through `EnablePackageValidation=true`
with:
MSBUILD : error : The task failed to load because it requires the
MSBuild .NET Runtime Task Host, but the
.NET Runtime Task Host could not be found for the
specified version.
See https://aka.ms/nettaskhost.
Tracing through MSBuild's AssemblyTaskFactory:
private static TaskHostParameters AddNetHostParamsIfNeeded(
in TaskHostParameters currentParams,
Func<string, ProjectPropertyInstance> getProperty)
{
...
string dotnetHostPath = getProperty(Constants.DotnetHostPathEnvVarName)?.EvaluatedValue;
string netCoreSdkRoot = getProperty(Constants.NetCoreSdkRoot)?.EvaluatedValue?.TrimEnd('/', '\\');
...
// Both DOTNET_HOST_PATH and NetCoreSdkRoot are required to launch .NET task host.
if (string.IsNullOrEmpty(dotnetHostPath) || string.IsNullOrEmpty(netCoreSdkRoot))
{
return currentParams;
}
...
}
When MSBuild can't populate `MSBuildAssemblyPath` from those two
properties, GetMSBuildPath throws NETHostTaskLoad_Failed on .NET
Framework MSBuild (the #if NET branch only exists on .NET Core).
`dotnet.exe` propagates `DOTNET_HOST_PATH` to the MSBuild it spawns;
`msbuild.exe` does not. SetupHelixEnvironment.cmd was setting
`DOTNET_ROOT` and prepending it to PATH but never setting
`DOTNET_HOST_PATH`, so on the FullFramework leg (which uses desktop
`msbuild.exe` for the inner Pack/Build invocations) the .NET runtime
task host could not be located and the SDK's in-tree
`Microsoft.DotNet.ApiCompat.Task` (now netcore-only after this PR's
.NET Framework removal) failed to load.
Fix: set `DOTNET_HOST_PATH=%DOTNET_ROOT%\dotnet.exe` in
SetupHelixEnvironment.cmd. MSBuild auto-imports environment variables
as evaluation-time properties, so this satisfies the
`getProperty("DOTNET_HOST_PATH")` lookup. `NetCoreSdkRoot` already
ships with .NET 11 SDK and is set by Microsoft.NET.Sdk itself.
The .sh script doesn't need the same change because the affected
Linux/macOS work items go through `dotnet test` (which uses the
.NET CLI host that sets `DOTNET_HOST_PATH` automatically); the
desktop-MSBuild path only runs on the FullFramework Windows leg.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nt.cmd Add a REM comment above the DOTNET_HOST_PATH export documenting why it's needed (so a future reader doesn't drop the line as 'looks duplicated with DOTNET_ROOT'). The actual fix shipped in the previous commit; this is comment-only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a BeforeTargets=Build;Pack target to PackageValidationTestProject that dumps the MSBuild properties consulted by AssemblyTaskFactory.AddNetHostParamsIfNeeded (DOTNET_HOST_PATH, NetCoreSdkRoot, RuntimeIdentifierGraphPath, MSBuildSDKsPath). Setting DOTNET_HOST_PATH in the Helix env var didn't unblock the failure, so we need to see which property MSBuild is actually evaluating as empty before deciding the next fix. Will be removed once the FullFramework leg is green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The harness was wiping DOTNET_HOST_PATH from the msbuild.exe environment to mirror the customer scenario where bare msbuild.exe is launched outside a Developer Command Prompt. That clear was added in 2019 by #3192 ("Locate dotnet host for illink on desktop") to expose ILLink's dependence on the env var, and ILLink got a project-level fallback at the same time. That customer scenario no longer reflects reality. Since #50066 (commit ea55de9, 2025-07-19, "set env vars from resolver APIs"), Microsoft.DotNet.MSBuildSdkResolver returns DOTNET_HOST_PATH via the EnvironmentVariablesToAdd API whenever it resolves Microsoft.NET.Sdk. The MSBuild engine then injects it into the resolver's process environment, so by the time tasks (Csc, Microsoft.DotNet.ApiCompat.Task, etc.) execute, DOTNET_HOST_PATH is set even if msbuild.exe was launched without it. Real customers running bare msbuild.exe today are auto-fixed by the resolver; Arcade's eng/common scripts also rely on this and never set DOTNET_HOST_PATH explicitly. Removing the clear restores natural propagation through the test harness: dotnet exec sets DOTNET_HOST_PATH on the test process (per Muxer.cs), the test process inherits it, and msbuild.exe inherits it from the test process. This unblocks .NETCoreApp tasks declared with `Runtime="NET"` (notably the new ApiCompat / GenAPI tasks introduced when removing .NET Framework support from src/Compatibility) when running on the FullFramework Helix leg, without needing per-test env-var workarounds. Also revert two ineffective workarounds previously added while diagnosing this: - The DOTNET_HOST_PATH set in build/SetupHelixEnvironment.cmd (it never reached msbuild.exe because the harness immediately cleared it). - The temporary _DiagnoseTaskHostProps target in test/TestAssets/TestProjects/PackageValidationTestProject. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ericstj
reviewed
May 8, 2026
ericstj
reviewed
May 8, 2026
ericstj
reviewed
May 8, 2026
The ApiCompat / GenAPI / ApiDiff CLI tools target $(NetMinimum) (net10.0).
On the macOS arm64 Helix queue, the bundled Microsoft.NETCore.App/10.0.0
shared framework is x86_64-only, so dlopen of libhostpolicy.dylib fails
with an architecture mismatch when the tools are exec'd via the redist
SDK on an arm64 host:
mach-o file, but is an incompatible architecture
(have 'x86_64', need 'arm64e' or 'arm64')
<RollForward>Major</RollForward> in each tool csproj does not help here:
the host successfully resolves 10.0.0 (it exists on disk), then fails on
load. Roll-forward only triggers when no matching version is found, not
on architecture mismatch.
Pass --roll-forward LatestMajor as a host-level option in the integration
tests that exec the tool DLLs. The redist SDK's dotnet host
(DotnetCommand.FileName = ToolsetUnderTest.DotNetHostPath) then loads the
tool under its own net11.0 runtime, which is arm64-correct.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet host treats arguments before the verb as a tool-name lookup, so `dotnet --roll-forward LatestMajor exec foo.dll` was resolved as a missing `dotnet--roll-forward` tool. Move the host option to follow `exec` per the documented `dotnet exec [options] <PATH>` form so the roll-forward actually applies on macOS arm64 (its original purpose) without breaking other legs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
--roll-forward LatestMajor alone won't roll forward to a prerelease major version. The redist SDK ships net11.0 as 11.0.0-preview.x, so without DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 the host stays on the only release-quality framework available (net10.0), which is x86_64-only on the macOS arm64 helix payload and fails to dlopen libhostpolicy.dylib. Set DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 in the spawned dotnet process so the roll-forward actually reaches the redist SDK's net11.0 prerelease runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses Eric's review feedback on #54205: - Add README.md to Microsoft.DotNet.ApiCompat.Task and Microsoft.DotNet.GenAPI.Task packages documenting the desktop MSBuild requirement (>= 18.0, equivalently .NET SDK >= 10.0.100) and the failure modes consumers hit on older toolsets. - Drop the redundant `net` segment from the task package layout (`tools/net/X.dll` -> `tools/X.dll`) and update the targets that reference the task assembly to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The per-test NuGet.config is layered on top of the surrounding environment's NuGet.config (helix payload's pre-curated config or developer's local config), which already provides nuget.org. We only need to add the locally-built testpackages feed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ectSources Replaces the per-asset NuGet.config writes (NuGetConfigHelper.WriteNuGetConfigWithTestPackages) with MSBuild/CLI properties: - `dotnet add package` is given `--source SdkTestContext.Current.TestPackages`. - Subsequent `BuildCommand.Execute` calls that need to restore the dev-versioned Microsoft.DotNet.ApiCompat.Task / GenAPI.Task package pass `-p:RestoreAdditionalProjectSources=...` so restore can find it on the local feed. Deletes both NuGetConfigHelper.cs files and removes the no-longer-needed using import. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3 tasks
Forcing --roll-forward LatestMajor + DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 made the tests stop exercising the customer-facing minimum runtime (net10.0) and instead always run on the SDK's bundled prerelease 11.x. Revert to plain `dotnet exec ToolDll` so the tests cover what real customers will hit, and skip the four ApiCompat / three GenAPI / four ApiDiff tool tests on macOS arm64 only via [PlatformSpecificFact(skipPlatforms: TestPlatforms.OSX, skipArchitecture: Architecture.Arm64)] linking to #54248 (the helix arm64 payload bundles an x86_64-only Microsoft.NETCore.App/10.0.0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modernizes the Compatibility tooling (ApiCompat / GenAPI tasks and tools) by dropping .NET Framework targeting and relying on MSBuild’s Runtime="NET" task hosting, while adding customer-layer end-to-end integration tests for the tools/tasks and fixing apidiff’s exit-code propagation.
Changes:
- Switch ApiCompat/GenAPI tasks to be loaded via
UsingTask ... Runtime="NET"and remove .NET Framework-specific targets/polyfills. - Add/restructure integration tests (ApiCompat, new GenAPI, new ApiDiff) plus supporting test assets and test package build plumbing.
- Fix
apidiffto return a non-zero exit code on parser/CLI errors.
Show a summary per file
| File | Description |
|---|---|
| test/UnitTests.proj | Ensure compatibility integration tests run in PR legs (incl. net472 row under full framework leg). |
| test/TestAssets/TestProjects/GenAPITaskTestProject/NuGet.Config | Add isolated NuGet config for test asset. |
| test/TestAssets/TestProjects/GenAPITaskTestProject/GenAPITaskTestProject.csproj | Add GenAPI test asset project. |
| test/TestAssets/TestProjects/GenAPITaskTestProject/Calculator.cs | Add GenAPI test asset source. |
| test/TestAssets/TestProjects/ApiCompatValidateAssembliesTestProject/NuGet.Config | Add isolated NuGet config for test asset. |
| test/TestAssets/TestProjects/ApiCompatValidateAssembliesTestProject/Greeter.cs | Add ApiCompat ValidateAssemblies test asset source. |
| test/TestAssets/TestProjects/ApiCompatValidateAssembliesTestProject/ApiCompatValidateAssembliesTestProject.csproj | Add ApiCompat ValidateAssemblies test asset project. |
| test/Microsoft.NET.TestFramework/ToolsetInfo.cs | Adjust environment behavior for full framework MSBuild invocation. |
| test/Microsoft.NET.TestFramework/BuildTestPackages.targets | Build compatibility task packages into test feed. |
| test/Microsoft.DotNet.PackageValidation.Tests/ValidatePackageInProcessTests.cs | Move/add in-process package validation tests. |
| test/Microsoft.DotNet.PackageValidation.Tests/Microsoft.DotNet.PackageValidation.Tests.csproj | Single-target PackageValidation tests on SDK TFM. |
| test/Microsoft.DotNet.PackageValidation.Tests/CompatibleFrameworkInPackageValidatorTests.cs | Move/rename validator tests into PackageValidation test project. |
| test/Microsoft.DotNet.GenAPI.IntegrationTests/ToolPaths.cs | Resolve staged genapi tool path from test output. |
| test/Microsoft.DotNet.GenAPI.IntegrationTests/Tool/GenAPIToolIntegrationTests.cs | Add E2E CLI tests for genapi. |
| test/Microsoft.DotNet.GenAPI.IntegrationTests/Task/GenAPITaskIntegrationTests.cs | Add E2E MSBuild-task tests for GenAPI package. |
| test/Microsoft.DotNet.GenAPI.IntegrationTests/Microsoft.DotNet.GenAPI.IntegrationTests.csproj | New GenAPI integration test project + tool staging. |
| test/Microsoft.DotNet.ApiSymbolExtensions.Tests/Microsoft.DotNet.ApiSymbolExtensions.Tests.csproj | Single-target ApiSymbolExtensions tests on SDK TFM. |
| test/Microsoft.DotNet.ApiDiff.IntegrationTests/ToolPaths.cs | Resolve staged apidiff tool path from test output. |
| test/Microsoft.DotNet.ApiDiff.IntegrationTests/Tool/ApiDiffToolIntegrationTests.cs | Add E2E CLI tests for apidiff. |
| test/Microsoft.DotNet.ApiDiff.IntegrationTests/Microsoft.DotNet.ApiDiff.IntegrationTests.csproj | New ApiDiff integration test project + tool staging. |
| test/Microsoft.DotNet.ApiCompatibility.Tests/Microsoft.DotNet.ApiCompatibility.Tests.csproj | Single-target ApiCompatibility tests on SDK TFM. |
| test/Microsoft.DotNet.ApiCompat.Tests/Microsoft.DotNet.ApiCompat.Tests.csproj | Single-target ApiCompat tests on SDK TFM. |
| test/Microsoft.DotNet.ApiCompat.IntegrationTests/ToolPaths.cs | Resolve staged apicompat tool path from test output. |
| test/Microsoft.DotNet.ApiCompat.IntegrationTests/Tool/ApiCompatToolIntegrationTests.cs | Add E2E CLI tests for apicompat. |
| test/Microsoft.DotNet.ApiCompat.IntegrationTests/Task/ValidatePackageTargetIntegrationTests.cs | Refocus ApiCompat task integration tests on MSBuild target behavior. |
| test/Microsoft.DotNet.ApiCompat.IntegrationTests/Task/ValidateAssembliesTargetIntegrationTests.cs | Add E2E ValidateAssemblies target coverage (strict/suppressions). |
| test/Microsoft.DotNet.ApiCompat.IntegrationTests/Microsoft.DotNet.ApiCompat.IntegrationTests.csproj | Restructure ApiCompat integration tests + tool staging. |
| src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.ApiCompat.targets | Point SDK ApiCompat task assembly to net11-only payload. |
| src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.ApiCompat.Common.targets | Load ApiCompat tasks via Runtime="NET". |
| src/Tasks/Microsoft.NET.Build.Tasks/Microsoft.NET.Build.Tasks.csproj | Avoid referencing ApiCompat task project on .NET Framework build. |
| src/Compatibility/Microsoft.DotNet.ApiSymbolExtensions/Microsoft.DotNet.ApiSymbolExtensions.csproj | Single-target ApiSymbolExtensions to NetMinimum. |
| src/Compatibility/Microsoft.DotNet.ApiSymbolExtensions/Filtering/DocIdSymbolFilter.cs | Remove .NET Framework conditional code path. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI/Microsoft.DotNet.GenAPI.csproj | Single-target GenAPI library to NetMinimum. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI/GenAPIApp.cs | Remove .NET Framework-specific using/branching. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI/CSharpFileBuilder.cs | Use ReplaceLineEndings() unconditionally. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI.Task/README.md | Document GenAPI task package requirements + Runtime=NET hosting. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI.Task/Microsoft.DotNet.GenAPI.Task.csproj | Single-target + adjust packing layout/readme for GenAPI task package. |
| src/Compatibility/GenAPI/Microsoft.DotNet.GenAPI.Task/build/Microsoft.DotNet.GenAPI.Task.targets | Use Runtime="NET" for GenAPI task loading. |
| src/Compatibility/compatibility.slnf | Add new integration test projects to compatibility solution filter. |
| src/Compatibility/ApiDiff/Microsoft.DotNet.ApiDiff.Tool/Program.cs | Return Task<int> and propagate System.CommandLine exit code. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.PackageValidation/Validators/CompatibleTFMValidator.cs | Remove .NET Framework conditional dictionary code. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.PackageValidation/Microsoft.DotNet.PackageValidation.csproj | Single-target PackageValidation to NetMinimum. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.PackageValidation/Filtering/TargetFrameworkFilter.cs | Remove .NET Framework conditional wildcard check. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompatibility/Microsoft.DotNet.ApiCompatibility.csproj | Single-target ApiCompatibility to NetMinimum. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompatibility/MetadataInformation.cs | Remove .NET Framework hash-code fallback. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompat.Task/README.md | Document ApiCompat task package requirements + Runtime=NET hosting. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompat.Task/Microsoft.DotNet.ApiCompat.Task.csproj | Single-target + adjust packing layout/readme for ApiCompat task package. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompat.Task/build/Microsoft.DotNet.ApiCompat.Task.targets | Update task assembly path layout for single netcore payload. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompat.Shared/ValidateAssemblies.cs | Remove .NET Framework conditional glob check. |
| src/Compatibility/ApiCompat/Microsoft.DotNet.ApiCompat.Shared/RoslynResolver.cs | Remove AppDomain-based resolver path (netcore-only). |
| sdk.slnx | Add new integration test projects to solution. |
Copilot's findings
Comments suppressed due to low confidence (1)
test/Microsoft.DotNet.PackageValidation.Tests/ValidatePackageInProcessTests.cs:50
- Typo in the generated test source: "ItermediateBaseClass" should be "IntermediateBaseClass" (second occurrence).
- Files reviewed: 51/51 changed files
- Comments generated: 2
…rrorWhenAReferenceIsRequired Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ItermediateBaseClasstypo →IntermediateBaseClass(lines 38 and 49)ValidateOnlyErrorWhenAReferenceIsRequiredso the test verifies behavior