IlcCompiler never ends

[timcassell]

Description

I was following the guide at https://devblogs.microsoft.com/dotnet/creating-aot-compatible-libraries/ to make sure my library is AOT compatible. When I try to build and publish, the IlcCompile step never ends.

Reproduction Steps

Create a library project, add this type:

public readonly struct TypeBuilder<T1, T2>
{
    public TypeBuilder<(T1, T2), T3> Add<T3>() => throw null;
}

Create an exe project like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net8.0</TargetFrameworks>
    <PublishAot>true</PublishAot>
    <TrimmerSingleWarn>false</TrimmerSingleWarn>
    <SelfContained>true</SelfContained>
    <AnalysisLevel>latest-all</AnalysisLevel>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\TypeBuilder\TypeBuilder.csproj" />
    <TrimmerRootAssembly Include="TypeBuilder" />
  </ItemGroup>

</Project>

Run dotnet publish -c Release -f net8.0

Expected behavior

Compilation ends.

Actual behavior

Compilation runs forever (I let it run for 1.5 hours before I killed it).

Regression?

No response

Known Workarounds

No response

Configuration

.Net 9.0.203
Windows 10
Ryzen 9800X3D

Other information

Consuming this library works just fine in normal projects, it seems to only be an issue with TrimmerRootAssembly used to verify AOT compatibility.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[teo-tsirpanis]

Performance regressions of some kind by the use of TrimmerRootAssembly seem by design. I guess here it caused NAOT to compile TypeBuilder for all possible generic parameter permutations of types that exist in the assembly.

I reproduced it locally and saw ilc being in deeply recursive calls of ilc.exe!ILCompiler_Compiler_ILCompiler_DependencyAnalysis_NodeFactory_NativeLayoutHelper__TemplateConstructableTypes_d__3__MoveNext+0x61a first and ilc.exe!ILCompiler_TypeSystem_Internal_TypeSystem_InstantiatedType__IsCanonicalSubtype+0x2b on top of it.

[agocke]

Trimmer root assembly is part of our guidance, so if it’s hitting problems that wouldn’t be hit in practice, we’ll need another solution for verifying warnings.

[MichalStrehovsky]

This is a generic recursion that we likely fail to detect. TypeBuilder<T1, T2>.Add<T3> would force TypeBuilder<(T1, T2), T3> into existence (because the method is a target of reflection and we might need to box the return value of the method like we do for all reflection targets) and that likely forces instantiation of Add on that type into existence. We miss a call to generic recursion detector somewhere. This should generate the generic recursion warning and move on.

[MichalStrehovsky]

Turns out I fixed this one in March as part of #111610. I ran into the missing recursion detection too in some of our tests. Latest .NET 10 public previews should already have the fix.

The fix was this line:

runtime/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs

Line 100 in b683862

factory.TypeSystemContext.DetectGenericCycles(type, referent);

I'll add a more targeted regression test and resolve this with that.