AOT compilation produce stacks overflow

[rodrigovaras]

Description

When I was testing generating an AOT binary for my TUnit test the dotnet publish take a very long time and eds on a StackOverflow.
Extract from the CLI:
Rpc.Json net9.0 succeeded (0.1s) → artifacts\bin\Rpc.Json\debug_net9.0\Teksoft.Rpc.Json.dll
Rpc.Protobuf net9.0 succeeded (0.1s) → artifacts\bin\Rpc.Protobuf\debug_net9.0\Teksoft.Rpc.Proto.dll
Rpc.Protobuf-net net9.0 succeeded (0.1s) → artifacts\bin\Rpc.Protobuf-net\debug_net9.0\Teksoft.Rpc.Proto-net.dll
Rpc.Session net9.0 succeeded (0.2s) → artifacts\bin\Rpc.Session\debug_net9.0\Teksoft.Rpc.Session.dll
Rpc.Test failed with 1 error(s) (474.9s) → test\artifacts\bin\Rpc.Test\debug_win-x64\Rpc.Test.dll
Generating native code

Process is terminating due to StackOverflowException.
C:\Users\rodrigov\.nuget\packages\microsoft.dotnet.ilcompiler\9.0.5\build\Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""C:\Users\rodrigov\.nuget\packages\runtime.win-x64.microsoft.dotnet.ilcompiler\9.0.5\tools\\ilc" @"D:\teksoft\tek.shared.net\test\artifacts\obj\Rpc.Test\debug_win-x64\native\Rpc.Test.ilc.rsp"" exited with code -1073741571.

Reproduction Steps

A project with a dependency of TUnit version 0.21.13 and my rpc libraries.
Using .NET 9 with 9.0.300

Expected behavior

dotnet publish succeed and generate the .exe in the case of Windows

Actual behavior

dotnet publish CLI takes a long time and eventually finished with a StackOverflow

Regression?

Not sure

Known Workarounds

No workarounds

Configuration

.NET 9 with 9.0.300 bits trying on a winx64 runtime id

Other information

No response

[agocke]

Can you produce a minimal repro?

[rodrigovaras]

Can you produce a minimal repro?

Very difficult to reproduce without upload a lot of source code. I have a solution with around 30 projects with unit test. I recently converted to use TUnit in order to run AOT native test. Some of my existing unit tests pass the 'dotnet publish' but this particular does not and not sure what it have that make them fail. I would try removing all the unit tests and then see if a particular one fails. Would be possible to enable a verbose option for the ILC step to explore the logging ?

[rodrigovaras]

I commented all unit test except a relatively simple one and now the ILC is passing. I can paste the full code of the unit test here to see if you see something suspicious. I have more time today I can enable slowly until i found the offending one

[rodrigovaras]

Ok, I investigated and found that by commenting out some code I was able to eventually get the ILC compiler to pass. Then one by one put it back until i scope to this piece of code (Note: it could be that other parts were also causing troubles but this one really make it to pass or fail:
Func<RpcSession, RpcSession, Task> testCallback = async (rpcServer, rpcClient) =>
{
IResponseTest1 responseTest1 = await rpcClient.RequestAsync<IRequestTest1, IResponseTest1>(rpcClient.PayloadFactory.CreatePayload(p =>
{
p.Value1 = 100;
p.Value2 = true;
}), default);
};

Commenting the code inside the lambda cause the publish fail or succeed

[agocke]

Thanks! Can you post the rest of the repro somewhere? We can do it privately if you don't want to share the code.

[MichalStrehovsky]

If it's okay to do it publicly, you can create a ZIP with everything necessary to repro the issue on our side by adding:

<ItemGroup>
    <IlcArg Include="--make-repro-path:c:\Temp" />
</ItemGroup>

to the project file and running publish again. This should create a ZIP file under c:\Temp (you can replace c:\Temp with any other path). Then drag and drop the ZIP to the comment window here.

[rodrigovaras]

2264700095_Rpc.Test.zip
Here his the .zip file you asked for.
The project and code were untouched as it was when i originally reported, not after I tried trimming and scoping in a specific method problem.

[MichalStrehovsky]

The compiler is running into infinite recursion while trying to reverse Roslyn's alpha renaming as part of reflection dataflow analysis. The stack looks like this:

>	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 374	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache..ctor.__MapGeneratedTypeTypeParameters|7_2(Internal.TypeSystem.MetadataType generatedType, System.Collections.Generic.Dictionary<Internal.TypeSystem.MetadataType, ILCompiler.Dataflow.CompilerGeneratedState.TypeArgumentInfo> generatedTypeToTypeArgs, ref ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.<>c__DisplayClass7_0 value) Line 420	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCache.TypeCache(Internal.TypeSystem.MetadataType type, Internal.IL.ILProvider ilProvider) Line 333	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCacheHashtable.CreateValueFromKey(Internal.TypeSystem.MetadataType key) Line 52	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TypeCacheHashtable.GetOrCreateValue(Internal.TypeSystem.MetadataType key, out bool created) Line 61	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.GetCompilerGeneratedStateForType(Internal.TypeSystem.MetadataType type) Line 624	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.CompilerGeneratedState.TryGetOwningMethodForCompilerGeneratedMember(Internal.TypeSystem.TypeSystemEntity sourceMember, out Internal.TypeSystem.MethodDesc owningMethod) Line 691	C#

Roslyn generated two nested classes that look like this:

And the compiler is lead to believe that in order to build the map for Teksoft.Rpc.Test.RpcTest+<>c__12 we need to build the map for Teksoft.Rpc.Test.RpcTest+<>c__12+<<TestRequest1InternalAsync>b__12_1>d that in turn needs Teksoft.Rpc.Test.RpcTest+<>c__12 again.

I was not able to create a minimal repro from this, I cannot make Roslyn generate nested classes like this.

Hopefully @sbomer would be able to make more sense of it.

It is possible to repro this by downloading the above ZIP file, unpacking it, opening the ILC solution (build.cmd -vs src\coreclr\tools\aot\ilc.slnx) and configuring the debugging options for ILC (set current directory to the extracted directory and set the command line arguments for ILC to @repro.rsp (it's at the root of the ZIP, so no path needed). ILC will whine because the CoreLib doesn't exactly match (using 9.0 CoreLib with 10.0 compiler), but the two asserts can be ignored and the stack overflow will be reached.