Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dotnet-sdk-8.0.100-preview.6.23316.32] Applications failed with Fatal error. Internal CLR error. (0x80131506) #87807

Closed
Junjun-zhao opened this issue Jun 20, 2023 · 11 comments
Assignees
Labels
area-VM-coreclr tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Milestone

Comments

@Junjun-zhao
Copy link
Member

Junjun-zhao commented Jun 20, 2023

Description

When testing the 3rd party apps with the latest .NET 8.0 Preview 6 SDK build, multiple apps (6 apps in our Lab) were affected and reported error: 'Fatal error. Internal CLR error. (0x80131506)' when launching or running.

Application Name: EventStore, SmartStore, grandnode, GoNorth, grandnode2, globaltools
OS: Windows 10 21H2
CPU: X64
.NET Build Number: dotnet-sdk-8.0.100-preview.6.23316.32
App Github Link: https://github.com/EventStore/EventStore
Serilog.Expression Source: Releases · serilog/serilog-expressions (github.com)
or checking the app source at https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1839140

Reproduction Steps

Minimal Repro steps (Demo attached): InternalCLRDemo.zip

  1. Create a default Console project.
  2. Reference Package: Serilog.Expressions.
  3. Write the following code in Program.cs.
   using Serilog.Templates;
   using Serilog.Templates.Themes;
   var test = new ExpressionTemplate("[{@t:HH:mm:ss} {@l:u3}", theme: TemplateTheme.Code);
  1. Build the project.
  2. Change the runtime.config file to let the app run against with dotnet-sdk-8.0.100-preview.6.23316.32
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "8.0.0-preview.6.23316.3"
    }
  1. Launch the demo app.

Expected behavior

App will launch success.

Actual behavior

Launch failed with exception "System.ExecutionEngineException: 'Exception of type 'System.ExecutionEngineException' was thrown.'" and the following error:

Fatal error. Internal CLR error. (0x80131506)
   at Serilog.Templates.Compilation.UnreferencedProperties.TemplateReferencedPropertiesFinder..ctor()
   at Serilog.Templates.Compilation.UnreferencedProperties.UnreferencedPropertiesFunction..ctor(Serilog.Templates.Ast.Template)
   at Serilog.Templates.Compilation.TemplateFunctionNameResolver.Build(Serilog.Expressions.NameResolver, Serilog.Templates.Ast.Template)
   at Serilog.Templates.ExpressionTemplate..ctor(System.String, System.IFormatProvider, Serilog.Expressions.NameResolver, Serilog.Templates.Themes.TemplateTheme, Boolean)
   at Program.<Main>$(System.String[])

Regression?

Yes.
Verify Scenarios:

  1. Windows10 21h2 x64 + dotnet-sdk-8.0.100-preview.6.23316.32: Fail
  2. Windows10 21h2 x64 + dotnet-sdk-6.0.410(default): Pass
  3. Windows10 21h2 x64 + dotnet-sdk-8.0.100-preview.6.23312.5: Pass

Known Workarounds

No response

Configuration

No response

Other information

App Repro steps:

The machine only has dotnet-sdk-8.0.100-preview.6.23316.32 installed and DOTNET_ROLL_FORWARD_ON_NO_CANDIDATE_FX set as 2.

  1. Open CMD,  input command “cd \EventStore\App”.
  2. Input command “dotnet EventStore.ClusterNode.dll --insecure --db ./tmp/data --index ./tmp/index --log ./tmp/log -runprojections all --startstandardprojections --EnableAtomPubOverHttp”.

Expected Result:
launch the app successfully.

Actual Result:
Launch the app failed with the following error:

Fatal error. Internal CLR error. (0x80131506)
   at Serilog.Templates.Compilation.UnreferencedProperties.TemplateReferencedPropertiesFinder..ctor()
   at Serilog.Templates.Compilation.UnreferencedProperties.UnreferencedPropertiesFunction..ctor(Serilog.Templates.Ast.Template)
   at Serilog.Templates.Compilation.TemplateFunctionNameResolver.Build(Serilog.Expressions.NameResolver, Serilog.Templates.Ast.Template)
   at Serilog.Templates.ExpressionTemplate..ctor(System.String, System.IFormatProvider, Serilog.Expressions.NameResolver, Serilog.Templates.Themes.TemplateTheme, Boolean)
   at EventStore.Common.Log.EventStoreLoggerConfiguration+<>c__DisplayClass16_0.<.ctor>g__Stats|3(Serilog.LoggerConfiguration)
   at Serilog.Configuration.LoggerSinkConfiguration.Logger(System.Action`1<Serilog.LoggerConfiguration>, Serilog.Events.LogEventLevel, Serilog.Core.LoggingLevelSwitch)
   at EventStore.Common.Log.EventStoreLoggerConfiguration+<>c__DisplayClass16_0.<.ctor>b__7(Serilog.LoggerConfiguration)
   at Serilog.Configuration.LoggerSinkConfiguration.Logger(System.Action`1<Serilog.LoggerConfiguration>, Serilog.Events.LogEventLevel, Serilog.Core.LoggingLevelSwitch)
   at EventStore.Common.Log.EventStoreLoggerConfiguration+<>c__DisplayClass16_0.<.ctor>g__AsyncSink|0(Serilog.Configuration.LoggerSinkConfiguration)
   at Serilog.Configuration.LoggerSinkConfiguration.Wrap(Serilog.Configuration.LoggerSinkConfiguration, System.Func`2<Serilog.Core.ILogEventSink,Serilog.Core.ILogEventSink>, System.Action`1<Serilog.Configuration.LoggerSinkConfiguration>, Serilog.Events.LogEventLevel, Serilog.Core.LoggingLevelSwitch)
   at Serilog.LoggerConfigurationAsyncExtensions.Async(Serilog.Configuration.LoggerSinkConfiguration, System.Action`1<Serilog.Configuration.LoggerSinkConfiguration>, Serilog.Sinks.Async.IAsyncLogEventSinkMonitor, Int32, Boolean)
   at Serilog.LoggerConfigurationAsyncExtensions.Async(Serilog.Configuration.LoggerSinkConfiguration, System.Action`1<Serilog.Configuration.LoggerSinkConfiguration>, Int32, Boolean)
   at EventStore.Common.Log.EventStoreLoggerConfiguration..ctor(System.String, System.String, Microsoft.Extensions.Configuration.IConfigurationRoot, EventStore.Common.Options.LogConsoleFormat, Serilog.RollingInterval, Int32, Int32, Boolean)
   at EventStore.Common.Log.EventStoreLoggerConfiguration.Default(System.String, System.String, Microsoft.Extensions.Configuration.IConfigurationRoot, EventStore.Common.Options.LogConsoleFormat, Serilog.RollingInterval, Int32, Int32, Boolean)
   at EventStore.Common.Log.EventStoreLoggerConfiguration.Initialize(System.String, System.String, EventStore.Common.Options.LogConsoleFormat, Int32, Serilog.RollingInterval, Int32, Boolean, System.String)
   at EventStore.ClusterNode.Program+<Main>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at EventStore.ClusterNode.Program.Main(System.String[])
   at EventStore.ClusterNode.Program.<Main>(System.String[])

@dotnet-actwx-bot @dotnet/compat

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 20, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 20, 2023
@hoyosjs
Copy link
Member

hoyosjs commented Jun 20, 2023

Full stack:

00 (Inline Function) --------`--------     coreclr!TokenPairList::Exists [D:\a\_work\1\s\src\coreclr\vm\siginfo.hpp @ 452] 
01 00000099`fd377da0 00007ffc`27a556fb     coreclr!CompareTypeDefsForEquivalence+0x47 [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp @ 3255] 
02 00000099`fd377ec0 00007ffc`27af40dd     coreclr!CompareTypeTokens+0x13f [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp @ 3470] 
03 00000099`fd377f80 00007ffc`279cb9c8     coreclr!MetaSig::CompareElementType+0x42d [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp @ 3931] 
04 00000099`fd378090 00007ffc`27a519ca     coreclr!MetaSig::CompareMethodSigs+0x588 [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp @ 4504] 
05 (Inline Function) --------`--------     coreclr!MethodTableBuilder::MethodSignature::SignaturesEquivalent+0x9d [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 871] 
06 (Inline Function) --------`--------     coreclr!MethodTableBuilder::MethodSignature::Equivalent+0x1c4 [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 899] 
07 (Inline Function) --------`--------     coreclr!MethodTableBuilder::LoaderFindMethodInParentClass+0x258 [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 253] 
08 00000099`fd378160 00007ffc`27a0bf9d     coreclr!MethodTableBuilder::PlaceVirtualMethods+0x32a [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 5439] 
09 00000099`fd3782e0 00007ffc`279afa5e     coreclr!MethodTableBuilder::BuildMethodTableThrowing+0x18ad [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 1617] 
0a 00000099`fd3787e0 00007ffc`279ae513     coreclr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0x12ee [D:\a\_work\1\s\src\coreclr\vm\methodtablebuilder.cpp @ 12603] 
0b 00000099`fd37ad00 00007ffc`279d50de     coreclr!ClassLoader::CreateTypeHandleForTypeKey+0x16f [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp @ 2937] 
0c (Inline Function) --------`--------     coreclr!ClassLoader::DoIncrementalLoad+0x987 [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp @ 2877] 
0d 00000099`fd37ade0 00007ffc`279d371a     coreclr!ClassLoader::LoadTypeHandleForTypeKey_Body+0x17ee [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp @ 3553] 
0e 00000099`fd37bba0 00007ffc`279b0f8d     coreclr!ClassLoader::LoadTypeHandleForTypeKey+0xd2 [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp @ 3272] 
0f 00000099`fd37bc30 00007ffc`279c9dbb     coreclr!ClassLoader::LoadTypeDefThrowing+0x3dd [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp @ 2252] 
10 (Inline Function) --------`--------     coreclr!MemberLoader::GetMethodDescFromMethodDef+0xa6c [D:\a\_work\1\s\src\coreclr\vm\memberload.cpp @ 613] 
11 00000099`fd37bd60 00007ffc`266888d4     coreclr!CEEInfo::resolveToken+0xf4b [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 1118] 
12 00000099`fd37c090 00007ffc`26686372     clrjit!Compiler::impImportBlockCode+0xdc4 [D:\a\_work\1\s\src\coreclr\jit\importer.cpp @ 8470] 
13 00000099`fd37c880 00007ffc`266860a9     clrjit!Compiler::impImportBlock+0x62 [D:\a\_work\1\s\src\coreclr\jit\importer.cpp @ 11307] 
14 00000099`fd37c990 00007ffc`2668580e     clrjit!Compiler::impImport+0x829 [D:\a\_work\1\s\src\coreclr\jit\importer.cpp @ 12257] 
15 00000099`fd37ca20 00007ffc`26703ef6     clrjit!Compiler::fgImport+0xe [D:\a\_work\1\s\src\coreclr\jit\flowgraph.cpp @ 580] 
16 (Inline Function) --------`--------     clrjit!Phase::Run+0x1b [D:\a\_work\1\s\src\coreclr\jit\phase.cpp @ 61] 
17 (Inline Function) --------`--------     clrjit!DoPhase+0x49 [D:\a\_work\1\s\src\coreclr\jit\phase.h @ 136] 
18 00000099`fd37ca50 00007ffc`266eedc0     clrjit!Compiler::compCompile+0x186 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 4520] 
19 00000099`fd37d280 00007ffc`266ee537     clrjit!Compiler::compCompileHelper+0x5b0 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 7073] 
1a 00000099`fd37d360 00007ffc`266edd1e     clrjit!Compiler::compCompile+0x257 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 6215] 
1b 00000099`fd37d460 00007ffc`266ecc8a     clrjit!jitNativeCode+0x242 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 7706] 
1c 00000099`fd37d600 00007ffc`27a06b93     clrjit!CILJit::compileMethod+0xaa [D:\a\_work\1\s\src\coreclr\jit\ee_il_dll.cpp @ 272] 
1d 00000099`fd37d6a0 00007ffc`27a06a46     coreclr!invokeCompileMethodHelper+0x8b [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 12682] 
1e 00000099`fd37d740 00007ffc`27af161d     coreclr!invokeCompileMethod+0x92 [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 12745] 
1f 00000099`fd37d7b0 00007ffc`27af2fea     coreclr!UnsafeJitFunction+0x33d [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 13194] 
20 00000099`fd37dbc0 00007ffc`279f5976     coreclr!MethodDesc::JitCompileCodeLocked+0x13a [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 930] 
21 00000099`fd37dd30 00007ffc`279f4662     coreclr!MethodDesc::JitCompileCodeLockedEventWrapper+0x44a [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 757] 
22 (Inline Function) --------`--------     coreclr!MethodDesc::JitCompileCode+0xa05 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 698] 
23 (Inline Function) --------`--------     coreclr!MethodDesc::PrepareILBasedCode+0xa05 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 424] 
24 (Inline Function) --------`--------     coreclr!MethodDesc::PrepareCode+0xa05 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 321] 
25 00000099`fd37de70 00007ffc`279f37a8     coreclr!CodeVersionManager::PublishVersionableCodeIfNecessary+0xb42 [D:\a\_work\1\s\src\coreclr\vm\codeversion.cpp @ 1709] 
26 00000099`fd37e300 00007ffc`279f350e     coreclr!MethodDesc::DoPrestub+0x168 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 2574] 
27 00000099`fd37e420 00007ffc`27ae7ba5     coreclr!PreStubWorker+0x1fe [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 2401] 
28 00000099`fd37e5c0 00007ffb`c7f81a97     coreclr!ThePreStub+0x55 [D:\a\_work\1\s\src\coreclr\vm\amd64\ThePreStubAMD64.asm @ 21] 
29 00000099`fd37e670 00007ffb`c7f816db     Serilog_Expressions!Serilog.Templates.Compilation.UnreferencedProperties.UnreferencedPropertiesFunction..ctor+0x47
2a 00000099`fd37e6c0 00007ffb`c7f59efa     Serilog_Expressions!Serilog.Templates.Compilation.TemplateFunctionNameResolver.Build+0x9b
2b 00000099`fd37e720 00007ffb`c7f56c3e     Serilog_Expressions!Serilog.Templates.ExpressionTemplate..ctor+0x10a
2c 00000099`fd37e7a0 00007ffc`27ae6863     InternalCLRDemo!Program.<Main>$+0x5e

It happened while loading virtual method Transform for Serilog.Templates.Compilation.UnreferencedProperties.ExpressionReferencedPropertiesFinder into the method table from Serilog.Expressions.dll, while compiling Serilog.Templates.Compilation.UnreferencedProperties.TemplateReferencedPropertiesFinder..ctor. We are doing signature comparison between two sigs trying to find the signature in the parent class, but the CompareState passed down from CompareMethodSigs is corrupt by the time we iterate over teh list to check for existence. cc: @AaronRobinsonMSFT

@hoyosjs hoyosjs added area-VM-coreclr and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jun 20, 2023
@hoyosjs
Copy link
Member

hoyosjs commented Jun 21, 2023

Looks like tk in CorSigUncompressToken consistently gets the address of the m_pNext in the TokenPairList entry, so this looks like funny business of CompareState being a stack allocated struct.

Looked a little with @jkoritzinsky and there's some oddities where MetaSig::CompareMethodSigs does:

00007ffc`2d48b988 488d45f7         lea     rax, [rbp-9]
00007ffc`2d48b98c 4d8bcf           mov     cbDataSize (r9), pEndSig2 (r15)
00007ffc`2d48b98f 4889442440       mov     qword ptr [rsp+40h], rax  # spill CompareState state to stack
00007ffc`2d48b994 488d5567         lea     rdx, [rbp+67h]
00007ffc`2d48b998 488b457f         mov     rax, qword ptr [rbp+7Fh]
00007ffc`2d48b99c 488d4d47         lea     rcx, [rbp+47h]
00007ffc`2d48b9a0 4889442438       mov     qword ptr [rsp+38h], pSubst2 (rax)
00007ffc`2d48b9a5 4d8bc4           mov     r8, pEndSig1 (r12)
00007ffc`2d48b9a8 488b455f         mov     pSubst2 (rax), qword ptr [rbp+5Fh]
00007ffc`2d48b9ac 4889442430       mov     qword ptr [rsp+30h], pSubst1 (rax)
00007ffc`2d48b9b1 488b4577         mov     pSubst1 (rax), qword ptr [rbp+77h]
00007ffc`2d48b9b5 4889442428       mov     qword ptr [rsp+28h], pModule2 (rax)
00007ffc`2d48b9ba 488b4557         mov     pModule2 (rax), qword ptr [rbp+57h]
00007ffc`2d48b9be 4889442420       mov     qword ptr [rsp+20h], pModule1 (rax)
00007ffc`2d48b9c3 e8e8821200       call    coreclr!MetaSig::CompareElementType (7ffc2d5b3cb0)

Then MetaSig::CompareElementType loads it correctly from the stack at that location. From the Prolog:

 3655 00007ffc`2d5b3cb0 48895c2410      mov     qword ptr [rsp+10h],rbx
 3655 00007ffc`2d5b3cb5 55              push    rbp
 3655 00007ffc`2d5b3cb6 56              push    rsi
 3655 00007ffc`2d5b3cb7 57              push    rdi
 3655 00007ffc`2d5b3cb8 4154            push    r12
 3655 00007ffc`2d5b3cba 4155            push    r13
 3655 00007ffc`2d5b3cbc 4156            push    r14
 3655 00007ffc`2d5b3cbe 4157            push    r15
 3655 00007ffc`2d5b3cc0 488d6c24f9      lea     rbp,[rsp-7]
 3655 00007ffc`2d5b3cc5 4881ecd0000000  sub     rsp,0D0h
 3673 00007ffc`2d5b3ccc 488b8587000000  mov     rax,qword ptr [rbp+87h] # this is the slot for the CompareState* parameter.

And later in the method this part:

CorElementType Type1 = ELEMENT_TYPE_MAX; // initialize to illegal
CorElementType Type2 = ELEMENT_TYPE_MAX; // initialize to illegal
IfFailThrow(CorSigUncompressElementType_EndPtr(pSig1, pEndSig1, &Type1));
IfFailThrow(CorSigUncompressElementType_EndPtr(pSig2, pEndSig2, &Type2));

gets translated to

00007ffc`2d5b3d67 4c8d8587000000         lea     r8, [rbp+87h]
00007ffc`2d5b3d6e c7858700000022000000   mov     dword ptr [rbp+87h], 22h
00007ffc`2d5b3d78 488bd7                 mov     rdx, pEndSig1 (rdi)
00007ffc`2d5b3d7b c7454722000000         mov     dword ptr [rbp+47h], 22h
00007ffc`2d5b3d82 488bcb                 mov     rcx, pSig1 (rbx)
00007ffc`2d5b3d85 e83672f9ff             call    coreclr!CorSigUncompressElementType_EndPtr (7ffc2d54afc0)
00007ffc`2d5b3d8a 8bc8                   mov     ecx, eax
00007ffc`2d5b3d8c e8af17f6ff             call    coreclr!IfFailThrow (7ffc2d515540)
00007ffc`2d5b3d91 4c8d4547               lea     r8, [rbp+47h]
00007ffc`2d5b3d95 498bd7                 mov     rdx, r15
00007ffc`2d5b3d98 498bce                 mov     rcx, r14
00007ffc`2d5b3d9b e82072f9ff             call    coreclr!CorSigUncompressElementType_EndPtr (7ffc2d54afc0)

So the third parameter for CorSigUncompressElementType_EndPtr gets the same address as the state parameter. Similarly in the switch:

case ELEMENT_TYPE_VALUETYPE:
case ELEMENT_TYPE_CLASS:
{
mdToken tk1, tk2;
IfFailThrow(CorSigUncompressToken_EndPtr(pSig1, pEndSig1, &tk1));
IfFailThrow(CorSigUncompressToken_EndPtr(pSig2, pEndSig2, &tk2));
return CompareTypeTokens(tk1, tk2, pModule1, pModule2, state->Visited);
}

gets translated to:

0007ffc`2d5b4090 4c8d4547               lea     r8, [rbp+47h]
00007ffc`2d5b4094 488bd7                 mov     rdx, pEndSig1 (rdi)
00007ffc`2d5b4097 488bcb                 mov     rcx, pSig1 (rbx)
00007ffc`2d5b409a e87528f6ff             call    coreclr!CorSigUncompressToken_EndPtr (7ffc2d516914)
00007ffc`2d5b409f 8bc8                   mov     ecx, eax
00007ffc`2d5b40a1 e89a14f6ff             call    coreclr!IfFailThrow (7ffc2d515540)
00007ffc`2d5b40a6 4c8d8587000000         lea     r8, [rbp+87h]
00007ffc`2d5b40ad 498bd7                 mov     rdx, r15
00007ffc`2d5b40b0 498bce                 mov     rcx, r14
00007ffc`2d5b40b3 e85c28f6ff             call    coreclr!CorSigUncompressToken_EndPtr (7ffc2d516914)
00007ffc`2d5b40b8 8bc8                   mov     ecx, eax
00007ffc`2d5b40ba e88114f6ff             call    coreclr!IfFailThrow (7ffc2d515540)
00007ffc`2d5b40bf 488b06                 mov     rax, qword ptr [rsi]
00007ffc`2d5b40c2 4c8b4d6f               mov     r9, qword ptr [rbp+6Fh]
00007ffc`2d5b40c6 4c8b4567               mov     r8, qword ptr [rbp+67h]
00007ffc`2d5b40ca 8b9587000000           mov     edx, dword ptr [rbp+87h]
00007ffc`2d5b40d0 8b4d47                 mov     ecx, dword ptr [rbp+47h]
00007ffc`2d5b40d3 4889442420             mov     qword ptr [rsp+20h], rax
00007ffc`2d5b40d8 e8df14f6ff             call    coreclr!CompareTypeTokens (7ffc2d5155b

And for example, the busted pNext gets written at:

00 0000009c`cf377e48 00007ffc`2d516963     coreclr!CorSigUncompressData+0x66 [D:\a\_work\1\s\src\coreclr\inc\cor.h @ 1913] 
01 0000009c`cf377e50 00007ffc`2d516939     coreclr!CorSigUncompressToken+0x13 [D:\a\_work\1\s\src\coreclr\inc\cor.h @ 2001] 
02 0000009c`cf377e90 00007ffc`2d5b40b8     coreclr!CorSigUncompressToken_EndPtr+0x25 [D:\a\_work\1\s\src\coreclr\inc\corhlprpriv.h @ 779] 

@hoyosjs
Copy link
Member

hoyosjs commented Jun 22, 2023

Talked with @jkotas, this is caused by a PGO optimization issue. Opened https://developercommunity.visualstudio.com/t/PGO-optimized-build-ends-up-reusing-argu/10398027

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Jun 26, 2023
@mangod9 mangod9 added this to the 8.0.0 milestone Jun 26, 2023
@Junjun-zhao
Copy link
Member Author

@hoyosjs @mangod9 Thank you for looking into this bug. We are in validation of .NET 8.0 Preview 6, could you confirm whether this issue is a blocker for .NET 8.0 Preview 6?
PS: We are the .NET AppCompat team and have a representative set of apps in our lab, 6 apps affected by this issue till now, it potentially impacts a bigger customer base in the real world.

@jkotas
Copy link
Member

jkotas commented Jun 29, 2023

This issue should be .NET 8.0 Preview 6 blocking. Does it still repro in the latest .NET 8.0 Preview 6 builds? If it still repros, it should be fixed by backporting up-to-date PGO data to P6.

@jeffschwMSFT
Copy link
Member

who has the next step to address this issue? Is work in-flight?

@tommcdon
Copy link
Member

tommcdon commented Jun 29, 2023

who has the next step to address this issue? Is work in-flight?

@hoyosjs is investigating. Per JanK's suggestion, we are determining if current PGO data in main resolves the issue against P6, else we shall consider other options. One other option would be to disable PGO optimization in P6.
This is tracking the bug on the C++ side: https://developercommunity.visualstudio.com/t/PGO-optimized-build-ends-up-reusing-argu/10398027

@Junjun-zhao
Copy link
Member Author

Thank you all for the quick turnaround. Yes, it still repros on the latest P6 build dotnet-sdk-8.0.100-preview.6.23328.9

@hoyosjs
Copy link
Member

hoyosjs commented Jun 30, 2023

The latest p6 SDK fixed this confirmed: 8.0.100-preview.6.23329.15

@hoyosjs hoyosjs added tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly and removed blocking-release labels Jun 30, 2023
@hoyosjs hoyosjs modified the milestones: 8.0.0, Future Jun 30, 2023
@jkotas
Copy link
Member

jkotas commented Jun 30, 2023

There is no remaining work to do for this issue in this repo.

@jkotas jkotas closed this as completed Jun 30, 2023
@Junjun-zhao
Copy link
Member Author

Verified the 6 affected apps with dotnet-sdk-8.0.100-preview.6.23330.14, this issue has been fixed. Thank you!

@ghost ghost locked as resolved and limited conversation to collaborators Aug 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-VM-coreclr tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Projects
None yet
Development

No branches or pull requests

6 participants