Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STATUS_ACCESS_VIOLATION in coreclr.dll version 9.0.24.52809 #109981

Open
rstutton opened this issue Nov 19, 2024 · 35 comments · Fixed by #110531
Open

STATUS_ACCESS_VIOLATION in coreclr.dll version 9.0.24.52809 #109981

rstutton opened this issue Nov 19, 2024 · 35 comments · Fixed by #110531
Assignees
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI in-pr There is an active PR which will close this issue when it is merged
Milestone

Comments

@rstutton
Copy link

Description

Faulting application name: internalWeb.exe, version: 2.6.2935.0, time stamp: 0x67200000
Faulting module name: coreclr.dll, version: 9.0.24.52809, time stamp: 0x672049fc
Exception code: 0xc0000005
Fault offset: 0x00000000002eea8b
Faulting process id: 0x649c
Faulting application start time: 0x01db3988d312000b
Faulting application path: C:\Program Files\3DBackup\bin\internalWeb.exe
Faulting module path: C:\Program Files\3DBackup\bin\coreclr.dll
Report Id: 8d1a48a5-e27f-4a5f-acbf-1d40624ed842

Reproduction Steps

This happened 2 nights running before I backed out the migration to .net 9.
I have not isolated the fault.
WER reported the issue and captured data on both occasions:

Report Id: 8d1a48a5-e27f-4a5f-acbf-1d40624ed842
Report Status: 268435456
Hashed bucket: 2e7dfb54d62033d3fc1a7aa712618a8b

Report Id: 324f8194-84f9-4ced-b429-27044b78e4d5
Report Status: 268435456
Hashed bucket: 3700e920166dc8b4904ac6c8801323ea

Expected behavior

Not crashing.

Actual behavior

STATUS_ACCESS_VIOLATION in coreclr.dll crashes the app.

Regression?

Yes. The same code runs fine in .net 8, has been in production, and on customer sites.

Known Workarounds

None.

Configuration

.net sdk 9.0.100
aspnet core.

OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.19045 N/A Build 19045
OS Manufacturer: Microsoft Corporation
OS Configuration: Member Workstation
OS Build Type: Multiprocessor Free
Product ID: 00330-51446-83390-AAOEM
Original Install Date: 27/05/2022, 1:27:58 pm
System Boot Time: 13/11/2024, 9:28:29 pm
System Manufacturer: HP
System Model: HP Z240 Tower Workstation
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 94 Stepping 3 GenuineIntel ~4001 Mhz
BIOS Version: HP N51 Ver. 01.70, 19/06/2018
Total Physical Memory: 32,553 MB

Other information

No response

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Nov 19, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Nov 19, 2024
@rsankardayal
Copy link

I am having the same issue; Works ok with .NET8, converted to .NET9 and then receive the following error below;

Faulting application name: dotnet.exe, version: 9.0.24.52809, time stamp: 0x67204d77
Faulting module name: KERNELBASE.dll, version: 10.0.19041.3930, time stamp: 0x9141f979
Exception code: 0xe0434352
Fault offset: 0x000000000002cf19
Faulting process id: 0x1460
Faulting application start time: 0x01db3b5c0d69c407
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\windows\System32\KERNELBASE.dll
Report Id: d8932e9d-cdf6-4d78-9ece-963e1e6a4a02
Faulting package full name:
Faulting package-relative application ID:

@jkotas
Copy link
Member

jkotas commented Nov 20, 2024

I am having the same issue

This is a different issue. For example, it has different Exception Code. You may want to open a new issue on it.

I have not isolated the fault.

Do you see more detailed error message on the console or in the event log (open Event Viewer application that comes with Windows to and go to Windows Logs / Application)? Here is an example what the more detailed error looks like:

Image

@rsankardayal
Copy link

This is in my event viewer:
Image

@jkotas
Copy link
Member

jkotas commented Nov 20, 2024

This is in my event viewer:

This looks like an issue in your application code. What does the code in Program.cs around line 71 look like? Where is the null value coming from?

@rsankardayal
Copy link

The null is from trying to use this...dotnet a.dll -ENVIRONMENT=dev
However run it from visual studio, I am getting this...

Image

@jkotas
Copy link
Member

jkotas commented Nov 20, 2024

The null is from trying to use this...dotnet a.dll -ENVIRONMENT=dev

Could you please elaborate? I do not see how this explains where the null is coming from.

@rsankardayal
Copy link

Hi,
Thanks for you assistance, it's working with .NET8, I'll wait until .NET9 is more stable.

@jkotas
Copy link
Member

jkotas commented Nov 21, 2024

@rsankardayal It is unlikely that the problem that you are experiencing is going to be fixed in .NET 9 unless we get to the bottom of the issue.

@shuijianfeng
Copy link

Faulting application name: hcmanagement.exe, version: 2.1.0.10, time: 0x66f10000 Faulting module name: coreclr.dll, version: 9.0.24.52809, time: 0x672049fc Exception code: 0xc0000602 Fault offset: 0x000000000032a356 Faulting process ID: 0x28c8 Faulting application start time: 0x01db3cb3386587e4 Faulting application path: C:\Qingshan Changyuan Software\Qingshan Changyuan Cost System\hcmanagement.exe Faulting module path: C:\Program Files\dotnet\shared\Microsoft. NETCore.App\9.0.0\coreclr.dll Report ID: 7596c246-d81b-4054-9e57-de74abc50f1d Faulting package full name:

@jkotas
Copy link
Member

jkotas commented Nov 22, 2024

Exception code: 0xc0000602

@shuijianfeng This exception code is not STATUS_ACCESS_VIOLATION that this issue is about. Please open a new issue on it and include more detailed error message that you should be able to find in on the console or in the event log.

@jkotas jkotas added area-VM-coreclr needs-author-action An issue or pull request that requires more info or actions from the author. and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Nov 22, 2024
@rstutton
Copy link
Author

This issue is marked as needs-author-action.
I am the author but I can see no questions for me ?
Thanks

@dotnet-policy-service dotnet-policy-service bot removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Nov 26, 2024
@jkotas
Copy link
Member

jkotas commented Nov 26, 2024

Do you see more detailed error message on the console or in the event log (open Event Viewer application that comes with Windows to and go to Windows Logs / Application)? (#109981 (comment))

@rstutton
Copy link
Author

My understanding is that Microsoft staff have access to the data uploaded by Windows Error Reporting - including dmp files etc. Is this not the case? The WER event does indicate files were uploaded.

There was no console output. There is more in the event log which I have pasted below. Sorry I omitted this - it was because I thought you'd have access based on the report Id.

`Application:` internalweb.exe
CoreCLR Version: 9.0.24.52809
.NET Version: 9.0.0
Description: The process was terminated due to an unhandled exception.
Stack:
   at InternalWeb.Controllers.SSEClientController+<ClientConnect>d__26.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[InternalWeb.Controllers.SSEClientController+<ClientConnect>d__26, InternalWeb, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[InternalWeb.Controllers.SSEClientController+<ClientConnect>d__26, InternalWeb, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[InternalWeb.Controllers.SSEClientController+<ClientConnect>d__26, InternalWeb, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at BaseCore.Models.SKClient+<SetClientLastConnectedAsync>d__126.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[BaseCore.Models.SKClient+<SetClientLastConnectedAsync>d__126, BaseCore, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[BaseCore.Models.SKClient+<SetClientLastConnectedAsync>d__126, BaseCore, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[BaseCore.Models.SKClient+<SetClientLastConnectedAsync>d__126, BaseCore, Version=2.6.0.0, Culture=neutral, PublicKeyToken=35ef459a7c6e33da]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<Int32>, Int32)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32)
   at Npgsql.NpgsqlCommand+<ExecuteNonQuery>d__108.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteNonQuery>d__108, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteNonQuery>d__108, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteNonQuery>d__108, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)
   at Npgsql.NpgsqlCommand+<ExecuteReader>d__120.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteReader>d__120, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteReader>d__120, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlCommand+<ExecuteReader>d__120, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Boolean)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<Boolean>, Boolean)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Boolean)
   at Npgsql.NpgsqlDataReader+<NextResult>d__52.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlDataReader+<NextResult>d__52, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlDataReader+<NextResult>d__52, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.NpgsqlDataReader+<NextResult>d__52, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext()
   at Npgsql.Internal.NpgsqlConnector+<ReadMessageLong>d__233.MoveNext()
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.Internal.NpgsqlConnector+<ReadMessageLong>d__233, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.Internal.NpgsqlConnector+<ReadMessageLong>d__233, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext()
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder.SetResult()
   at Npgsql.Internal.NpgsqlReadBuffer+<<Ensure>g__EnsureLong|55_0>d.MoveNext()
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.Internal.NpgsqlReadBuffer+<<Ensure>g__EnsureLong|55_0>d, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Npgsql.Internal.NpgsqlReadBuffer+<<Ensure>g__EnsureLong|55_0>d, Npgsql, Version=8.0.5.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7]].MoveNext()
   at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__174_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading.PortableThreadPool+IOCompletionPoller+Callback.Invoke(Event)
   at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

@jkotas
Copy link
Member

jkotas commented Nov 26, 2024

My understanding is that Microsoft staff have access to the data uploaded by Windows Error Reporting - including dmp files etc

The system applies a lot of filters and it does not keep much information for one-off crashes. All I can see that there were two crashes in internalweb.exe in November, but no additional details.

InternalWeb.Controllers.SSEClientController+d__26.MoveNext

This says that there was unhandled exception in your app code. Are you able to find the exception type, message, and the app code that throws it?

@rstutton
Copy link
Author

rstutton commented Nov 26, 2024 via email

@rstutton
Copy link
Author

This fails regularly for me, and I have captured a mini dump from the crash.
Is this useful, and if so where do I put it? It's about 11MB zipped.

@jkotas
Copy link
Member

jkotas commented Nov 30, 2024

You can open an issue at https://developercommunity.visualstudio.com/, attach the dump to it and share the link to it here. https://developercommunity.visualstudio.com allows attaching larger files and it has better privacy controls.

@rstutton
Copy link
Author

rstutton commented Dec 1, 2024

Please see: VS report with attached dmp

@jkotas
Copy link
Member

jkotas commented Dec 2, 2024

That you for sharing the dump. The crash is caused by tiered compilation (cc @dotnet/jit-contrib):

00 00000046`023b8e00 00007ffe`b0e0c3ad     KERNELBASE!RaiseException+0x68 [d:\rs1\minkernel\kernelbase\xcpt.c @ 904] 
01 00000046`023b8ee0 00007ffe`beb6a173     coreclr!__FrameHandler4::CxxCallCatchBlock+0x1ad [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 1463] 
02 00000046`023b8fb0 00007ffe`b0d69e37     ntdll!RcFrameConsolidation+0x3 [d:\rs1\minkernel\ntos\rtl\amd64\capture.asm @ 596] 
03 00000046`023bc230 00007ffe`b0d69ca4     coreclr!MethodDesc::JitCompileCodeLocked+0xef [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 937] 
04 00000046`023bc340 00007ffe`b0d699a1     coreclr!MethodDesc::JitCompileCodeLockedEventWrapper+0x16c [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 818] 
05 00000046`023bc460 00007ffe`b0cc53e2     coreclr!MethodDesc::JitCompileCode+0x2f1 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 705] 
06 00000046`023bc500 00007ffe`b0df982c     coreclr!MethodDesc::PrepareILBasedCode+0xca [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 439] 
07 (Inline Function) --------`--------     coreclr!MethodDesc::PrepareCode+0xc [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 319] 
08 00000046`023bc590 00007ffe`b0df96a4     coreclr!JitPatchpointWorker+0xe4 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3803] 
09 00000046`023bc6a0 00007ffe`b0dae5f5     coreclr!JIT_Patchpoint_Framed+0xf4 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3813] 
0a 00000046`023bc810 00007ffe`527e5090     coreclr!JIT_Patchpoint+0x2f5 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3984] 

If possible, could you please do two more things - it would help us with diagnosing the problem faster:

  • Collect a full dump of the crash (set DOTNET_DbgMiniDumpType=4 in addition to the env variables that you have set) and attach it to the issue you have opened at developercommunity.visualstudio.com

  • Try to disable tiered compilation by setting DOTNET_TieredCompilation=0 and check whether it makes the crash to go away. If it is still crashing with DOTNET_TieredCompilation=0, collect the dump (with DOTNET_DbgMiniDumpType=4 ) and attach it to the issue as well.

Thank you!

@jkotas jkotas added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-VM-coreclr labels Dec 2, 2024
Copy link
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

@rstutton
Copy link
Author

rstutton commented Dec 2, 2024

Ok, I've set DOTNET_DbgMiniDumpType=4 as requested and the server is running.
It typically fails overnight, so I'll post the dump when I get my hands on it.
After that I'll try with tiered compilation disabled.

@rstutton
Copy link
Author

rstutton commented Dec 4, 2024

Sorry, it took an extra night to capture the dump.
My app failed as predicted but I hadn't set DOTNET_DbgEnableMiniDump=1, I had only set DOTNET_DbgMiniDumpType=4.
So, lesson learned.

Anyway, the full dump can be found attached here:
VS report with attached dmp

@rsankardayal
Copy link

I've installed 17.12.3 and here are the steps to reproduce.

  1. Create new ASP.NET Cor Web API with name "WebApplicationTest1" - See image below
  2. Select .NET 9.0 (Standard Term Support) - See image below
  3. Click on "https" to run program - See image below
  4. The debug window open briefly and then the error popup - See image below
  5. Cannot find visualstudio-js-debugger.txt file in the location previously mention is this issue.

Image
Image
Image
Image

@rstutton
Copy link
Author

rstutton commented Dec 4, 2024

I've installed 17.12.3 and here are the steps to reproduce.

That's not the same issue. That looks like you're trying to bind to port 443 without permission. Try running VS as admin.

@jkotas
Copy link
Member

jkotas commented Dec 5, 2024

@rstutton Thank you for collecting another dump! The latest dump confirms that the crash always occurs when the JIT is compiling the same method (InternalWeb.Controllers.SSEClientController+<ClientConnect>d__26.MoveNext). Unfortunately, some exception handling code executed before the dump was collected that makes it hard to determine the root cause.

Could you please collect a dump using procdump tool using following steps:

  1. Download procdump tool https://learn.microsoft.com/sysinternals/downloads/procdump
  2. Clear all DOTNET environment variables
  3. Start your server
  4. In second cmd window, run procdump.exe -accepteula -ma -e 1 -g -f C0000005 internalWeb.exe

When the server crashes, procdump should produce dump in the current directory. This crash dump should have better information about the root cause.

@rsankardayal
Copy link

Hi @rstutton,

Agreed not the same issue, where would I submit my issue.

  1. If I run it with "IIS Express" with .NET 9 and project name (WebApplication3Test1) it works.
  2. Reference Step (1) - Run the with "https", get an error "Access Denied"
  3. If I go to the bin/Debug/net9.0 folder, double click on WebApplication3Test1.exe, explorer crashes
  4. If I renamed the solution and project from "WebApplication3Test1" to "test1" it works with "https" - See image
  5. Reference Step (4) - If I go to the bin/Debug/net9.0 folder, double click on Test1.exe, it's runs

From what I can see it has something to do with the project name length, any project name over 15 char length seems to fail.

Image

@jkotas
Copy link
Member

jkotas commented Dec 5, 2024

Agreed not the same issue, where would I submit my issue.

You can submit it either to dotnet/runtime or dotnet/aspnetcore repos. We route issues as necessary.

@rstutton
Copy link
Author

rstutton commented Dec 6, 2024

A new dump (produced by procdump) has been uploaded:
VS report with attached dmp

@jkotas
Copy link
Member

jkotas commented Dec 6, 2024

Stacktrace of the crash:

clrjit!FlowGraphDominatorTree::IntersectDom+0x24 [inlined in clrjit!FlowGraphDominatorTree::Build+0x281]:
00007ffa`01f999e1 8b8294000000    mov     eax,dword ptr [rdx+94h] ds:00000000`00000094=????????
0:019> k
 # Child-SP          RetAddr               Call Site
00 (Inline Function) --------`--------     clrjit!FlowGraphDominatorTree::IntersectDom+0x24 [D:\a\_work\1\s\src\coreclr\jit\flowgraph.cpp @ 6116] 
01 00000085`2db7b1b0 00007ffa`01f9848d     clrjit!FlowGraphDominatorTree::Build+0x281 [D:\a\_work\1\s\src\coreclr\jit\flowgraph.cpp @ 6251] 
02 00000085`2db7b270 00007ffa`01f22759     clrjit!Compiler::optInductionVariables+0x9d [D:\a\_work\1\s\src\coreclr\jit\inductionvariableopts.cpp @ 2418] 
03 (Inline Function) --------`--------     clrjit!Phase::Run+0x21 [D:\a\_work\1\s\src\coreclr\jit\phase.cpp @ 61] 
04 00000085`2db7b620 00007ffa`01f224ac     clrjit!DoPhase+0x51 [D:\a\_work\1\s\src\coreclr\jit\phase.h @ 143] 
05 00000085`2db7b680 00007ffa`01f8c3b0     clrjit!Compiler::compCompile+0x18ec [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 5169] 
06 00000085`2db7ba20 00007ffa`01f8dcdd     clrjit!Compiler::compCompileHelper+0x360 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 7409] 
07 00000085`2db7bb00 00007ffa`01f8eaa7     clrjit!Compiler::compCompile+0x24d [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 6532] 
08 00000085`2db7bbf0 00007ffa`01fcf6e6     clrjit!jitNativeCode+0x257 [D:\a\_work\1\s\src\coreclr\jit\compiler.cpp @ 8035] 
09 00000085`2db7bd90 00007ffa`04c8f9b4     clrjit!CILJit::compileMethod+0xa6 [D:\a\_work\1\s\src\coreclr\jit\ee_il_dll.cpp @ 294] 
0a (Inline Function) --------`--------     coreclr!invokeCompileMethodHelper+0x6c [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 12464] 
0b (Inline Function) --------`--------     coreclr!invokeCompileMethod+0xa0 [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 12527] 
0c 00000085`2db7be30 00007ffa`04d29e37     coreclr!UnsafeJitFunction+0x5c4 [D:\a\_work\1\s\src\coreclr\vm\jitinterface.cpp @ 12971] 
0d 00000085`2db7c330 00007ffa`04d29ca4     coreclr!MethodDesc::JitCompileCodeLocked+0xef [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 937] 
0e 00000085`2db7c440 00007ffa`04d299a1     coreclr!MethodDesc::JitCompileCodeLockedEventWrapper+0x16c [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 818] 
0f 00000085`2db7c560 00007ffa`04c853e2     coreclr!MethodDesc::JitCompileCode+0x2f1 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 705] 
10 00000085`2db7c600 00007ffa`04db982c     coreclr!MethodDesc::PrepareILBasedCode+0xca [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 439] 
11 (Inline Function) --------`--------     coreclr!MethodDesc::PrepareCode+0xc [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 319] 
12 00000085`2db7c690 00007ffa`04db96a4     coreclr!JitPatchpointWorker+0xe4 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3803] 
13 00000085`2db7c7a0 00007ffa`04d6e5f5     coreclr!JIT_Patchpoint_Framed+0xf4 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3813] 
14 00000085`2db7c910 00007ff9`a6bfd760     coreclr!JIT_Patchpoint+0x2f5 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3984] 

@dotnet/jit-contrib Could you please take it from here?

@jakobbotsch jakobbotsch self-assigned this Dec 6, 2024
@jakobbotsch jakobbotsch removed the untriaged New issue has not been triaged by the area owner label Dec 6, 2024
@jakobbotsch jakobbotsch added this to the 10.0.0 milestone Dec 6, 2024
@jakobbotsch
Copy link
Member

jakobbotsch commented Dec 6, 2024

@rstutton Thanks for the report and for working with us to capture the dumps. It would be very helpful if you could additionally collect a SuperPMI recording of this particular JIT compilation, so that I can figure out how the JIT has gotten into the state that I see in the crash dump.
The SuperPMI recording should allow me to reproduce the problem on my side.

To record it please obtain superpmi-shim-collector.dll (I have attached a copy on the VS report you submitted). Place that file at C:\Program Files\3DBackup\bin\superpmi-shim-collector.dll. Then run your application with the following environment variables:

DOTNET_JitName=superpmi-shim-collector.dll
SuperPMIShimPath=C:\Program Files\3DBackup\bin\clrjit.dll
SuperPMIShimLogPath=<path to an empty folder on your file system>

When you run the application, you should see a .mc file (or potentially multiple .mc files) appear in the folder you specified in SuperPMIShimLogPath. Can you please reproduce the problem once more and then zip and attach those files to the VS report?

FWIW, it is very likely you can get a quicker reproduction of this problem with tiered compilation disabled (DOTNET_TieredCompilation=0).

Finally, there should be a workaround for the problem you are hitting by setting DOTNET_JitEnableInductionVariableOpts=0, if you'd like to continue trying .NET 9 until we resolve this issue and backport the fix to .NET 9.

@rstutton
Copy link
Author

rstutton commented Dec 7, 2024

@jakobbotsch I have uploaded the captured mc file to VS report with attached dmp.

@jakobbotsch
Copy link
Member

@rstutton Thank you! I am able to reproduce the problem locally now using the file you have provided. I'll work on a fix and post back with an update.

@jakobbotsch
Copy link
Member

Some technical notes: we have a case where BlockDominancePreds for a handler entry returns no blocks that are reachable, yet the handler block itself is considered reachable since it is part of the DFS post-order. This should not be possible.

@jakobbotsch
Copy link
Member

jakobbotsch commented Dec 9, 2024

Some more technical notes:

The failing case roughly looks like the following EH wise, translated into C#:

public static void Foo()
{
    try
    {
        return;

        // BeforeTry
        try
        {
            try
            {
                // InnerInnerTry
            }
            finally
            {
                // InnerFinally
            }
        }
        catch (SomeException)
        {
            // CatchHandler
        }
    }
    catch when (FilterCode) // FilterHandler
    {
    }
}

The JIT has made some late optimizations that allows it to realize that BeforeTry is not reachable (essentially, it has optimized some "if (x) return" into the "return;" at the top). This happens during RBO or assertion prop.

At the same time we still consider CatchHandler to be reachable. That has to do with how we overapproximate the flow in the face of EH. The JIT in this case sees the following spurious flow path: FilterCode-> InnerFinally -> CatchHandler. The FilterCode -> InnerFinally edge appears because in the case where an exception was thrown inside InnerInnerTry, the EH system will potentially first execute FilterCode, after which it will execute InnerFinally. Similarly, flow from InnerFinally -> CatchHandler is possible if the exception thrown happens to be caught by that exception handler.

All of these flow edges are spurious, meaning that they cannot actually happen at runtime, but we cannot statically make that determination without optimizing out some blocks that induce those spurious flow edges. We would normally do that, but in this case we do not because of how late the optimization happened.

Now, the real problem happens when we try to compute the dominator of CatchHandler. There we make the assumption that the dominator of a catch handler is the intersection of the dominators of the blocks that can enter the corresponding try region. However, in the particular flow graph we end up with based on the code above, that's not true.

@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Dec 9, 2024
@jakobbotsch
Copy link
Member

I opened #110531 with a fix for the issue. I will backport that fix to .NET 9 servicing after it is merged. The next servicing release won't be until January (at the earliest), but in the meantime the problem can be worked around by setting DOTNET_JitEnableInductionVariableOpts=0 before the application is started.

Thanks a lot for the report and for collecting all the extra information! I will leave this issue open until the servicing release with the fix has been published.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI in-pr There is an active PR which will close this issue when it is merged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants