Skip to content

Intermittent crash in ObjectNative::PulseAll #97034

Open
@SkadOrdos

Description

@SkadOrdos

Description

Our Application crashed with a internal NetCore Runtime error
It began after migrate from Net Framework 4.8 to NET 7.
Crash reproduce on NET 7.0.14 / 7.0.15 / 8.0.1

Faulting application name: server.exe, version: 1.0.10.1, time stamp: 0x65410000
Faulting module name: coreclr.dll, version: 8.0.123.58001, time stamp: 0x65684720
Exception code: 0xc0000005
Fault offset: 0x00000000000e87a6
Faulting process id: 0x48f8
Faulting application start time: 0x01da488073edf4b2
Faulting application path: D:\1.1.Server\server.exe
Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.1\coreclr.dll
Report Id: be1a8f26-efe5-4a91-9f2f-242672b1661e
Faulting package full name:
Faulting package-relative application ID:

Description: The process was terminated due to an internal error in the .NET Runtime at IP 0x00007FFA27E587A6 (0x00007FFA27D70000) with exit code 0x80131506.

Reproduction Steps

It crashes spontaneously: could be once a hour or few days without errors. By stack trace it seem depends on Task.Start

Expected behavior

Work without spontaneously crashes

Actual behavior

Dump analyze via WinDbg

************* Preparing the environment for Debugger Extensions Gallery repositories **************
ExtensionRepository : Implicit
UseExperimentalFeatureForNugetShare : true
AllowNugetExeUpdate : true
AllowNugetMSCredentialProviderInstall : true
AllowParallelInitializationOfLocalRepositories : true

-- Configuring repositories
----> Repository : LocalInstalled, Enabled: true
----> Repository : UserExtensions, Enabled: true

Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds
----> Repository : UserExtensions, Enabled: true, Packages count: 0
----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [H:\server.exe.18680.dmp]
User Mini Dump File with Full Memory: Only application data is available

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 10 Version 20348 MP (16 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 20348.1.amd64fre.fe_release.210507-1500
Debug session time: Tue Jan 16 15:58:19.000 2024 (UTC + 2:00)
System Uptime: 10 days 8:20:29.450
Process Uptime: 0 days 0:25:50.000
................................................................
................................................................
.....................
Loading unloaded module list
.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(48f8.4508): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
coreclr!CLREventBase::Set [inlined in coreclr!ObjectNative::PulseAll+0x126]:
00007ffa27e587a6 488b09 mov rcx,qword ptr [rcx] ds:000096ac68d2d80a=????????????????
0:904> !analyze -v


  •                                                                         *
    
  •                    Exception Analysis                                   *
    
  •                                                                         *
    

Failed to request MethodData, not in JIT code range

KEY_VALUES_STRING: 1

Key  : AV.Fault
Value: Read

Key  : Analysis.CPU.mSec
Value: 2983

Key  : Analysis.Elapsed.mSec
Value: 17261

Key  : Analysis.IO.Other.Mb
Value: 15

Key  : Analysis.IO.Read.Mb
Value: 1

Key  : Analysis.IO.Write.Mb
Value: 20

Key  : Analysis.Init.CPU.mSec
Value: 13218

Key  : Analysis.Init.Elapsed.mSec
Value: 1129306

Key  : Analysis.Memory.CommitPeak.Mb
Value: 243

Key  : CLR.Engine
Value: CORECLR

Key  : CLR.Version
Value: 8.0.123.58001

Key  : Failure.Bucket
Value: INVALID_POINTER_READ_c0000005_coreclr.dll!ObjectNative::PulseAll

Key  : Failure.Hash
Value: {be7e1d5f-f864-f771-a72a-7be333bc8c49}

Key  : Failure.Source.FileLine
Value: 340

Key  : Failure.Source.FilePath
Value: D:\a\_work\1\s\src\coreclr\classlibnative\bcltype\objectnative.cpp

Key  : Failure.Source.SourceServerCommand
Value: raw.githubusercontent.com/dotnet/runtime/bf5e279d9239bfef5bb1b8d6212f1b971c434606/src/coreclr/classlibnative/bcltype/objectnative.cpp

Key  : Timeline.OS.Boot.DeltaSec
Value: 894029

Key  : Timeline.Process.Start.DeltaSec
Value: 1550

Key  : WER.OS.Branch
Value: fe_release

Key  : WER.OS.Version
Value: 10.0.20348.1

Key  : WER.Process.Version
Value: 1.0.10.1

FILE_IN_CAB: server.exe.18680.dmp

NTGLOBALFLAG: 0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS: 0

CONTEXT: (.ecxr)
rax=000000ab838fef10 rbx=000000ab829ff200 rcx=000096ac68d2d80a
rdx=000000ab838fef30 rsi=000000ab829fea70 rdi=000000ab829fed10
rip=00007ffa27e587a6 rsp=000000ab829ff2d0 rbp=000000ab829ff4a0
r8=0000000000000054 r9=0000000000000056 r10=0000000000000007
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000027c7107f878 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010200
coreclr!CLREventBase::Set [inlined in coreclr!ObjectNative::PulseAll+0x126]:
00007ffa27e587a6 488b09 mov rcx,qword ptr [rcx] ds:000096ac68d2d80a=????????????????
Resetting default scope

EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffa27e587a6 (coreclr!CLREventBase::Set)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff

PROCESS_NAME: server.exe

READ_ADDRESS: ffffffffffffffff

ERROR_CODE: (NTSTATUS) 0xc0000005 - 0x%p 0x%p. %s.

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: ffffffffffffffff

IP_ON_HEAP: 00007ff9c9423f8a
The fault address in not in any loaded module, please check your build's rebase
log at \bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

FRAME_ONE_INVALID: 1

STACK_TEXT:
000000ab829ff2d0 00007ff9c9423f8a : 0000023c675483a0 0000023c675483a0 0000023c608a3118 0000023c67548388 : coreclr!ObjectNative::PulseAll+0x126
000000ab829ff430 00007ff9ca5b0048 : 0000023c67548388 00007ff900000000 000000ab829ff4a0 0000027c6f03e3d0 : System_Private_CoreLib!System.Threading.ManualResetEventSlim.Set+0x8a
000000ab829ff4b0 00007ff9ca224447 : 0000023c67548210 0000000000000000 000000ab829ff540 0000000000000000 : System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x128
000000ab829ff550 00007ff9ca177076 : 0000027c72496170 0000000000000000 0000027c72496170 00007ffa56a87563 : System_Private_CoreLib!System.Threading.Tasks.Task.FinishSlow+0x37
000000ab829ff5b0 00007ffa27eca333 : 0000023c67548210 0000023c675495d8 0000023c5c822590 0000000000000000 : System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0x116
000000ab829ff660 00007ffa27d84cb4 : 0000000000000008 0000000000000130 000000ab829ff770 00007ffa00000000 : coreclr!CallDescrWorkerInternal+0x83
000000ab829ff6a0 00007ffa27eb74f3 : 0000023c5c81a250 0000000000000000 0000027c6f03df7c 0000000000000000 : coreclr!DispatchCallSimple+0x60
000000ab829ff730 00007ffa27e589dd : 0000027c6f03df70 00007ffa27e59308 0000027c00004508 0000023c5c820008 : coreclr!ThreadNative::KickOffThread_Worker+0x63
000000ab829ff790 00007ffa27e588f3 : 000000ab829ff808 0000023c5c853fe0 0000000000000000 0000000000000000 : coreclr!ManagedThreadBase_DispatchMiddle+0x85
000000ab829ff870 00007ffa27e58a8e : 0000027c00000001 ffffffffffffffff 0000023c5c822590 0000023c5c822590 : coreclr!ManagedThreadBase_DispatchOuter+0xab
000000ab829ff910 00007ffa56ee4de0 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : coreclr!ThreadNative::KickOffThread+0x7e
000000ab829ff970 00007ffa5741ed9b : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0x10
000000ab829ff9a0 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x2b

FAULTING_SOURCE_LINE: D:\a_work\1\s\src\coreclr\classlibnative\bcltype\objectnative.cpp

FAULTING_SOURCE_FILE: D:\a_work\1\s\src\coreclr\classlibnative\bcltype\objectnative.cpp

FAULTING_SOURCE_LINE_NUMBER: 340

FAULTING_SOURCE_SRV_COMMAND: https://raw.githubusercontent.com/dotnet/runtime/bf5e279d9239bfef5bb1b8d6212f1b971c434606/src/coreclr/classlibnative/bcltype/objectnative.cpp

FAULTING_SOURCE_CODE:
No source found for 'D:\a_work\1\s\src\coreclr\vm\synch.cpp'

SYMBOL_NAME: coreclr!ObjectNative::PulseAll+126

MODULE_NAME: coreclr

IMAGE_NAME: coreclr.dll

STACK_COMMAND: ~904s; .ecxr ; kb

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_coreclr.dll!ObjectNative::PulseAll

OS_VERSION: 10.0.20348.1

BUILDLAB_STR: fe_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 8.0.123.58001

FAILURE_ID_HASH: {be7e1d5f-f864-f771-a72a-7be333bc8c49}

Followup: MachineOwner

Regression?

We dont have crash on Framework 4.5 / 4.8, but it reproduced on NET 7 / 8

Known Workarounds

Service MySQL56

Configuration

NET 8.0.1, VM with Windows Server 2022 x64, 48 GB RAM

Other information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-System.Threadingneeds-further-triageIssue has been initially triaged, but needs deeper consideration or reconsideration

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions