Assert failure: (FARPROC) (TADDR)m_pvHJRetAddr != NULL

[BruceForstall]

Seen on Ubuntu.1804 x64 with release corefx tests and Checked coreclr and:

COMPlus_TieredCompilation=0
COMPlus_JITMinOpts=1

in Microsoft.VisualBasic.Core.Tests

https://mc.dot.net/#/user/coreclr-corefx-jitstress/pr~2Fdotnet~2Fcoreclr~2Frefs~2Fpull~2F24962~2Fmerge/test~2Ffunctional~2Fcorefx~2F/20190610.2/workItem/Microsoft.VisualBasic.Core.Tests/wilogs

(corefx testing is not yet enabled in master; it is here: dotnet/coreclr#24962)

Executed on a001C8K
+ export __TestEnv=/home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/SetStressModes_jitminopts.sh
+ cat /home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/SetStressModes_jitminopts.sh
#!/usr/bin/env bash
export COMPlus_TieredCompilation=0
export COMPlus_JITMinOpts=1
+ . /home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/SetStressModes_jitminopts.sh
+ export COMPlus_TieredCompilation=0
+ export COMPlus_JITMinOpts=1
+ printenv
+ grep COMPlus
COMPlus_TieredCompilation=0
COMPlus_JITMinOpts=1
+ ./RunTests.sh -r /home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload --rsp-file /home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/CoreFX.issues.rsp
----- start Tue Jun 11 01:03:54 UTC 2019 =============== To repro directly: =====================================================
pushd .
/home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/dotnet xunit.console.dll Microsoft.VisualBasic.Core.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing @/home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/CoreFX.issues.rsp
popd
===========================================================================================================
~/work/7f4438de-93cd-4a69-99ba-3121b50961da/Work/73c5d6bc-5f1a-42f7-af2e-25983d11ec9b/Exec ~/work/7f4438de-93cd-4a69-99ba-3121b50961da/Work/73c5d6bc-5f1a-42f7-af2e-25983d11ec9b/Exec
  Discovering: Microsoft.VisualBasic.Core.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  Microsoft.VisualBasic.Core.Tests (found 723 of 730 test cases)
  Starting:    Microsoft.VisualBasic.Core.Tests (parallel test collections = on, max threads = 2)
    Microsoft.VisualBasic.Tests.FileSystemTests.CurDir_Drive [SKIP]
      Condition(s) not met: "IsWindows"
    Microsoft.VisualBasic.Tests.FileSystemTests.Lock_Unlock [SKIP]
      Condition(s) not met: "IsWindows"

Assert failure(PID 17514 [0x0000446a], Thread: 17514 [0x446a]): (FARPROC) (TADDR)m_pvHJRetAddr != NULL
    File: /__w/6/s/src/vm/threadsuspend.cpp Line: 5431
    Image: /home/helixbot/work/7f4438de-93cd-4a69-99ba-3121b50961da/Payload/dotnet

./RunTests.sh: line 164: 17514 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" xunit.console.dll Microsoft.VisualBasic.Core.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing $RSP_FILE

part of stack trace from dump:

dotnet/coreclr#7  0x00007f060748c2c9 in DbgAssertDialog (szFile=0x7f0607f2e718 "/__w/6/s/src/vm/threadsuspend.cpp", iLine=5431, szExpr=0x7f0607f2f871 "(FARPROC) (TADDR)m_pvHJRetAddr != NULL") at /__w/6/s/src/utilcode/debug.cpp:708
        szExprToDisplay = 0x7f0607f2f871 "(FARPROC) (TADDR)m_pvHJRetAddr != NULL"
        fGotStackTrace = 1
        fConstrained = 0
        dwAssertStacktrace = 1
        lAlreadyOwned = -1
dotnet/coreclr#8  0x00007f060780bc69 in Thread::HijackThread (this=0xa3fb60, pvHijackAddr=0x7f060786bf18 <OnHijackTripThread>, esb=0x7ffe5bbd0160) at /__w/6/s/src/vm/threadsuspend.cpp:5431
        hijackLockHolder = {m_value = 0xa3fb60, m_acquired = 1}
dotnet/coreclr#9  0x00007f060780d73d in HandleGCSuspensionForInterruptedThread (interruptedContext=0x7ffe5bbd0e50) at /__w/6/s/src/vm/threadsuspend.cpp:6758
        action = SWA_ABORT
        regDisplay = {<REGDISPLAY_BASE> = {pContext = 0x7ffe5bbd0e50, pCurrentContext = 0x7ffe5bbd01e0, pCallerContext = 0x7ffe5bbd06b0, pCurrentContextPointers = 0x7ffe5bbd0b80, pCallerContextPointers = 0x7ffe5bbd0c80, IsCallerContextValid = 1, IsCallerSPValid = 0, ctxOne = {P1Home = 0, P2Home = 0, P3Home = 0, P4Home = 0, P5Home = 0, P6Home = 0, ContextFlags = 1048587, MxCsr = 32517, SegCs = 51, SegDs = 27002, SegEs = 32517, SegFs = 0, SegGs = 50904, SegSs = 27002, EFlags = 582, Dr0 = 139661221217984, Dr1 = 139661221217896, Dr2 = 139661221217960, Dr3 = 0, Dr6 = 0, Dr7 = 0, Rax = 140730437515056, Rcx = 0, Rdx = 139661221219568, Rbx = 140730437569608, Rsp = 140730437540056, Rbp = 140730437556592, Rsi = 139661221219568, Rdi = 139661218950680, R8 = 10746720, R9 = 139663910797472, R10 = 9223441867774963868, R11 = 139661825345416, R12 = 139661201418992, R13 = 139661218950776, R14 = 14, R15 = 11091952, Rip = 139661861772857, {FltSave = {ControlWord = 895, StatusWord = 0, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 154808064, ErrorSelector = 0, Reserved2 = 0, DataOffset = 1539151080, DataSelector = 0, Reserved3 = 0, MxCsr = 8111, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, XmmRegisters = {{Low = 32651591226294356, High = 28429466575372355}, {Low = 1090519040, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 512, High = 7229172904379708760}, {Low = 139661196329067, High = 139661196386304}, {Low = 0, High = 0}, {Low = 140730437545336, High = 140730437556336}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4602678819172646912, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4408728642709317424, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}}, Reserved4 = '\000' <repeats 95 times>}, {Header = {{Low = 895, High = 154808064}, {Low = 1539151080, High = 281470681751471}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, Xmm0 = {Low = 32651591226294356, High = 28429466575372355}, Xmm1 = {Low = 1090519040, High = 0}, Xmm2 = {Low = 18446744073709551615, High = -1}, Xmm3 = {Low = 512, High = 7229172904379708760}, Xmm4 = {Low = 139661196329067, High = 139661196386304}, Xmm5 = {Low = 0, High = 0}, Xmm6 = {Low = 140730437545336, High = 140730437556336}, Xmm7 = {Low = 0, High = 0}, Xmm8 = {Low = 0, High = 0}, Xmm9 = {Low = 0, High = 0}, Xmm10 = {Low = 4602678819172646912, High = 0}, Xmm11 = {Low = 0, High = 0}, Xmm12 = {Low = 0, High = 0}, Xmm13 = {Low = 4408728642709317424, High = 0}, Xmm14 = {Low = 0, High = 0}, Xmm15 = {Low = 0, High = 0}}}, VectorRegister = {{Low = 0, High = 0} <repeats 26 times>}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 0, LastBranchFromRip = 0, LastExceptionToRip = 0, LastExceptionFromRip = 0}, ctxTwo = {P1Home = 0, P2Home = 0, P3Home = 0, P4Home = 0, P5Home = 0, P6Home = 0, ContextFlags = 1048587, MxCsr = 32517, SegCs = 51, SegDs = 27002, SegEs = 32517, SegFs = 0, SegGs = 50904, SegSs = 27002, EFlags = 582, Dr0 = 139661221217984, Dr1 = 139661221217896, Dr2 = 139661221217960, Dr3 = 0, Dr6 = 0, Dr7 = 0, Rax = 140730437515056, Rcx = 0, Rdx = 139661221219568, Rbx = 140730437569608, Rsp = 140730437540080, Rbp = 0, Rsi = 139661221219568, Rdi = 139661218950680, R8 = 10746720, R9 = 139663910797472, R10 = 9223441867774963868, R11 = 139661825345416, R12 = 139661201418992, R13 = 0, R14 = 14, R15 = 11091952, Rip = 0, {FltSave = {ControlWord = 895, StatusWord = 0, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 154808064, ErrorSelector = 0, Reserved2 = 0, DataOffset = 1539151080, DataSelector = 0, Reserved3 = 0, MxCsr = 8111, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, XmmRegisters = {{Low = 32651591226294356, High = 28429466575372355}, {Low = 1090519040, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 512, High = 7229172904379708760}, {Low = 139661196329067, High = 139661196386304}, {Low = 0, High = 0}, {Low = 140730437545336, High = 140730437556336}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4602678819172646912, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4408728642709317424, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}}, Reserved4 = '\000' <repeats 95 times>}, {Header = {{Low = 895, High = 154808064}, {Low = 1539151080, High = 281470681751471}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, Xmm0 = {Low = 32651591226294356, High = 28429466575372355}, Xmm1 = {Low = 1090519040, High = 0}, Xmm2 = {Low = 18446744073709551615, High = -1}, Xmm3 = {Low = 512, High = 7229172904379708760}, Xmm4 = {Low = 139661196329067, High = 139661196386304}, Xmm5 = {Low = 0, High = 0}, Xmm6 = {Low = 140730437545336, High = 140730437556336}, Xmm7 = {Low = 0, High = 0}, Xmm8 = {Low = 0, High = 0}, Xmm9 = {Low = 0, High = 0}, Xmm10 = {Low = 4602678819172646912, High = 0}, Xmm11 = {Low = 0, High = 0}, Xmm12 = {Low = 0, High = 0}, Xmm13 = {Low = 4408728642709317424, High = 0}, Xmm14 = {Low = 0, High = 0}, Xmm15 = {Low = 0, High = 0}}}, VectorRegister = {{Low = 0, High = 0} <repeats 26 times>}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 0, LastBranchFromRip = 0, LastExceptionToRip = 0, LastExceptionFromRip = 0}, ctxPtrsOne = {{FloatingContext = {0x0 <repeats 16 times>}, {Xmm0 = 0x0, Xmm1 = 0x0, Xmm2 = 0x0, Xmm3 = 0x0, Xmm4 = 0x0, Xmm5 = 0x0, Xmm6 = 0x0, Xmm7 = 0x0, Xmm8 = 0x0, Xmm9 = 0x0, Xmm10 = 0x0, Xmm11 = 0x0, Xmm12 = 0x0, Xmm13 = 0x0, Xmm14 = 0x0, Xmm15 = 0x0}}, {IntegerContext = {0x7ffe5bbd0ec8, 0x7ffe5bbd0ed0, 0x7ffe5bbd0ed8, 0x7ffe5bbd0ee0, 0x7ffe5bbd0ee8, 0x7ffe5bbd0ef0, 0x7ffe5bbd0ef8, 0x7ffe5bbd0f00, 0x7ffe5bbd0f08, 0x7ffe5bbd0f10, 0x7ffe5bbd0f18, 0x7ffe5bbd0f20, 0x7ffe5bbd0f28, 0x7ffe5bbd0f30, 0x7ffe5bbd0f38, 0x7ffe5bbd0f40}, {Rax = 0x7ffe5bbd0ec8, Rcx = 0x7ffe5bbd0ed0, Rdx = 0x7ffe5bbd0ed8, Rbx = 0x7ffe5bbd0ee0, Rsp = 0x7ffe5bbd0ee8, Rbp = 0x7ffe5bbd0ef0, Rsi = 0x7ffe5bbd0ef8, Rdi = 0x7ffe5bbd0f00, R8 = 0x7ffe5bbd0f08, R9 = 0x7ffe5bbd0f10, R10 = 0x7ffe5bbd0f18, R11 = 0x7ffe5bbd0f20, R12 = 0x7ffe5bbd0f28, R13 = 0x7ffe5bbd0f30, R14 = 0x7ffe5bbd0f38, R15 = 0x7ffe5bbd0f40}}}, ctxPtrsTwo = {{FloatingContext = {0x0 <repeats 16 times>}, {Xmm0 = 0x0, Xmm1 = 0x0, Xmm2 = 0x0, Xmm3 = 0x0, Xmm4 = 0x0, Xmm5 = 0x0, Xmm6 = 0x0, Xmm7 = 0x0, Xmm8 = 0x0, Xmm9 = 0x0, Xmm10 = 0x0, Xmm11 = 0x0, Xmm12 = 0x0, Xmm13 = 0x0, Xmm14 = 0x0, Xmm15 = 0x0}}, {IntegerContext = {0x7ffe5bbd0ec8, 0x7ffe5bbd0ed0, 0x7ffe5bbd0ed8, 0x7ffe5bbd0ee0, 0x7ffe5bbd0ee8, 0x7ffe5bbd18e0, 0x7ffe5bbd0ef8, 0x7ffe5bbd0f00, 0x7ffe5bbd0f08, 0x7ffe5bbd0f10, 0x7ffe5bbd0f18, 0x7ffe5bbd0f20, 0x7ffe5bbd0f28, 0x7ffe5bbd18d8, 0x7ffe5bbd0f38, 0x7ffe5bbd0f40}, {Rax = 0x7ffe5bbd0ec8, Rcx = 0x7ffe5bbd0ed0, Rdx = 0x7ffe5bbd0ed8, Rbx = 0x7ffe5bbd0ee0, Rsp = 0x7ffe5bbd0ee8, Rbp = 0x7ffe5bbd18e0, Rsi = 0x7ffe5bbd0ef8, Rdi = 0x7ffe5bbd0f00, R8 = 0x7ffe5bbd0f08, R9 = 0x7ffe5bbd0f10, R10 = 0x7ffe5bbd0f18, R11 = 0x7ffe5bbd0f20, R12 = 0x7ffe5bbd0f28, R13 = 0x7ffe5bbd18d8, R14 = 0x7ffe5bbd0f38, R15 = 0x7ffe5bbd0f40}}}, _pThread = 0xa3fb60, SP = 140730437540056, ControlPC = 139661861772857}, <No data fields>}
        unused = 1
        executionState = {m_FirstPass = 1, m_IsJIT = 1, m_pFD = 0x7f058edb0700, m_ppvRetAddrPtr = 0x7ffe5bbd18e8, m_RelOffset = 25, m_pJitManager = 0xa29fc0, m_MethodToken = {m_pRangeSection = 0xfc2680, m_pCodeHeader = 139661861772824}, m_IsInterruptible = 0}
        threadStackWalking = {m_PreviousValue = 0x0, m_slot = TlsIdx_StackWalkerWalkingThread}
        pvHijackAddr = 0x7f060786bf18 <OnHijackTripThread>
        pThread = 0xa3fb60
        ip = 139661861772857
        workingOnThreadContext = {m_value = 0xa3fb60, m_acquired = 1}
        codeInfo = {m_codeAddress = 139661861772857, m_methodToken = {m_pRangeSection = 0xfc2680, m_pCodeHeader = 139661861772824}, m_pMD = 0x7f058edb0700, m_pJM = 0xa29fc0, m_relOffset = 25, m_pFunctionEntry = 0x7f058fc6a044}
        addrOffset = 25
        pEECM = 0x9985f0
        isAtSafePoint = false
dotnet/coreclr#10 0x00007f0607cefac3 in inject_activation_handler (code=34, siginfo=0x7ffe5bbd1470, context=0x7ffe5bbd1340) at /__w/6/s/src/pal/src/exception/signal.cpp:605
        ucontext = 0x7ffe5bbd1340
        winContext = {P1Home = 0, P2Home = 0, P3Home = 0, P4Home = 0, P5Home = 0, P6Home = 0, ContextFlags = 1048587, MxCsr = 32517, SegCs = 51, SegDs = 27002, SegEs = 32517, SegFs = 0, SegGs = 50904, SegSs = 27002, EFlags = 582, Dr0 = 139661221217984, Dr1 = 139661221217896, Dr2 = 139661221217960, Dr3 = 0, Dr6 = 0, Dr7 = 0, Rax = 140730437515056, Rcx = 0, Rdx = 139661221219568, Rbx = 140730437569608, Rsp = 140730437540056, Rbp = 140730437556592, Rsi = 139661221219568, Rdi = 139661218950680, R8 = 10746720, R9 = 139663910797472, R10 = 9223441867774963868, R11 = 139661825345416, R12 = 139661201418992, R13 = 139661218950776, R14 = 14, R15 = 11091952, Rip = 139661861772857, {FltSave = {ControlWord = 895, StatusWord = 0, TagWord = 0 '\000', Reserved1 = 0 '\000', ErrorOpcode = 0, ErrorOffset = 154808064, ErrorSelector = 0, Reserved2 = 0, DataOffset = 1539151080, DataSelector = 0, Reserved3 = 0, MxCsr = 8111, MxCsr_Mask = 65535, FloatRegisters = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, XmmRegisters = {{Low = 32651591226294356, High = 28429466575372355}, {Low = 1090519040, High = 0}, {Low = 18446744073709551615, High = -1}, {Low = 512, High = 7229172904379708760}, {Low = 139661196329067, High = 139661196386304}, {Low = 0, High = 0}, {Low = 140730437545336, High = 140730437556336}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4602678819172646912, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 4408728642709317424, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}}, Reserved4 = '\000' <repeats 95 times>}, {Header = {{Low = 895, High = 154808064}, {Low = 1539151080, High = 281470681751471}}, Legacy = {{Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 0, High = 0}, {Low = 14339461213547659264, High = 16390}, {Low = 14339461213547659264, High = 16390}, {Low = 12682136550675316736, High = 16386}}, Xmm0 = {Low = 32651591226294356, High = 28429466575372355}, Xmm1 = {Low = 1090519040, High = 0}, Xmm2 = {Low = 18446744073709551615, High = -1}, Xmm3 = {Low = 512, High = 7229172904379708760}, Xmm4 = {Low = 139661196329067, High = 139661196386304}, Xmm5 = {Low = 0, High = 0}, Xmm6 = {Low = 140730437545336, High = 140730437556336}, Xmm7 = {Low = 0, High = 0}, Xmm8 = {Low = 0, High = 0}, Xmm9 = {Low = 0, High = 0}, Xmm10 = {Low = 4602678819172646912, High = 0}, Xmm11 = {Low = 0, High = 0}, Xmm12 = {Low = 0, High = 0}, Xmm13 = {Low = 4408728642709317424, High = 0}, Xmm14 = {Low = 0, High = 0}, Xmm15 = {Low = 0, High = 0}}}, VectorRegister = {{Low = 0, High = 0} <repeats 26 times>}, VectorControl = 0, DebugControl = 0, LastBranchToRip = 0, LastBranchFromRip = 0, LastExceptionToRip = 0, LastExceptionFromRip = 0}
dotnet/coreclr#11 <signal handler called>
No locals.
dotnet/coreclr#12 0x00007f058fa8de39 in ?? ()

[BruceForstall]

@janvorli

[janvorli]

@BruceForstall can you share the core dump?

[BruceForstall]

@janvorli The core dump was taken on a Helix lab machine, so I don't have it (and they haven't updated Helix to save these yet, I believe).

[janvorli]

@BruceForstall ah, I see. Is there a way to run the tests locally (or do we plan to add the ability back to runtests.cmd?)

[BruceForstall]

It is certainly possible to run the tests locally. It's a little hard to describe. Basically: (1) build or obtain a prebuilt copy of the corefx tests (or the one test assembly in question), (2) overlay a Checked coreclr, (3) run.

You can do that via the "old" method of the dotnet/coreclr tests/scripts/run-corefx-tests.py script, which does all this (by cloning and building corefx; run the script in an environment with an already-built coreclr).

The "new" method is, basically (but not easily): (1) build coreclr, (2) build a "test host" using build-test.cmd checked x64 buildtesthostonly (similar on Linux), (3) download and unzip the appropriate test listed in https://dotnetfeed.blob.core.windows.net/dotnet-core/corefx-tests/4.6.0-preview7.19309.5/Windows_NT.x64/netcoreapp/corefx-test-assets.xml (this list/URL changes based on the current corefx version in coreclr), (4) run the test using, e.g., RunTests.cmd script. Don't forget to set the appropriate COMPlus variables first.

[janvorli]

I have been running the Microsoft.VisualBasic.Core.Tests in a loop for three days (around 6000 iterations) and I wasn't able to repro the issue. I guess that a likely reason for the failure is an incorrect extraction of the return address location for some function(s).

[janvorli]

The tests that were failing with this issue were re-enabled 4 days ago and the issue hasn't reproduced in the lab yet. Since I was unable to repro the issue locally when running the test both with and without
added stress, I am closing the issue for now.