Bad async codegen for Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync

[MichalStrehovsky]

In #122526 I'm running native AOT outerloop pipelines with runtime async enabled. I see consistent crashes that don't seem platform specific while running the Microsoft.Extensions.Caching.Memory.Tests. Curiously the CI is not hitting it on Windows x64 (and I can't repro it locally on Win x64), but it does repro on all Linux/mac legs and also Windows ARM64.

Repro steps are basically:

• curl -L https://github.com/dotnet/runtime/pull/122526.diff | git apply --3way
• curl -L https://github.com/dotnet/runtime/pull/124488.diff | git apply --3way
• build clr.aot+libs -rc Checked -lc Release
• dotnet build src\libraries\Microsoft.Extensions.Caching.Memory\tests\Microsoft.Extensions.Caching.Memory.Tests.csproj -f net11.0 -p:TestNativeAot=true -p:RuntimeConfiguration=Checked -c Release
• Microsoft.Extensions.Caching.Memory.Tests -notrait category=failing -class Microsoft.Extensions.Caching.Memory.CacheEntryScopeExpirationTests

We're hitting a nullref at address 00005555`560063f0 below (register r12 is zero).

As to why register r12 is zero (didn't debug this, inferring this from a crashdump and disassembly):

• The callstack in the crashdump shows GetOrCreateAsync was called from a resumption stub
• I assume we took the first jne after the prolog to 00005555`56006349 because we're resuming. Before we took the jne, we zeroed out r12.
• r12 was the not touched (looks like we did restore r15 though)
• We jumped back to the normal method body and dereferenced the null we never replaced with something useful.

   Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync: CFG
00005555`56006310 55                 push    rbp
00005555`56006311 4157               push    r15
00005555`56006313 4156               push    r14
00005555`56006315 4155               push    r13
00005555`56006317 4154               push    r12
00005555`56006319 53                 push    rbx
00005555`5600631a 4883ec58           sub     rsp, 58h
00005555`5600631e 488dac2480000000   lea     rbp, [rsp+80h]
00005555`56006326 450f57c0           xorps   xmm8, xmm8
00005555`5600632a 440f2945c0         movaps  xmmword ptr [rbp-40h], xmm8
00005555`5600632f 4533e4             xor     r12d, r12d
00005555`56006332 4c8965d0           mov     qword ptr [rbp-30h], r12
00005555`56006336 48897d88           mov     qword ptr [rbp-78h], rdi
00005555`5600633a 488bde             mov     rbx, rsi
00005555`5600633d 4c8bfa             mov     r15, rdx
00005555`56006340 4c8bf1             mov     r14, rcx
00005555`56006343 4d8be8             mov     r13, r8
00005555`56006346 4885ff             test    rdi, rdi
00005555`56006349 0f85d5010000       jne     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x214 (555556006524)
00005555`5600634f e86c53d1ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Threading_Thread::get_CurrentThreadAssumedInitialized (555555d1b6c0)
00005555`56006354 488b7008           mov     rsi, qword ptr [rax+8]
00005555`56006358 488975d0           mov     qword ptr [rbp-30h], rsi
00005555`5600635c 488b5010           mov     rdx, qword ptr [rax+10h]
00005555`56006360 488955c8           mov     qword ptr [rbp-38h], rdx
00005555`56006364 488d55c0           lea     rdx, [rbp-40h]
00005555`56006368 488bfb             mov     rdi, rbx
00005555`5600636b 498bf7             mov     rsi, r15
00005555`5600636e 4c8d1ddb111900     lea     r11, [555556197550h]
00005555`56006375 41ff13             call    qword ptr [r11]
00005555`56006378 85c0               test    eax, eax
00005555`5600637a 0f85a5000000       jne     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x115 (555556006425)
00005555`56006380 488bfb             mov     rdi, rbx
00005555`56006383 498bf7             mov     rsi, r15
00005555`56006386 4c8d1d93931900     lea     r11, [55555619F720h]
00005555`5600638d 41ff13             call    qword ptr [r11]
00005555`56006390 48894580           mov     qword ptr [rbp-80h], rax
00005555`56006394 4d85ed             test    r13, r13
00005555`56006397 740f               je      Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x98 (5555560063a8)
00005555`56006399 488bf8             mov     rdi, rax
00005555`5600639c 498bf5             mov     rsi, r13
00005555`5600639f e8dc90c1ff         call    Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheEntryExtensions::SetOptions (555555c1f480)
00005555`560063a4 488b4580           mov     rax, qword ptr [rbp-80h]
00005555`560063a8 488bf0             mov     rsi, rax
00005555`560063ab 498b7e08           mov     rdi, qword ptr [r14+8]
00005555`560063af 41ff5620           call    qword ptr [r14+20h]
00005555`560063b3 488bd8             mov     rbx, rax
00005555`560063b6 381b               cmp     byte ptr [rbx], bl
00005555`560063b8 488d3d69581c00     lea     rdi, [5555561CBC28h]
00005555`560063bf e82c23baff         call    Microsoft_Extensions_Caching_Memory!RhpNewFast (555555ba86f0)
00005555`560063c4 4c8bf8             mov     r15, rax
00005555`560063c7 4d8d6708           lea     r12, [r15+8]
00005555`560063cb 488bf3             mov     rsi, rbx
00005555`560063ce 33d2               xor     edx, edx
00005555`560063d0 33ff               xor     edi, edi
00005555`560063d2 e869f5ffff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::Await (555556005940)
00005555`560063d7 48894590           mov     qword ptr [rbp-70h], rax
00005555`560063db 48895598           mov     qword ptr [rbp-68h], rdx
00005555`560063df 4885c9             test    rcx, rcx
00005555`560063e2 0f85bc000000       jne     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x194 (5555560064a4)
00005555`560063e8 0f104590           movups  xmm0, xmmword ptr [rbp-70h]
00005555`560063ec 0f1145a0           movups  xmmword ptr [rbp-60h], xmm0
00005555`560063f0 410f110424         movups  xmmword ptr [r12], xmm0
00005555`560063f5 4c897dc0           mov     qword ptr [rbp-40h], r15
00005555`560063f9 488b75c0           mov     rsi, qword ptr [rbp-40h]
00005555`560063fd 488b4580           mov     rax, qword ptr [rbp-80h]
00005555`56006401 488d7820           lea     rdi, [rax+20h]
00005555`56006405 e88632baff         call    Microsoft_Extensions_Caching_Memory!RhpAssignRefESI (555555ba9690)
00005555`5600640a 488b4580           mov     rax, qword ptr [rbp-80h]
00005555`5600640e c6404401           mov     byte ptr [rax+44h], 1
00005555`56006412 eb09               jmp     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x10d (55555600641d)
00005555`56006414 488bf8             mov     rdi, rax
00005555`56006417 e8d425baff         call    Microsoft_Extensions_Caching_Memory!RhpThrowExact (555555ba89f0)
00005555`5600641c cc                 int     3
00005555`5600641d 488bf8             mov     rdi, rax
00005555`56006420 e81b9cc1ff         call    Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Memory_Microsoft_Extensions_Caching_Memory_CacheEntry::Dispose (555555c20040)
00005555`56006425 488b75c0           mov     rsi, qword ptr [rbp-40h]
00005555`56006429 488d3df8571c00     lea     rdi, [5555561CBC28h]
00005555`56006430 48393e             cmp     qword ptr [rsi], rdi
00005555`56006433 7410               je      Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x135 (555556006445)
00005555`56006435 488b75c0           mov     rsi, qword ptr [rbp-40h]
00005555`56006439 488d3de8571c00     lea     rdi, [5555561CBC28h]
00005555`56006440 e84b78d7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_RuntimeExports::RhUnbox2 (555555d7dc90)
00005555`56006445 488b45c0           mov     rax, qword ptr [rbp-40h]
00005555`56006449 0f104008           movups  xmm0, xmmword ptr [rax+8]
00005555`5600644d 0f1145b0           movups  xmmword ptr [rbp-50h], xmm0
00005555`56006451 48837d8800         cmp     qword ptr [rbp-78h], 0
00005555`56006456 7533               jne     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x17b (55555600648b)
00005555`56006458 e86352d1ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Threading_Thread::get_CurrentThreadAssumedInitialized (555555d1b6c0)
00005555`5600645d 488bd8             mov     rbx, rax
00005555`56006460 488b55c8           mov     rdx, qword ptr [rbp-38h]
00005555`56006464 483b5310           cmp     rdx, qword ptr [rbx+10h]
00005555`56006468 740c               je      Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x166 (555556006476)
00005555`5600646a 488d7b10           lea     rdi, [rbx+10h]
00005555`5600646e 488bf2             mov     rsi, rdx
00005555`56006471 e81a32baff         call    Microsoft_Extensions_Caching_Memory!RhpAssignRefESI (555555ba9690)
00005555`56006476 488b5308           mov     rdx, qword ptr [rbx+8]
00005555`5600647a 488b75d0           mov     rsi, qword ptr [rbp-30h]
00005555`5600647e 483bf2             cmp     rsi, rdx
00005555`56006481 7408               je      Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x17b (55555600648b)
00005555`56006483 488bfb             mov     rdi, rbx
00005555`56006486 e86d7bd1ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Threading_ExecutionContext::RestoreChangedContextToThread (555555d1dff8)
00005555`5600648b 488b45b0           mov     rax, qword ptr [rbp-50h]
00005555`5600648f 488b55b8           mov     rdx, qword ptr [rbp-48h]
00005555`56006493 33c9               xor     ecx, ecx
00005555`56006495 4883c458           add     rsp, 58h
00005555`56006499 5b                 pop     rbx
00005555`5600649a 415c               pop     r12
00005555`5600649c 415d               pop     r13
00005555`5600649e 415e               pop     r14
00005555`560064a0 415f               pop     r15
00005555`560064a2 5d                 pop     rbp
00005555`560064a3 c3                 ret     
00005555`560064a4 488bf9             mov     rdi, rcx
00005555`560064a7 488d3542cb1a00     lea     rsi, [5555561B2FF0h]
00005555`560064ae e8edbad7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::AllocContinuation (555555d81fa0)
00005555`560064b3 4c8be0             mov     r12, rax
00005555`560064b6 488d3db3480900     lea     rdi, [Microsoft_Extensions_Caching_Memory!__readwritedata_Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions__<AsyncCallable>Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions__GetOrCreateAsync_0<S_P_CoreLib_System_Guid> (55555609ad70)]
00005555`560064bd 49897c2410         mov     qword ptr [r12+10h], rdi
00005555`560064c2 49c74424180a000000 mov     qword ptr [r12+18h], 0Ah
00005555`560064cb 498d7c2440         lea     rdi, [r12+40h]
00005555`560064d0 488b7580           mov     rsi, qword ptr [rbp-80h]
00005555`560064d4 e8b731baff         call    Microsoft_Extensions_Caching_Memory!RhpAssignRefESI (555555ba9690)
00005555`560064d9 498d7c2448         lea     rdi, [r12+48h]
00005555`560064de 498bf7             mov     rsi, r15
00005555`560064e1 e8aa31baff         call    Microsoft_Extensions_Caching_Memory!RhpAssignRefESI (555555ba9690)
00005555`560064e6 e869bcd7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::CaptureExecutionContext (555555d82154)
00005555`560064eb 498d7c2438         lea     rdi, [r12+38h]
00005555`560064f0 488bf0             mov     rsi, rax
00005555`560064f3 e89831baff         call    Microsoft_Extensions_Caching_Memory!RhpAssignRefESI (555555ba9690)
00005555`560064f8 48837d8800         cmp     qword ptr [rbp-78h], 0
00005555`560064fd 400f95c7           setne   dil
00005555`56006501 400fb6ff           movzx   edi, dil
00005555`56006505 488b75d0           mov     rsi, qword ptr [rbp-30h]
00005555`56006509 488b55c8           mov     rdx, qword ptr [rbp-38h]
00005555`5600650d e806bdd7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::RestoreContextsOnSuspension (555555d82218)
00005555`56006512 498bcc             mov     rcx, r12
00005555`56006515 4883c458           add     rsp, 58h
00005555`56006519 5b                 pop     rbx
00005555`5600651a 415c               pop     r12
00005555`5600651c 415d               pop     r13
00005555`5600651e 415e               pop     r14
00005555`56006520 415f               pop     r15
00005555`56006522 5d                 pop     rbp
00005555`56006523 c3                 ret     
00005555`56006524 488b7d88           mov     rdi, qword ptr [rbp-78h]
00005555`56006528 488b7f38           mov     rdi, qword ptr [rdi+38h]
00005555`5600652c e837bcd7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::RestoreExecutionContext (555555d82168)
00005555`56006531 488b7d88           mov     rdi, qword ptr [rbp-78h]
00005555`56006535 488b7740           mov     rsi, qword ptr [rdi+40h]
00005555`56006539 48897580           mov     qword ptr [rbp-80h], rsi
00005555`5600653d 4c8b7f48           mov     r15, qword ptr [rdi+48h]
00005555`56006541 488b4720           mov     rax, qword ptr [rdi+20h]
00005555`56006545 4885c0             test    rax, rax
00005555`56006548 0f85c6feffff       jne     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x104 (555556006414)
00005555`5600654e 0f104728           movups  xmm0, xmmword ptr [rdi+28h]
00005555`56006552 0f114590           movups  xmmword ptr [rbp-70h], xmm0
00005555`56006556 e98dfeffff         jmp     Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0xd8 (5555560063e8)
00005555`5600655b 50                 push    rax
00005555`5600655c 48837d8000         cmp     qword ptr [rbp-80h], 0
00005555`56006561 7409               je      Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Abstractions_Microsoft_Extensions_Caching_Memory_CacheExtensions::GetOrCreateAsync+0x25c (55555600656c)
00005555`56006563 488b7d80           mov     rdi, qword ptr [rbp-80h]
00005555`56006567 e8d49ac1ff         call    Microsoft_Extensions_Caching_Memory!Microsoft_Extensions_Caching_Memory_Microsoft_Extensions_Caching_Memory_CacheEntry::Dispose (555555c20040)
00005555`5600656c 90                 nop     
00005555`5600656d 4883c408           add     rsp, 8
00005555`56006571 c3                 ret     
00005555`56006572 50                 push    rax
00005555`56006573 48837d8800         cmp     qword ptr [rbp-78h], 0
00005555`56006578 400f95c7           setne   dil
00005555`5600657c 400fb6ff           movzx   edi, dil
00005555`56006580 488b75d0           mov     rsi, qword ptr [rbp-30h]
00005555`56006584 488b55c8           mov     rdx, qword ptr [rbp-38h]
00005555`56006588 e82fbcd7ff         call    Microsoft_Extensions_Caching_Memory!S_P_CoreLib_System_Runtime_CompilerServices_AsyncHelpers::RestoreContexts (555555d821bc)
00005555`5600658d 90                 nop     
00005555`5600658e 4883c408           add     rsp, 8
00005555`56006592 c3                 ret     
00005555`56006593 90                 nop     

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

The thing that seems to be special with r12 seems to be that it's an interior pointer into a boxed GUID (I assume interior pointers are going to be very special around resumption):

It was created here

00005555`560063bf e82c23baff         call    Microsoft_Extensions_Caching_Memory!RhpNewFast (555555ba86f0) ; <---- VTable of System.Guid
00005555`560063c4 4c8bf8             mov     r15, rax
00005555`560063c7 4d8d6708           lea     r12, [r15+8]

And we try to use it a couple lines below, past the resumption point.

[jakobbotsch]

I assume interior pointers are going to be very special around resumption

Yes, we cannot save/restore them. So Roslyn has to be careful that it doesn't create them across suspension points and the JIT similarly must take care not to create them. Likely we are missing some case here.