Assert failure: orObject->HasEmptySyncBlockInfo()

[jakobbotsch]

Error Blob

{
  "ErrorMessage": "orObject->HasEmptySyncBlockInfo()",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Example run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=518688&view=results
Console log: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-938d4603b99d40ee8c/System.Linq.Tests/1/console.059191e6.log?helixlogtype=result

Assert failure(PID 3960 [0x00000f78], Thread: 1740 [0x06cc]): orObject->HasEmptySyncBlockInfo()

CORECLR! GetCLRRuntimeHost + 0x20B8E2 (0x720e4292)
CORECLR! GetCLRRuntimeHost + 0x20D873 (0x720e6223)
CORECLR! GetCLRRuntimeHost + 0x20D95F (0x720e630f)
CORECLR! GetCLRRuntimeHost + 0x1B338F (0x7208bd3f)
<no module>! <no symbol> + 0x0 (0x0dd6c86b)
<no module>! <no symbol> + 0x0 (0x0dd6c0e6)
<no module>! <no symbol> + 0x0 (0x0dd6bfc3)
<no module>! <no symbol> + 0x0 (0x0dd6bf19)
<no module>! <no symbol> + 0x0 (0x0dd6bc8a)
<no module>! <no symbol> + 0x0 (0x0cdc1b51)
    File: D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp Line: 312
    Image: C:\h\w\A460091B\p\dotnet.exe

----- end Tue 01/09/2024  1:02:09.32 ----- exit code -1073740286 ----------------------------------------------------------

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=518688
Error message validated: orObject->HasEmptySyncBlockInfo()
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/10/2024 5:48:37 PM UTC

Report

Build	Definition	Test	Pull Request
532750	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#97129
532618	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96826
532153	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#97163
531443	dotnet/runtime	System.Linq.Tests.WorkItemExecution	#97075
531671	dotnet/runtime	System.Memory.Tests.WorkItemExecution	#97122
531562	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#97140
531561	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
531536	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution
531484	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#97114
531157	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#94873
530971	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#97080
531127	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#97075
530614	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96806
530405	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96386
530402	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96386
530363	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#97075
529245	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#90505
529908	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#97080
528662	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96445
529901	dotnet/runtime	System.Linq.Tests.WorkItemExecution
528807	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#97035
528493	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#96974
528246	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
528161	dotnet/runtime	System.Memory.Tests.WorkItemExecution	#97014
528107	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#97012
527944	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96951
527920	dotnet/runtime	System.Linq.Tests.WorkItemExecution	#96982
527832	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#96890
527514	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96445
527531	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96982
527253	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96741
527143	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
526876	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
2353335	dotnet-runtime	System.Collections.Immutable.Tests.WorkItemExecution
526820	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
526607	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
526456	dotnet/runtime	System.Linq.Tests.WorkItemExecution
526430	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution
526156	dotnet/runtime	System.Linq.Tests.WorkItemExecution
526132	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#96931
525892	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96201
525785	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96913
525140	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#96676
525086	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96893
524625	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96877
524210	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
524193	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96858
524101	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96699
524015	dotnet/runtime	System.Linq.Tests.WorkItemExecution	#96827
523717	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	#96445
523353	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
522397	dotnet/runtime	System.Linq.Tests.WorkItemExecution	#96773
521032	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#96684
520326	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96684
520539	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution	#96609
518688	dotnet/runtime	System.Linq.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
9	45	56

[BruceForstall]

Another case: https://dev.azure.com/dnceng-public/public/_build/results?buildId=523353&view=ms.vss-test-web.build-test-results-tab

[BruceForstall]

I've seen this in various pipelines in the last week.

[BruceForstall]

@mangod9

[BruceForstall]

Another case (internal pipeline):

https://dev.azure.com/dnceng/internal/_build/results?buildId=2353335&view=ms.vss-test-web.build-test-results-tab&runId=52868327&resultId=207146&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

[BruceForstall]

Another:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=528246&view=ms.vss-test-web.build-test-results-tab&runId=12443784&resultId=210683&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

[BruceForstall]

@janvorli @trylek

[mangod9]

Possibly some GC hole. Assume we can narrow down to a set of PRs when this started occuring?

[jkotas]

The typical stacktrace of the crash has GC.AllocateUninitializedArray called from SegmentedArrayBuilder:

03 0ec9dcb0 71d332b2     coreclr!DbgAssertDialog+0x20c [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 459] 
04 0ec9dd04 71d35243     coreclr!PublishObjectAndNotify<ArrayBase>+0x2b [D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp @ 314] 
05 0ec9dda8 71d3532f     coreclr!AllocateSzArray+0x315 [D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp @ 487] 
06 0ec9de28 71cdaf5f     coreclr!AllocateSzArray+0x98 [D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp @ 367] 
07 0ec9df44 1269ff48     coreclr!GCInterface::AllocateNewArray+0x1af [D:\a\_work\1\s\src\coreclr\vm\comutilnative.cpp @ 959] 
08 0ec9df6c 1270fcde     System_Private_CoreLib!System.GC.AllocateUninitializedArray[[System.SByte, System.Private.CoreLib]](Int32, Boolean)+0x2a58
09 0ec9dfa8 1270f163     System_Linq!System.Collections.Generic.SegmentedArrayBuilder`1[[System.Double, System.Private.CoreLib]].ToArray(Int32)+0x76

SegmentedArray builder was introduced by #96570 last week. We are calling and stressing GC.AllocateUninitializedArray a lot more now than before. It is likely a pre-existing bug that #96570 exposed.

The assert is complaining about syncblk index memory returned by GCHeap::Alloc not being zeroed. We may be missing a path in GC to clear the syncblk index with GC_ALLOC_ZEROING_OPTIONAL. cc @dotnet/gc

Note that we are hitting this crash on x86 only (no GC regions).

[mangod9]

Ok interesting. @cshung, would you be able to take a look since this is CI blocking? Thx!

[cshung]

Is it possible to make this easier to reproduce locally?

[mangod9]

should be easy to repro given frequent failures in CI. Note that this is on x86 only.

[cshung]

should be easy to repro given frequent failures in CI. Note that this is on x86 only.

It maybe obvious for you, but exactly how? I have got the repo built.

[jkotas]

It maybe obvious for you, but exactly how? I have got the repo built.

It is documented at https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/libraries/testing.md#testing-libraries . The command line for this case would be: build.cmd -subset clr+libs+libs.tests -test -arch x86 -rc checked -c release

[jkotas]

I have reviewed all allocation paths and found a few places where we are missing clearning of the syncblock: #97174