You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Access violation in WRITE_BARRIER_BIT_REGIONS64 reading the region-to-generation map (uncommitted page) — .NET 8.0.22, Workstation GC + regions, x64
Description
Generated by Claude Opus 5 after a few hours of debugging the memory dump:
A long-running Windows service fail-fasted with no managed exception. Analysis of a full WER dump shows the process died from an access violation inside the JIT write barrier (WRITE_BARRIER_BIT_REGIONS64) while reading the region-to-generation map. The byte being read lies on a page that is MEM_RESERVE — never committed.
Because an AV inside a write barrier is not recoverable, ProcessCLRException routed it to EEPolicy::HandleFatalError, so the process was fail-fasted and no managed exception object was ever produced.
@kkokosa This looks like the regions/x64 counterpart of #123490 (fixed by #131310 for Workstation-segments/x86): a GC bookkeeping-table grow leaving a sub-table's pages uncommitted, with a later access faulting on reserved memory.
!analyze -v is misleading — please don't triage from it
The exception record in this dump has NumberParameters: 0, so ExceptionInformation[0]/[1] are absent, and .ecxr yields the FailFast context (ntdll!NtRaiseException) rather than the original fault. !analyze -v consequently produces a fabricated bucket:
None of that is real. The same applies to the System.AccessViolationException on the heap: its _target (0x2D8B6F889B3) and _accessType (1) do not agree with the actual faulting instruction, which is consistent with SetExceptionAVParameters having been handed a record with no AV parameters. All numbers below come from the FaultingExceptionFrame's saved CONTEXT instead.
Faulting context
Recovered via .cxr on FaultingExceptionFrame.m_ctx:
The fault is pinned to 63a90024: rcx is already shifted (0xa63cb) while rdx is still unshifted (0x298f2ff97b0, pre-63a9004f), and 63a90024 is the only memory access in that window.
g_region_to_generation_table = TODO: run dp coreclr!g_region_to_generation_table L1
TODO
Since g_region_to_generation_table and g_card_table are both stomped by StompWriteBarrierResize(), and the card-table immediate is current, a partial/torn stomp seems unlikely — pointing at the table's memory rather than the barrier's constant.
Unskewing to the table's real base:
g_lowest_address >> 22 = 0x298a3400000 >> 22 = 0xa628d
real table base = 0x2d8b6f889b3 + 0xa628d = 000002d8`b702ec40
index for destination = 0xa63cb - 0xa628d = 0x13e
fault address = 0x2d8b702ec40 + 0x13e = 000002d8`b702ed7e (matches)
The faulting byte is only 0x13e bytes into the table — the table's own first page is not mapped.
Memory state of the region-to-generation table
0:024> !address 000002d8`b6f889b3
Base Address: 000002d8`b30c8000
End Address: 000002d8`b702e000
Region Size: 00000000`03f66000 ( 63.398 MB)
State: 00002000 MEM_RESERVE
Protect: <info not present at the target>
Type: 00020000 MEM_PRIVATE
Allocation Base: 000002d8`a3010000
Allocation Protect: 00000004 PAGE_READWRITE
The GC bookkeeping allocation is based at 000002d8a3010000, with committed pages ending at 000002d8b302e000/b30c8000 and ~63 MB reserved beyond. The region-to-generation table base (0x2d8b702ec40) sits just past the end of that reserved chunk, and the byte at index 0x13e is unmapped.
For contrast, the card table is healthy — a live object's card resolves inside committed space:
Nothing unusual on the application side — a plain Regex.Match(string) against a process-wide static readonly Regex, on a thread-pool worker doing a large configuration rebuild. No unsafe code, no P/Invoke, no GC.TryStartNoGCRegion on this path. Stack usage was ~22 KB of a 1.5 MB reservation, so no stack overflow.
Match..ctor had multiple native code versions (dynamic PGO active), if relevant:
Two anomalies I could not resolve — flagging rather than explaining
1. The destination region is not owned by any generation.
rcx pre-shift was the barrier's destination; rcx >> 22 = 0xa63cb inverts to region base 0x298f2c00000. That region is MEM_COMMIT/PAGE_READWRITE inside the GC's reservation (Allocation Base 00000298a3010000), but it is absent from !eeheap -gc and rejected by !GCWhere:
0:024> !GCWhere 00000298`f2ff9740
Address 298f2ff9740 not found in the managed heap.
0:024> !verifyobj 00000298`f2ff9740
0x00 ObjectNotOnTheHeap Tried to validate 298f2ff9740 but its address was not on any segment.
Neighbouring 4 MB regions at 0x298f2000000, 0x298f2800000, 0x298f3000000 are all present in !eeheap -gc; 0x298f2400000 and 0x298f2c00000 are not. Note 0x298f2c00000is inside [g_lowest_address, g_highest_address], so it should be covered by the region map.
Despite that, the object at 0x298f2ff9740 is a well-formed Match:
0:024> dq 00000298`f2ff9740 L10
+0x00 00007ff9`64282e18 ; Match MethodTable (confirmed by !ip2md)
+0x08 00000299`0f03bc38 ; Text — the input string
+0x18 00000299`0f03bdb0 ; _caps
+0x40 00000298`a546a238 ; _regex — the actual Regex instance
+0x48 00000299`0f03bdf0 ; _matches
+0x50 00000299`0f03bdd0 ; _matchcount
All of those references are valid and land in the live gen0 region (02990f000028–02990f3ad9f8).
!verifyheap reported 12,557,451 objects verified, 0 errors — but it would not have walked this object, since its region isn't enumerated.
2. rcx does not correspond to the array element address.
The resume IP follows the CORINFO_HELP_ARRADDR_ST call at 654f2e03, whose array operand was _matches = 0x2990f03bdf0:
So the shifted rcx matches a destination inside the Match's own region rather than the gen0 array element. I cannot reconcile that with the resume IP and am reporting it as observed.
#131310 (fixing #123490) describes, for Workstation-segments on x86:
A compacting foreground GC creates expansion segment S and marks its mark-array slice committed on the current array → before S is linked to generation lists, a card-table grow fires and allocates a new, larger mark array → commit_new_mark_array walks only the generation segment lists, missing the unlinked S → S's pages are never committed on the new array while its heap_segment_flags_ma_committed flag remains set → a later BGC trusts the stale flag and crashes writing to uncommitted memory.
The failure mode here is the same shape — a bookkeeping-table grow leaving a sub-table's pages uncommitted, with the fault surfacing later on reserved memory — but on a different path: x64, regions (USE_REGIONS), the region-to-generation map, consumed by the JIT write barrier rather than the mark array consumed by BGC marking. Filing separately in case the regions path needs its own fix; happy to have it closed as a duplicate if #131310's root cause covers it.
Repro
None. Single occurrence after 15.2 days of continuous operation in a production service. Not reproducible on demand. The workload that was running is a large configuration rebuild that allocates heavily (~1.45 GB live heap), so it crosses bookkeeping-table growth boundaries repeatedly.
What would help
Confirmation of whether the barrier's region-to-generation immediate is expected to match g_region_to_generation_table at all times, and whether the region map is expected to be fully committed for [g_lowest_address, g_highest_address].
Whether a region can legitimately be inside [g_lowest_address, g_highest_address], hold live objects, and be absent from !eeheap -gc/!GCWhere — or whether that is itself evidence of the bug.
Description
Access violation in WRITE_BARRIER_BIT_REGIONS64 reading the region-to-generation map (uncommitted page) — .NET 8.0.22, Workstation GC + regions, x64
Description
Generated by Claude Opus 5 after a few hours of debugging the memory dump:
A long-running Windows service fail-fasted with no managed exception. Analysis of a full WER dump shows the process died from an access violation inside the JIT write barrier (
WRITE_BARRIER_BIT_REGIONS64) while reading the region-to-generation map. The byte being read lies on a page that isMEM_RESERVE— never committed.Because an AV inside a write barrier is not recoverable,
ProcessCLRExceptionrouted it toEEPolicy::HandleFatalError, so the process was fail-fasted and no managed exception object was ever produced.@kkokosa This looks like the regions/x64 counterpart of #123490 (fixed by #131310 for Workstation-segments/x86): a GC bookkeeping-table grow leaving a sub-table's pages uncommitted, with a later access faulting on reserved memory.
Configuration
coreclr.dll8.0.2225.52707,C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.22ge_release), x64Number of GC Heaps: 1; 4 MB regions)WRITE_BARRIER_BIT_REGIONS64(bit-marking tail, no software write watch)ServiceBase.Run), self-contained-ish deployment on shared frameworkMiniDumpWithFullMemory+MiniDumpWithFullMemoryInfo)Memory state at the crash — no pressure of any kind:
!analyze -vis misleading — please don't triage from itThe exception record in this dump has
NumberParameters: 0, soExceptionInformation[0]/[1]are absent, and.ecxryields the FailFast context (ntdll!NtRaiseException) rather than the original fault.!analyze -vconsequently produces a fabricated bucket:None of that is real. The same applies to the
System.AccessViolationExceptionon the heap: its_target(0x2D8B6F889B3) and_accessType(1) do not agree with the actual faulting instruction, which is consistent withSetExceptionAVParametershaving been handed a record with no AV parameters. All numbers below come from theFaultingExceptionFrame's savedCONTEXTinstead.Faulting context
Recovered via
.cxronFaultingExceptionFrame.m_ctx:ripis the resume IP written byAdjustContextForJITHelpers, not the faulting instruction. The instruction there is a post-call spill reload:The barrier and the faulting instruction
The fault is pinned to
63a90024:rcxis already shifted (0xa63cb) whilerdxis still unshifted (0x298f2ff97b0, pre-63a9004f), and63a90024is the only memory access in that window.Globals vs. patched immediates
63a9002elower bound298A3400000hg_lowest_address=00000298a340000063a9003eupper bound2D8A3000000hg_highest_address=000002d8a300000063a9005ecard table2D84FEC9840hg_card_table=000002d84fec984063a90016region-to-gen2D8B6F889B3hg_region_to_generation_table= TODO: rundp coreclr!g_region_to_generation_table L1Since
g_region_to_generation_tableandg_card_tableare both stomped byStompWriteBarrierResize(), and the card-table immediate is current, a partial/torn stomp seems unlikely — pointing at the table's memory rather than the barrier's constant.Unskewing to the table's real base:
The faulting byte is only 0x13e bytes into the table — the table's own first page is not mapped.
Memory state of the region-to-generation table
The GC bookkeeping allocation is based at
000002d8a3010000, with committed pages ending at000002d8b302e000/b30c8000and ~63 MB reserved beyond. The region-to-generation table base (0x2d8b702ec40) sits just past the end of that reserved chunk, and the byte at index0x13eis unmapped.For contrast, the card table is healthy — a live object's card resolves inside committed space:
Escalation path
Managed stack (thread 24, TID 0x1db8 — ThreadPool worker)
Nothing unusual on the application side — a plain
Regex.Match(string)against a process-widestatic readonly Regex, on a thread-pool worker doing a large configuration rebuild. No unsafe code, no P/Invoke, noGC.TryStartNoGCRegionon this path. Stack usage was ~22 KB of a 1.5 MB reservation, so no stack overflow.Match..ctorhad multiple native code versions (dynamic PGO active), if relevant:Two anomalies I could not resolve — flagging rather than explaining
1. The destination region is not owned by any generation.
rcxpre-shift was the barrier's destination;rcx >> 22 = 0xa63cbinverts to region base0x298f2c00000. That region isMEM_COMMIT/PAGE_READWRITEinside the GC's reservation (Allocation Base 00000298a3010000), but it is absent from!eeheap -gcand rejected by!GCWhere:Neighbouring 4 MB regions at
0x298f2000000,0x298f2800000,0x298f3000000are all present in!eeheap -gc;0x298f2400000and0x298f2c00000are not. Note0x298f2c00000is inside[g_lowest_address, g_highest_address], so it should be covered by the region map.Despite that, the object at
0x298f2ff9740is a well-formedMatch:All of those references are valid and land in the live gen0 region (
02990f000028–02990f3ad9f8).!verifyheapreported 12,557,451 objects verified, 0 errors — but it would not have walked this object, since its region isn't enumerated.2.
rcxdoes not correspond to the array element address.The resume IP follows the
CORINFO_HELP_ARRADDR_STcall at654f2e03, whose array operand was_matches = 0x2990f03bdf0:So the shifted
rcxmatches a destination inside theMatch's own region rather than the gen0 array element. I cannot reconcile that with the resume IP and am reporting it as observed.Relationship to #123490 / #131310
#131310 (fixing #123490) describes, for Workstation-segments on x86:
The failure mode here is the same shape — a bookkeeping-table grow leaving a sub-table's pages uncommitted, with the fault surfacing later on reserved memory — but on a different path: x64, regions (
USE_REGIONS), the region-to-generation map, consumed by the JIT write barrier rather than the mark array consumed by BGC marking. Filing separately in case the regions path needs its own fix; happy to have it closed as a duplicate if #131310's root cause covers it.Repro
None. Single occurrence after 15.2 days of continuous operation in a production service. Not reproducible on demand. The workload that was running is a large configuration rebuild that allocates heavily (~1.45 GB live heap), so it crosses bookkeeping-table growth boundaries repeatedly.
What would help
g_region_to_generation_tableat all times, and whether the region map is expected to be fully committed for[g_lowest_address, g_highest_address].[g_lowest_address, g_highest_address], hold live objects, and be absent from!eeheap -gc/!GCWhere— or whether that is itself evidence of the bug.USE_REGIONSanalogue, and if a fix would be backported to 8.0.Reproduction Steps
No repro
Expected behavior
App should not crash
Actual behavior
App crashes
Regression?
No response
Known Workarounds
None
Configuration
No response
Other information
No response