JIT: ref-struct return via forwarded retbuf loses writes across compacting GC

[HMBSbige]

Description

On .NET 10 x64, when the JIT forwards a heap byref as the hidden return-buffer argument of a method that returns a byref-like (ref struct) value, a compacting GC inside that callee causes subsequent stores through the forwarded pointer to land at the pre-compaction (stale) address. The caller reads the relocated object afterwards and sees zeros — the write is silently lost.

Regression from .NET 9.

Reproduction Steps

using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

byte[] buf = Alloc();
Unsafe.WriteUnaligned(ref buf[0], Make());
Console.WriteLine(Unsafe.ReadUnaligned<ulong>(ref buf[0]) == 0xDEADBEEFCAFEBABEUL ? "OK" : "LOST");

[MethodImpl(MethodImplOptions.NoInlining)]
static byte[] Alloc() => new byte[16];

[MethodImpl(MethodImplOptions.NoInlining)]
static Payload Make()
{
    GC.Collect(2, GCCollectionMode.Forced, true, true);
    Payload p = default;
    p.A = 0xDEADBEEFCAFEBABEUL;
    p.B = 0x0123456789ABCDEFUL;
    return p;
}

[StructLayout(LayoutKind.Sequential, Size = 16)]
ref struct Payload { public ulong A, B; }

dotnet run -c Release

Full pass/fail matrix (7 permutations, including controls for ordinary struct, stackalloc, pinned heap, and the intermediate-local workaround): https://github.com/HMBSbige/ByrefLikeWriteUnalignedRepro

Expected behavior

Prints OK.

Actual behavior

Prints LOST.

Regression?

Yes — same source passes on the .NET 9 runtime (9.0.15), fails on the .NET 10 runtime (10.0.6).

Known Workarounds

Store the byref-like return into a local before writing it back:

var p = Make();
Unsafe.WriteUnaligned(ref buf[0], p); // OK

Configuration

• .NET SDK 10.0.202, Release build
• x64 Windows 11 (26200)
• Default (Workstation) GC
• Reproduces under FullOpts

Analysis

Failing case disassembly (FullOpts):

mov  rbx, rdx                    ; rbx = byte[] object ref (callee-saved)
lea  rcx, bword ptr [rbx+0x10]   ; rcx = &array[0], passed as Make's retbuf
call Program:...Make():Payload   ; compacting GC inside
mov  rax, qword ptr [rbx+0x10]   ; rbx was updated by GC; reads relocated array -> 0

Same caller pattern but with an ordinary struct return uses a stack slot as retbuf (no forwarding) and the byref survives in a callee-saved register:

lea  rbp, bword ptr [rbx+0x10]   ; rbp = heap byref, tracked as GC_BYREF
lea  rcx, [rsp+0x28]             ; retbuf = stack slot
call Program:...MakeStruct():PayloadStruct
vmovups xmm0, xmmword ptr [rsp+0x28]
vmovups xmmword ptr [rbp], xmm0  ; rbp still valid after the call

So byref tracking in callee-saved registers works in general. The bug appears specific to the retbuf-forwarding path taken for byref-like returns; the forwarded retbuf does not appear to be reported as GC_BYREF in the callee's GC info.

Removing any one of the three ingredients turns the failure into a pass: the ref struct return, the compacting GC inside the callee, or the retbuf forwarding (introducing an intermediate local).

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

The code is invalid to attempt to write ref struct onto heap. JIT assumes ref struct is never on heap and doesn't the address to GC.

It fails on .NET 10 because JIT optimizes more aggressively and uses the write target as return buffer.

[HMBSbige]

So is this the expected behavior?

[huoyaoyuan]

The behavior is undefined because it breaks the contract of ref struct. Unsafe are meant to bypass limitations of the type system, but the underlying contracts still apply.

It's especially incorrect to abuse ref struct as data buffer. The types under https://github.com/HMBSbige/CryptoBase/tree/master/src/CryptoBase.Abstractions/Vectors should just be InlineArray16<Vector512<byte>> etc.

[EgorBo]

The change that broke this unsafe code is this #112060 (well, if I'm reading it correctly..)