[arm64] Assertion failed 'opt != INS_OPTS_1D' shuffling Vector64<nint>/Vector64<nuint> with variable indices

[lewing]

Description

On arm64, a checked JIT asserts when compiling Vector64.Shuffle / Vector64.ShuffleNative for a 64-bit element type with variable indices:

Assertion failed 'opt != INS_OPTS_1D' in
'System.Runtime.Intrinsics.Vector64:ShuffleNative(System.Runtime.Intrinsics.Vector64`1[nuint],System.Runtime.Intrinsics.Vector64`1[nuint]):System.Runtime.Intrinsics.Vector64`1[nuint]'
during 'Generate code' (IL size 8; hash 0x6f28d991; FullOpts)

from src/coreclr/jit/emitarm64.cpp:5394.

Root cause

Compiler::gtNewSimdShuffleVariableNode scales variable shuffle indices into byte offsets (gentree.cpp):

// shift all indices to the left by tzcnt(size)
cnsNode = gtNewIconNode(BitOperations::TrailingZeroCount(static_cast<uint64_t>(elementSize)), TYP_INT);
op2     = gtNewSimdHWIntrinsicNode(type, op2, cnsNode, NI_AdvSimd_ShiftLeftLogical, simdBaseType, simdSize);

When simdSize == 8 and elementSize == 8, the vector holds a single 64-bit element, i.e. a 1D arrangement. The vector form of the arm64 shift-by-immediate has no valid 1D encoding (reserved) — that case must use the scalar form.

The JIT already knows this. gtNewSimdBinOpNode applies exactly the right guard for GT_LSH:

#elif defined(TARGET_ARM64)
    if ((simdSize == 8) && (genTypeSize(simdBaseType) == 8))
    {
        id = op2->IsCnsIntOrI() ? NI_AdvSimd_ShiftLeftLogicalScalar : NI_AdvSimd_ShiftLogicalScalar;
    }
    else
    {
        id = op2->IsCnsIntOrI() ? NI_AdvSimd_ShiftLeftLogical : NI_AdvSimd_ShiftLogical;
    }

The shuffle path constructs the HWINTRINSIC node directly and so never gets that guard. The adjacent code does special-case (simdSize == 16) && (simdBaseType == TYP_INT), but nothing handles the 8/8 case.

Conditions required to hit it

• arm64, checked JIT (the assert is checked-only).
• A 64-bit element type in a Vector64. nint / nuint are the only Vector64 shuffle overloads with one — there are no long / ulong / double overloads — and only on a 64-bit platform.
• Variable indices. Constant indices take a different path that never emits the shift.

Because elementCount == 1 for these types, the index fix-up is degenerate; it is emitted only because the generic path always emits it.

Impact

Most visible under NativeAOT: ILC compiles CoreLib whole-program and instantiates ShuffleNative<nuint> regardless of whether anything calls it, so any NativeAOT build using a checked arm64 JIT fails:

error MSB3073: The command "…/ilc @…/System.Private.CoreLib.ilc.rsp" exited with code 133.

This blocks src/tests/build.sh nativeaot checked on an arm64 host entirely.

CI exposure

Not caught today. windows-x64 Checked NativeAOT is the only checked NativeAOT leg and x64 does not go through emitarm64.cpp; the arm64 NativeAOT legs are all Release, where the assert is compiled out.

Repro

Reproduced on osx-arm64 against 2aadf336 and current main, on clean full builds (build.sh -s clr.aot+libs.native+libs.sfx -rc Checked -lc Release). A minimal managed repro, with the indices kept variable by a non-inlined call:

[MethodImpl(MethodImplOptions.NoInlining)]
static Vector64<nuint> ShuffleNoInline(Vector64<nuint> vector, Vector64<nuint> indices)
    => Vector64.Shuffle(vector, indices);

// on a checked arm64 JIT:
ShuffleNoInline(Vector64.Create((nuint)1), Vector64.Create((nuint)0));

Assert failure: Assertion failed 'opt != INS_OPTS_1D' in
'…Vector64Tests:ShuffleNoInline(Vector64`1[nuint],Vector64`1[nuint]):Vector64`1[nuint]'
during 'Generate code' (IL size 8; hash 0xf1ceddf2; Tier0)
exit code 134 — SIGABRT

Fix

Mirror the guard already used by gtNewSimdBinOpNode and select NI_AdvSimd_ShiftLeftLogicalScalar when (simdSize == 8) && (elementSize == 8).

Note

This issue was prepared with GitHub Copilot assistance and reviewed by the submitting developer.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.