You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Avx.Compare/Avx.CompareScalar (all six overloads: Vector128<float>, Vector256<float>, Vector128<double>, Vector256<double> for Compare, and Vector128<float>, Vector128<double> for CompareScalar) miscompile when the FloatComparisonMode mode argument is not a compile-time constant, on hardware where the JIT can use EVEX encoding (e.g. AVX512F available). Under DOTNET_TieredCompilation=0 (FullOpts), this reliably produces an AccessViolationException or garbage results; it appears to work fine under normal tiered compilation.
This is a pre-existing bug on main, not a regression - reproduces against a clean top-of-tree build.
Root cause
For a non-constant mode, the JIT's non-constant-imm handling is supposed to keep the call expanded as an intrinsic (producing a 32-case range-checked jump table over FloatComparisonMode, one vcmpXXps/vcmppd/vcmpss/vcmpsd case per value) rather than converting it into a real call to the (intentionally self-recursive) managed fallback body.
On hardware where EVEX encoding is available, impSpecialIntrinsic (src/coreclr/jit/hwintrinsicxarch.cpp) promotes NI_AVX_Compare/NI_AVX_CompareScalar to the mask-returning NI_AVX512_CompareMask, and - before returning - wraps the mask-typed result back into a vector via NI_AVX512_ConvertMaskToVector (a single-operand wrapper node), since the managed API returns a vector, not a mask.
Back in the shared impHWIntrinsic (src/coreclr/jit/hwintrinsic.cpp), when the mode is non-constant, SetMethodHandle is called on whatever node impSpecialIntrinsic returned - which, on EVEX-capable hardware, is that single-operand ConvertMaskToVector wrapper, not the 3-operand mask node it wraps. SetMethodHandle sets GTF_HW_USER_CALL (marking the node as a candidate to become a real call later) using the method handle for Avx.Compare(left, right, mode) - a 3-argument method - but attaches it to a node that only has one operand.
At Rationalize time, Rationalizer::RewriteHWIntrinsicAsUserCall (src/coreclr/jit/rationalize.cpp) is invoked for this (now mismatched) node. Its intrinsicId is NI_AVX512_ConvertMaskToVector, which doesn't match the dedicated NI_AVX_Compare/CompareScalar/CompareMask switch case, so it falls into the default: path. That path's isImmOp check fails for ConvertMaskToVector (it isn't an imm intrinsic), so it unconditionally builds a call to Avx.Compare via RewriteNodeAsCall, using the wrapper's one operand (the already-correctly-computed jump-table result) as left. right and mode are never populated, leaving those registers/stack slots with whatever they held before - hence the AccessViolationException/garbage results.
Concretely, in FullOpts disassembly of a caller, you can see: a correct 32-case jump table computing vcmpXXps/etc. into xmm0, followed by a call to Avx:Compare that passes only a return buffer and the jump-table result as left - right (r8) and mode (r9) are never set up.
Non-EVEX hardware isn't affected: there, impSpecialIntrinsic returns the plain 3-operand NI_AVX_Compare node directly (no wrapper), so SetMethodHandle attaches correctly and the well known "convert to a genuine call to the self-recursive fallback, which itself re-imports as a must-expand intrinsic and succeeds" mechanism (the same one used by other non-constant-imm intrinsics like Shuffle/Blend) works as intended.
Reproduction
usingSystem.Runtime.CompilerServices;usingSystem.Runtime.Intrinsics;usingSystem.Runtime.Intrinsics.X86;[MethodImpl(MethodImplOptions.NoInlining|MethodImplOptions.AggressiveOptimization)]staticVector128<float>Compare(Vector128<float>left,Vector128<float>right,FloatComparisonModemode)=>Avx.Compare(left,right,mode);varleft=Vector128.Create(1f,2f,3f,4f);varright=Vector128.Create(5f,2f,7f,4f);for(inti=0;i<=(int)FloatComparisonMode.UnorderedTrueSignaling;i++){// mode is a genuine runtime value here, not a compile-time constantConsole.WriteLine(Compare(left,right,(FloatComparisonMode)i));}
On EVEX-capable hardware (e.g. AVX512F available) with DOTNET_TieredCompilation=0, this throws System.AccessViolationException (or returns garbage) instead of printing the 32 comparison results.
Configuration
Reproduces on main (top-of-tree), Windows x64, on hardware with AVX512F support.
Requires DOTNET_TieredCompilation=0 (FullOpts) to reproduce reliably, or MethodImplOptions.AggressiveOptimization on the containing method.
Note
This issue was investigated and drafted with the help of GitHub Copilot.
Description
Avx.Compare/Avx.CompareScalar(all six overloads:Vector128<float>,Vector256<float>,Vector128<double>,Vector256<double>forCompare, andVector128<float>,Vector128<double>forCompareScalar) miscompile when theFloatComparisonMode modeargument is not a compile-time constant, on hardware where the JIT can use EVEX encoding (e.g. AVX512F available). UnderDOTNET_TieredCompilation=0(FullOpts), this reliably produces anAccessViolationExceptionor garbage results; it appears to work fine under normal tiered compilation.This is a pre-existing bug on
main, not a regression - reproduces against a clean top-of-tree build.Root cause
For a non-constant
mode, the JIT's non-constant-imm handling is supposed to keep the call expanded as an intrinsic (producing a 32-case range-checked jump table overFloatComparisonMode, onevcmpXXps/vcmppd/vcmpss/vcmpsdcase per value) rather than converting it into a real call to the (intentionally self-recursive) managed fallback body.On hardware where EVEX encoding is available,
impSpecialIntrinsic(src/coreclr/jit/hwintrinsicxarch.cpp) promotesNI_AVX_Compare/NI_AVX_CompareScalarto the mask-returningNI_AVX512_CompareMask, and - before returning - wraps the mask-typed result back into a vector viaNI_AVX512_ConvertMaskToVector(a single-operand wrapper node), since the managed API returns a vector, not a mask.Back in the shared
impHWIntrinsic(src/coreclr/jit/hwintrinsic.cpp), when the mode is non-constant,SetMethodHandleis called on whatever nodeimpSpecialIntrinsicreturned - which, on EVEX-capable hardware, is that single-operandConvertMaskToVectorwrapper, not the 3-operand mask node it wraps.SetMethodHandlesetsGTF_HW_USER_CALL(marking the node as a candidate to become a real call later) using the method handle forAvx.Compare(left, right, mode)- a 3-argument method - but attaches it to a node that only has one operand.At Rationalize time,
Rationalizer::RewriteHWIntrinsicAsUserCall(src/coreclr/jit/rationalize.cpp) is invoked for this (now mismatched) node. ItsintrinsicIdisNI_AVX512_ConvertMaskToVector, which doesn't match the dedicatedNI_AVX_Compare/CompareScalar/CompareMaskswitch case, so it falls into thedefault:path. That path'sisImmOpcheck fails forConvertMaskToVector(it isn't an imm intrinsic), so it unconditionally builds a call toAvx.CompareviaRewriteNodeAsCall, using the wrapper's one operand (the already-correctly-computed jump-table result) asleft.rightandmodeare never populated, leaving those registers/stack slots with whatever they held before - hence theAccessViolationException/garbage results.Concretely, in FullOpts disassembly of a caller, you can see: a correct 32-case jump table computing
vcmpXXps/etc. intoxmm0, followed by acalltoAvx:Comparethat passes only a return buffer and the jump-table result asleft-right(r8) andmode(r9) are never set up.Non-EVEX hardware isn't affected: there,
impSpecialIntrinsicreturns the plain 3-operandNI_AVX_Comparenode directly (no wrapper), soSetMethodHandleattaches correctly and the well known "convert to a genuine call to the self-recursive fallback, which itself re-imports as a must-expand intrinsic and succeeds" mechanism (the same one used by other non-constant-imm intrinsics likeShuffle/Blend) works as intended.Reproduction
On EVEX-capable hardware (e.g. AVX512F available) with
DOTNET_TieredCompilation=0, this throwsSystem.AccessViolationException(or returns garbage) instead of printing the 32 comparison results.Configuration
main(top-of-tree), Windows x64, on hardware with AVX512F support.DOTNET_TieredCompilation=0(FullOpts) to reproduce reliably, orMethodImplOptions.AggressiveOptimizationon the containing method.Note
This issue was investigated and drafted with the help of GitHub Copilot.