Skip to content

JIT: Avx.Compare/Avx.CompareScalar miscompile (AccessViolationException) with a non-constant FloatComparisonMode on EVEX-capable hardware #131472

Description

@tannergooding

Description

Avx.Compare/Avx.CompareScalar (all six overloads: Vector128<float>, Vector256<float>, Vector128<double>, Vector256<double> for Compare, and Vector128<float>, Vector128<double> for CompareScalar) miscompile when the FloatComparisonMode mode argument is not a compile-time constant, on hardware where the JIT can use EVEX encoding (e.g. AVX512F available). Under DOTNET_TieredCompilation=0 (FullOpts), this reliably produces an AccessViolationException or garbage results; it appears to work fine under normal tiered compilation.

This is a pre-existing bug on main, not a regression - reproduces against a clean top-of-tree build.

Root cause

For a non-constant mode, the JIT's non-constant-imm handling is supposed to keep the call expanded as an intrinsic (producing a 32-case range-checked jump table over FloatComparisonMode, one vcmpXXps/vcmppd/vcmpss/vcmpsd case per value) rather than converting it into a real call to the (intentionally self-recursive) managed fallback body.

On hardware where EVEX encoding is available, impSpecialIntrinsic (src/coreclr/jit/hwintrinsicxarch.cpp) promotes NI_AVX_Compare/NI_AVX_CompareScalar to the mask-returning NI_AVX512_CompareMask, and - before returning - wraps the mask-typed result back into a vector via NI_AVX512_ConvertMaskToVector (a single-operand wrapper node), since the managed API returns a vector, not a mask.

Back in the shared impHWIntrinsic (src/coreclr/jit/hwintrinsic.cpp), when the mode is non-constant, SetMethodHandle is called on whatever node impSpecialIntrinsic returned - which, on EVEX-capable hardware, is that single-operand ConvertMaskToVector wrapper, not the 3-operand mask node it wraps. SetMethodHandle sets GTF_HW_USER_CALL (marking the node as a candidate to become a real call later) using the method handle for Avx.Compare(left, right, mode) - a 3-argument method - but attaches it to a node that only has one operand.

At Rationalize time, Rationalizer::RewriteHWIntrinsicAsUserCall (src/coreclr/jit/rationalize.cpp) is invoked for this (now mismatched) node. Its intrinsicId is NI_AVX512_ConvertMaskToVector, which doesn't match the dedicated NI_AVX_Compare/CompareScalar/CompareMask switch case, so it falls into the default: path. That path's isImmOp check fails for ConvertMaskToVector (it isn't an imm intrinsic), so it unconditionally builds a call to Avx.Compare via RewriteNodeAsCall, using the wrapper's one operand (the already-correctly-computed jump-table result) as left. right and mode are never populated, leaving those registers/stack slots with whatever they held before - hence the AccessViolationException/garbage results.

Concretely, in FullOpts disassembly of a caller, you can see: a correct 32-case jump table computing vcmpXXps/etc. into xmm0, followed by a call to Avx:Compare that passes only a return buffer and the jump-table result as left - right (r8) and mode (r9) are never set up.

Non-EVEX hardware isn't affected: there, impSpecialIntrinsic returns the plain 3-operand NI_AVX_Compare node directly (no wrapper), so SetMethodHandle attaches correctly and the well known "convert to a genuine call to the self-recursive fallback, which itself re-imports as a must-expand intrinsic and succeeds" mechanism (the same one used by other non-constant-imm intrinsics like Shuffle/Blend) works as intended.

Reproduction

using System.Runtime.CompilerServices;
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;

[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.AggressiveOptimization)]
static Vector128<float> Compare(Vector128<float> left, Vector128<float> right, FloatComparisonMode mode)
    => Avx.Compare(left, right, mode);

var left = Vector128.Create(1f, 2f, 3f, 4f);
var right = Vector128.Create(5f, 2f, 7f, 4f);

for (int i = 0; i <= (int)FloatComparisonMode.UnorderedTrueSignaling; i++)
{
    // mode is a genuine runtime value here, not a compile-time constant
    Console.WriteLine(Compare(left, right, (FloatComparisonMode)i));
}

On EVEX-capable hardware (e.g. AVX512F available) with DOTNET_TieredCompilation=0, this throws System.AccessViolationException (or returns garbage) instead of printing the 32 comparison results.

Configuration

  • Reproduces on main (top-of-tree), Windows x64, on hardware with AVX512F support.
  • Requires DOTNET_TieredCompilation=0 (FullOpts) to reproduce reliably, or MethodImplOptions.AggressiveOptimization on the containing method.

Note

This issue was investigated and drafted with the help of GitHub Copilot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIbuguntriagedNew issue has not been triaged by the area owner

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions