JIT: invalid `vpextrw` encoding on AVX-512 HW

[AndyAyersMS]

Bug found with LLM-Fuzz.

The JIT is producing an invalid vpextrw encoding on AVX-512 HW

Fatal error.
System.ExecutionEngineException: Illegal instruction: Attempted to execute an instruction code not defined by the processor.
   at VectorMathTests.Program.Main()

JIT disassembly

E8838A715F           call     CORINFO_HELP_NEWSFAST
C4E34D18F701         vinsertf128 ymm6, ymm6, xmm7, 1
C4E37D15700800       vpextrw  word  ptr [rax+0x08], ymm6, 0
488BC8               mov      rcx, rax
C4E37D19F701         vextractf128 xmm7, ymm6, 1
FF157F337B00         call     [VectorMathTests.Program:<Main>g__ProcessValue|2_0(System.Object):int]

Debugger disassembly

e8838a705f           call    coreclr!JIT_NewS_MP_FastPortable (7ff8d52bc7c0)
c4e34d18f701         vinsertf128 ymm6, ymm6, xmm7, 1
c4                   ???     
e37d                 jrcxz   00007FF875BB3DC3
1570080048           adc     eax, 48000870h
8bc8                 mov     ecx, eax
c4e37d19f701         vextractf128 xmm7, ymm6, 1

Run the attached with `DOTNET_TieredCompilation=0

// Mutation 6
using System;
using System.Collections.Generic;
using System.Numerics;
using System.Globalization;

namespace VectorMathTests
{
    public class Program
    {
        public const int DefaultSeed = 20010415;
        public static int Seed = Environment.GetEnvironmentVariable("CORECLR_SEED") switch
        {
            string s when s.Equals("random", StringComparison.OrdinalIgnoreCase) => 2048,
            string s when int.TryParse(s, out int seedVal) => seedVal,
            _ => DefaultSeed
        };

        public static void Main()
        {
            int checksum = 44444;
            int exitCode = 100;

            // Use a local function with pattern matching.
            int ProcessValue(object val)
            {
                return val switch
                {
                    int i => i,
                    short s => s,
                    _ => 0
                };
            }

            var a = new Vector<short>(25);
            a = Vector.SquareRoot(a);
            checksum = unchecked(checksum + ProcessValue(a[0]));
            if (a[0] != 5)
            {
                exitCode = 0;
            }
            
            var b = Vector<int>.One;
            b = Vector.SquareRoot(b);
            checksum = unchecked(checksum + ProcessValue(b[3]));
            if (b[3] != 1)
            {
                exitCode = 0;
            }
            
            var c = new Vector<long>(1231111222 * (long)1231111222);
            c = Vector.SquareRoot(c);
            checksum = unchecked(checksum + ProcessValue((int)c[1]));
            if (c[1] != 1231111222)
            {
                exitCode = 0;
            }
            
            var d = new Vector<double>(100.0);
            d = Vector.SquareRoot(d);
            checksum = unchecked(checksum + ProcessValue((int)d[0]));
            if (((int)d[0]) != 10)
            {
                exitCode = 0;
            }
            
            var e = new Vector<float>(64);
            e = Vector.SquareRoot(e);
            checksum = unchecked(checksum + ProcessValue((int)e[3]));
            if (((int)e[3]) != 8)
            {
                exitCode = 0;
            }
            
            var f = new Vector<ushort>(36);
            f = Vector.SquareRoot(f);
            checksum = unchecked(checksum + ProcessValue(f[7]));
            if (f[7] != 6)
            {
                exitCode = 0;
            }
            
            var g = new Vector<ulong>(16);
            g = Vector.SquareRoot(g);
            checksum = unchecked(checksum + ProcessValue((int)g[1]));
            if (g[1] != 4)
            {
                exitCode = 0;
            }

            // Shared array for loops.
            int[] shared = new int[] { 7, 14, 21, 28, 35 };

            // Clonable loop: for-loop with invariant bound.
            if (exitCode == 100)
            {
                for (int i = 0, len = shared.Length; i < len; i++)
                {
                    checksum = unchecked(checksum + shared[i]);
                }
                Console.WriteLine("Checksum after first shared loop: " + checksum);
            }

            // Clonable loop: do-while loop based on a condition.
            if (checksum % 7 == 0)
            {
                int index = 0;
                do
                {
                    checksum = unchecked(checksum + shared[index] / 7);
                    index++;
                } while (index < shared.Length);
                Console.WriteLine("Checksum after do-while loop: " + checksum);
            }

            Console.WriteLine("Final Checksum: " + checksum);
            Environment.Exit(exitCode);
        }
    }
}

[AndyAyersMS]

FYI @dotnet/jit-contrib

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[BruceForstall]

@dotnet/intel PTAL

[Ruihan-Yin]

Just for reference, PEXTRW does not have the form with ymm operand (or I might be missing some other instructions with similar function?)

The error encoding comes from VEX.L is set, but the instruction itself does not support setting this bit.

decoder can decode with unsetting VEX.L, but it changed the intended semantic

[BruceForstall]

Slightly reduced test case:

using System;
using System.Numerics;

namespace VectorMathTests
{
    public class Program
    {
        public static void Main()
        {
            int checksum = 44444;
            int exitCode = 100;

            int ProcessValue(object val)
            {
                return val switch
                {
                    int i => i,
                    short s => s,
                    _ => 0
                };
            }

            var a = new Vector<short>(25);
            a = Vector.SquareRoot(a);
            checksum = unchecked(checksum + ProcessValue(a[0]));
            if (a[0] != 5)
            {
                exitCode = 0;
            }
            Console.WriteLine("Final Checksum: " + checksum);
            Environment.Exit(exitCode);
        }
    }
}

[tannergooding]

decoder can decode with unsetting VEX.L, but it changed the intended semantic

It's extracting index 0, so the semantic is the same regardless.

There's likely just an issue here with the wrong EA_ATTR being passed down due to containment or similar. It would be helpful to see the tree passed into codegen.