cmd/internal/obj/x86: invalid encoding of VPERMQ with negative imm arg

[quasilyte]

Since CL100475, it's possible to use unsigned arguments for VPERMQ.

 var yvex_vpermpd = []ytab{
+	{Zvex_i_rm_r, 2, argList{Yu8, Yym, Yyr}},
+	// Allow int8 for backwards compatibility with negative values
+	// like $-1.
 	{Zvex_i_rm_r, 2, argList{Yi8, Yym, Yyr}},
 }

New ytab entry requires additional optab bytes (Optab.op) or, if both ytabs use the same encoding (like in this case), zoffset should be 0 (it's 2 in diff above).

The VPERMQ ptab entry was not updates, leaving second ytab with "all zeros", hence invalid encoding is produced.
Tests that check for output also use invalid encoding:
VPERMQ $-40, Y8, Y8 // c4407800c0d8

./xed -64 -d 'c4407800c0d8'
C4407800C0D8
ERROR: GENERAL_ERROR Could not decode at offset: 0x0 PC: 0x0: [C4407800C0D8000000000000000000]

For readers convenience, here is external assembler output for negative const:

vpermq ymm8, ymm8, -40
=>
c4 43 fd 00 c0 d8       vpermq ymm8,ymm8,0xd8 

I'll send a fix today.

Found this issue during comparison of existing ytabs and automatically generated tables.
Recently mailed AVX512-enabled assembler does not have this issue.

[gopherbot]

Change https://golang.org/cl/113615 mentions this issue: cmd/internal/obj/x86: fix VPERMQ and VPERMPD ytab