[X86][MC][AsmParser] Reject H-byte regs with VEX/EVEX-encoded 8-bit RR (NDD)

[woruyu]

Summary

This PR resolves #158585.

[llvmbot]

@llvm/pr-subscribers-backend-x86

Author: woruyu (woruyu)

Changes

Summary

This PR resolves #158585.

---

Full diff: https://github.com/llvm/llvm-project/pull/160039.diff

2 Files Affected:

• (modified) llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp (+18)
• (added) llvm/test/MC/X86/encoder-fail-VEX-EVEX.s (+16)

diff --git a/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp b/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
index ce5e92135f706..d1ebe6bdb2dfd 100644
--- a/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
+++ b/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
@@ -4044,6 +4044,24 @@ bool X86AsmParser::validateInstruction(MCInst &Inst, const OperandVector &Ops) {
     }
   }
 
+  unsigned Enc = TSFlags & X86II::EncodingMask;
+  if (Enc == X86II::VEX || Enc == X86II::EVEX || Enc == X86II::XOP) {
+    unsigned NumOps = Inst.getNumOperands();
+    for (unsigned i = 0; i != NumOps; ++i) {
+      const MCOperand &MO = Inst.getOperand(i);
+      if (!MO.isReg())
+        continue;
+      MCRegister Reg = MO.getReg();
+      if (Reg == X86::AH || Reg == X86::BH || Reg == X86::CH ||
+          Reg == X86::DH) {
+        StringRef RegName = X86IntelInstPrinter::getRegisterName(Reg);
+        return Error(Ops[0]->getStartLoc(),
+                     "can't encode '" + RegName +
+                         "' in a VEX/EVEX-prefixed instruction");
+      }
+    }
+  }
+
   if ((Opcode == X86::PREFETCHIT0 || Opcode == X86::PREFETCHIT1)) {
     const MCOperand &MO = Inst.getOperand(X86::AddrBaseReg);
     if (!MO.isReg() || MO.getReg() != X86::RIP)
diff --git a/llvm/test/MC/X86/encoder-fail-VEX-EVEX.s b/llvm/test/MC/X86/encoder-fail-VEX-EVEX.s
new file mode 100644
index 0000000000000..90512741c9c6c
--- /dev/null
+++ b/llvm/test/MC/X86/encoder-fail-VEX-EVEX.s
@@ -0,0 +1,16 @@
+// RUN: not llvm-mc -triple x86_64-unknown-unknown -x86-asm-syntax=intel --show-encoding %s 2>&1 | FileCheck %s
+
+// CHECK: error: can't encode 'ah' in a VEX/EVEX-prefixed instruction
+add ah, ah, ah
+
+// CHECK: error: can't encode 'ah' in a VEX/EVEX-prefixed instruction
+and ah, byte ptr [-13426159], ah
+
+// CHECK: error: can't encode 'ah' in a VEX/EVEX-prefixed instruction
+ccmpa {dfv=of,cf} byte ptr [r8 + 4*rax + 291], ah
+
+// CHECK: error: can't encode 'ah' in a VEX/EVEX-prefixed instruction
+ccmpae {dfv=of,cf} byte ptr [r8 + 4*rax + 291], ah
+
+// CHECK: error: can't encode 'ah' in a VEX/EVEX-prefixed instruction
+sar ah, byte ptr [-13426159]
\ No newline at end of file

[phoebewang]

LGTM.

[KanRobert]

This seems an incomplete fix. In addition to REX_W, we may use REX prefix when other bit is set, e.g. REX_R/X/B.

[phoebewang]

https://godbolt.org/z/98xdvEbn3

[woruyu]

I can't understand this. Because I think REX_R/X/B means you use r8..r15 in opreand, so in current codes for add ah, byte ptr [r8], you will get UsesRex = true,HReg is noZero, so you will get error msg in the second if branch.

[phoebewang]

Yes, I mean we don't need extra change here.

[KanRobert]

Okay. I missed the line 4041

[phoebewang]

The failures look unrelated. Please merge main and try again.

[phoebewang]

Why don't we || UsesRex here? Or move ForcedOpcodePrefix == OpcodePrefix_REX to below condition?

[woruyu]

Done! combine all check.

[KanRobert]

LGTM in general