[X86][vectorcall] Pass built types byval when xmm0~6 exhausted

[phoebewang]

This is how MSVC handles it. https://godbolt.org/z/fG386bjnf

[llvmbot]

@llvm/pr-subscribers-backend-x86
@llvm/pr-subscribers-clang

@llvm/pr-subscribers-clang-codegen

Author: Phoebe Wang (phoebewang)

Changes

This is how MSVC handles it. https://godbolt.org/z/fG386bjnf

---

Full diff: https://github.com/llvm/llvm-project/pull/91846.diff

2 Files Affected:

• (modified) clang/lib/CodeGen/Targets/X86.cpp (+1-1)
• (modified) clang/test/CodeGen/vectorcall.c (+16)

diff --git a/clang/lib/CodeGen/Targets/X86.cpp b/clang/lib/CodeGen/Targets/X86.cpp
index 717a27fc9c574..5c09339bd7137 100644
--- a/clang/lib/CodeGen/Targets/X86.cpp
+++ b/clang/lib/CodeGen/Targets/X86.cpp
@@ -792,7 +792,7 @@ ABIArgInfo X86_32ABIInfo::classifyArgumentType(QualType Ty, CCState &State,
         return ABIArgInfo::getDirect();
       return ABIArgInfo::getExpand();
     }
-    return getIndirectResult(Ty, /*ByVal=*/false, State);
+    return getIndirectResult(Ty, IsVectorCall && Ty->isBuiltinType(), State);
   }
 
   if (isAggregateTypeForABI(Ty)) {
diff --git a/clang/test/CodeGen/vectorcall.c b/clang/test/CodeGen/vectorcall.c
index cb53ecc70351d..97a532a9e3a25 100644
--- a/clang/test/CodeGen/vectorcall.c
+++ b/clang/test/CodeGen/vectorcall.c
@@ -140,4 +140,20 @@ void __vectorcall vectorcall_indirect_vec(
 // X86-SAME: ptr inreg noundef %0,
 // X86-SAME: i32 inreg noundef %edx,
 // X86-SAME: ptr noundef %1)
+
+void __vectorcall vectorcall_indirect_fp(
+    double xmm0, double xmm1, double xmm2, double xmm3, double xmm4,
+    v4f32 xmm5, v4f32 ecx, int edx, double mem) {
+}
+
+// X86: define dso_local x86_vectorcallcc void @"\01vectorcall_indirect_fp@@{{[0-9]+}}"
+// X86-SAME: (double inreg noundef %xmm0,
+// X86-SAME: double inreg noundef %xmm1,
+// X86-SAME: double inreg noundef %xmm2,
+// X86-SAME: double inreg noundef %xmm3,
+// X86-SAME: double inreg noundef %xmm4,
+// X86-SAME: <4 x float> inreg noundef %xmm5,
+// X86-SAME: ptr inreg noundef %0,
+// X86-SAME: i32 inreg noundef %edx,
+// X86-SAME: ptr noundef byval(double) align 4 %1)
 #endif

[efriedma-quic]

Maybe try to avoid byval overhead:

if (IsVectorCall && Ty->isBuiltinType())
  return ABIArgInfo::getDirect();
return getIndirectResult(Ty, /*ByVal=*/false, State);

[phoebewang]

Done.

[efriedma-quic]

LGTM