[SPIRV] Implement support for SPV_KHR_expect_assume #66217

pmatos · 2023-09-13T15:00:59Z

Adds new extension SPV_KHR_expect_assume, new capability
ExpectAssumeKHR as well as the new instructions:

OpExpectKHR
OpAssumeTrueKHR

These are lowered from respectively llvm.expect. and llvm.assume
intrinsics.

Previously https://reviews.llvm.org/D157696

pmatos · 2023-09-13T15:02:02Z

Still need to address https://reviews.llvm.org/D157696#4614823

llvmbot · 2023-09-13T15:02:10Z

@llvm/pr-subscribers-llvm-ir

Changes

Adds new extension SPV_KHR_expect_assume, new capability ExpectAssumeKHR as well as the new instructions: * OpExpectKHR * OpAssumeTrueKHR

These are lowered from respectively llvm.expect. and llvm.assume
intrinsics.

Previously https://reviews.llvm.org/D157696

Full diff: https://github.com/llvm/llvm-project/pull/66217.diff

10 Files Affected:

(modified) llvm/include/llvm/IR/IntrinsicsSPIRV.td (+5-1)
(modified) llvm/lib/Target/SPIRV/SPIRVBuiltins.td (+1)
(modified) llvm/lib/Target/SPIRV/SPIRVInstrInfo.td (+4)
(modified) llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp (+12)
(modified) llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp (+7)
(modified) llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp (+32)
(modified) llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp (+1)
(modified) llvm/lib/Target/SPIRV/SPIRVSymbolicOperands.td (+1)
(added) llvm/test/CodeGen/SPIRV/assume.ll (+15)
(added) llvm/test/CodeGen/SPIRV/expect.ll (+20)

diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 2b8602f430dff1c..3ef47e09dec2553 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -32,4 +32,8 @@ let TargetPrefix = "spv" in {
   def int_spv_unreachable : Intrinsic<[], []>;
   def int_spv_alloca : Intrinsic<[llvm_any_ty], []>;
   def int_spv_undef : Intrinsic<[llvm_i32_ty], []>;
-}
+
+  // Expect, Assume Intrinsics
+  def int_spv_assume : Intrinsic<[], [llvm_i1_ty]>;
+  def int_spv_expect : Intrinsic<[llvm_anyint_ty], [LLVMMatchType<0>, LLVMMatchType<0>]>;
+}
\ No newline at end of file
diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.td b/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
index 8acd4691787e4c6..ca6ac0581a3f709 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
@@ -54,6 +54,7 @@ def Enqueue : BuiltinGroup;
 def AsyncCopy : BuiltinGroup;
 def VectorLoadStore : BuiltinGroup;
 def LoadStore : BuiltinGroup;
+def ExpectAssume : BuiltinGroup;
 
 //===----------------------------------------------------------------------===//
 // Class defining a demangled builtin record. The information in the record
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td b/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
index 44b5536becf7f4b..f3940ad8ed1b0bf 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
+++ b/llvm/lib/Target/SPIRV/SPIRVInstrInfo.td
@@ -757,3 +757,7 @@ def OpGroupNonUniformBitwiseXor: OpGroupNUGroup<"BitwiseXor", 361>;
 def OpGroupNonUniformLogicalAnd: OpGroupNUGroup<"LogicalAnd", 362>;
 def OpGroupNonUniformLogicalOr: OpGroupNUGroup<"LogicalOr", 363>;
 def OpGroupNonUniformLogicalXor: OpGroupNUGroup<"LogicalXor", 364>;
+
+// SPV_KHR_expect_assume : Expect assume instructions
+def OpAssumeTrueKHR: Op<5630, (outs), (ins ID:$cond), "OpAssumeTrueKHR $cond">;
+def OpExpectKHR: Op<5631, (outs ID:$res), (ins TYPE:$ty, ID:$val, ID:$expected), "$res = OpExpectKHR $ty $val $expected">;
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index bd3b9b0228434a3..73d6409fb265dd0 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -12,6 +12,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "MCTargetDesc/SPIRVMCTargetDesc.h"
 #include "SPIRV.h"
 #include "SPIRVGlobalRegistry.h"
 #include "SPIRVInstrInfo.h"
@@ -1395,6 +1396,17 @@ bool SPIRVInstructionSelector::selectIntrinsic(Register ResVReg,
     break;
   case Intrinsic::spv_alloca:
     return selectFrameIndex(ResVReg, ResType, I);
+  case Intrinsic::spv_assume:
+    BuildMI(BB, I, I.getDebugLoc(), TII.get(SPIRV::OpAssumeTrueKHR))
+        .addUse(I.getOperand(1).getReg());
+    break;
+  case Intrinsic::spv_expect:
+    BuildMI(BB, I, I.getDebugLoc(), TII.get(SPIRV::OpExpectKHR))
+        .addDef(ResVReg)
+        .addUse(GR.getSPIRVTypeID(ResType))
+        .addUse(I.getOperand(2).getReg())
+        .addUse(I.getOperand(3).getReg());
+    break;
   default:
     llvm_unreachable("Intrinsic selection not implemented");
   }
diff --git a/llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp b/llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp
index 065eacf6ad1f1f7..a938edbeaea9e7e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp
@@ -15,6 +15,8 @@
 //===----------------------------------------------------------------------===//
 
 #include "SPIRVModuleAnalysis.h"
+#include "MCTargetDesc/SPIRVBaseInfo.h"
+#include "MCTargetDesc/SPIRVMCTargetDesc.h"
 #include "SPIRV.h"
 #include "SPIRVSubtarget.h"
 #include "SPIRVTargetMachine.h"
@@ -881,6 +883,11 @@ void addInstrRequirements(const MachineInstr &MI,
   case SPIRV::OpGroupNonUniformBallotFindMSB:
     Reqs.addCapability(SPIRV::Capability::GroupNonUniformBallot);
     break;
+  case SPIRV::OpAssumeTrueKHR:
+  case SPIRV::OpExpectKHR:
+    Reqs.addExtension(SPIRV::Extension::SPV_KHR_expect_assume);
+    Reqs.addCapability(SPIRV::Capability::ExpectAssumeKHR);
+    break;
   default:
     break;
   }
diff --git a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
index 554e66988f09043..87a9a0e4fab845c 100644
--- a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
@@ -24,6 +24,8 @@
 #include "llvm/CodeGen/IntrinsicLowering.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/IntrinsicsSPIRV.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include "llvm/Transforms/Utils/LowerMemIntrinsics.h"
 
@@ -233,6 +235,32 @@ static void buildUMulWithOverflowFunc(Function *UMulFunc) {
   IRB.CreateRet(Res);
 }
 
+static void lowerExpectAssume(IntrinsicInst *II) {
+  // If we cannot use the SPV_KHR_expect_assume extension, then we need to
+  // ignore the intrinsic and move on. It should be removed later on by LLVM.
+  // Otherwise we should lower the intrinsic to the corresponding SPIR-V
+  // instruction.
+  // For @llvm.assume we have OpAssumeTrueKHR.
+  // For @llvm.expect we have OpExpectKHR.
+  //
+  // We need to lower this into a builtin and then the builtin into a SPIR-V
+  // instruction.
+  if (II->getIntrinsicID() == Intrinsic::assume) {
+    Function *F = Intrinsic::getDeclaration(
+        II->getModule(), Intrinsic::SPVIntrinsics::spv_assume);
+    II->setCalledFunction(F);
+  } else if (II->getIntrinsicID() == Intrinsic::expect) {
+    Function *F = Intrinsic::getDeclaration(
+        II->getModule(), Intrinsic::SPVIntrinsics::spv_expect,
+        {II->getOperand(0)->getType()});
+    II->setCalledFunction(F);
+  } else {
+    llvm_unreachable("Unknown intrinsic");
+  }
+
+  return;
+}
+
 static void lowerUMulWithOverflow(IntrinsicInst *UMulIntrinsic) {
   // Get a separate function - otherwise, we'd have to rework the CFG of the
   // current one. Then simply replace the intrinsic uses with a call to the new
@@ -270,6 +298,10 @@ bool SPIRVPrepareFunctions::substituteIntrinsicCalls(Function *F) {
       } else if (II->getIntrinsicID() == Intrinsic::umul_with_overflow) {
         lowerUMulWithOverflow(II);
         Changed = true;
+      } else if (II->getIntrinsicID() == Intrinsic::assume ||
+                 II->getIntrinsicID() == Intrinsic::expect) {
+        lowerExpectAssume(II);
+        Changed = true;
       }
     }
   }
diff --git a/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp b/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
index 1bad1d8109623b6..fdf103941f4b95e 100644
--- a/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "SPIRVSubtarget.h"
+#include "MCTargetDesc/SPIRVBaseInfo.h"
 #include "SPIRV.h"
 #include "SPIRVGlobalRegistry.h"
 #include "SPIRVLegalizerInfo.h"
diff --git a/llvm/lib/Target/SPIRV/SPIRVSymbolicOperands.td b/llvm/lib/Target/SPIRV/SPIRVSymbolicOperands.td
index c27d2826c0159d1..c9d31f18119ed57 100644
--- a/llvm/lib/Target/SPIRV/SPIRVSymbolicOperands.td
+++ b/llvm/lib/Target/SPIRV/SPIRVSymbolicOperands.td
@@ -450,6 +450,7 @@ defm PhysicalStorageBufferAddressesEXT : CapabilityOperand<5347, 0, 0, [], [Shad
 defm CooperativeMatrixNV : CapabilityOperand<5357, 0, 0, [], [Shader]>;
 defm ArbitraryPrecisionIntegersINTEL : CapabilityOperand<5844, 0, 0, [SPV_INTEL_arbitrary_precision_integers], [Int8, Int16]>;
 defm OptNoneINTEL : CapabilityOperand<6094, 0, 0, [SPV_INTEL_optnone], []>;
+defm ExpectAssumeKHR : CapabilityOperand<5629, 0, 0, [SPV_KHR_expect_assume], []>;
 
 //===----------------------------------------------------------------------===//
 // Multiclass used to define SourceLanguage enum values and at the same time
diff --git a/llvm/test/CodeGen/SPIRV/assume.ll b/llvm/test/CodeGen/SPIRV/assume.ll
new file mode 100644
index 000000000000000..50d532d200a3820
--- /dev/null
+++ b/llvm/test/CodeGen/SPIRV/assume.ll
@@ -0,0 +1,15 @@
+; RUN: llc -mtriple=spirv32-unknown-unknown < %s | FileCheck %s
+; RUN: llc -mtriple=spirv64-unknown-unknown < %s | FileCheck %s
+
+;      CHECK: OpCapability ExpectAssumeKHR
+; CHECK-NEXT: OpExtension "SPV_KHR_expect_assume"
+
+declare void @llvm.assume(i1)
+
+;  CHECK-DAG: %9 = OpIEqual %5 %6 %7
+; CHECK-NEXT: OpAssumeTrueKHR %9
+define void @assumeeq(i32 %x, i32 %y) {
+    %cmp = icmp eq i32 %x, %y
+    call void @llvm.assume(i1 %cmp)
+    ret void
+}
\ No newline at end of file
diff --git a/llvm/test/CodeGen/SPIRV/expect.ll b/llvm/test/CodeGen/SPIRV/expect.ll
new file mode 100644
index 000000000000000..da7acfb7504dfee
--- /dev/null
+++ b/llvm/test/CodeGen/SPIRV/expect.ll
@@ -0,0 +1,20 @@
+; RUN: llc -mtriple=spirv32-unknown-unknown < %s | FileCheck %s
+; RUN: llc -mtriple=spirv64-unknown-unknown < %s | FileCheck %s
+
+;      CHECK: OpCapability ExpectAssumeKHR
+; CHECK-NEXT: OpExtension "SPV_KHR_expect_assume"
+
+declare i32 @llvm.expect.i32(i32, i32) 
+declare i32 @getOne()
+
+; CHECK-DAG: %2 = OpTypeInt 32 0
+; CHECK-DAG: %6 = OpFunctionParameter %2
+; CHECK-DAG: %9 = OpIMul %2 %6 %8
+; CHECK-DAG: %10 = OpExpectKHR %2 %9 %6
+
+define i32 @test(i32 %x) {
+  %one = call i32 @getOne()
+  %val = mul i32 %x, %one
+  %v = call i32 @llvm.expect.i32(i32 %val, i32 %x)
+  ret i32 %v
+}

llvm/lib/Target/SPIRV/SPIRVBuiltins.td

llvm/lib/Target/SPIRV/SPIRVInstrInfo.td

llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp

iliya-diyachkov

In general it looks good, but I found a few small issues.

llvm/lib/Target/SPIRV/SPIRVInstrInfo.td

llvm/test/CodeGen/SPIRV/expect.ll

llvm/test/CodeGen/SPIRV/assume.ll

llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp

llvm/test/CodeGen/SPIRV/assume.ll

llvm/test/CodeGen/SPIRV/expect.ll

github-actions · 2023-09-22T10:36:05Z

✅ With the latest revision this PR passed the C/C++ code formatter.

pmatos · 2023-09-22T12:52:26Z

I have rebased on main to resolve conflicts with the just landed SPV_KHR_bit_instructions support. Force-pushed branch.

iliya-diyachkov

It looks good to me. Thanks for your patch, Paulo.

pmatos · 2023-09-25T06:54:24Z

It looks good to me. Thanks for your patch, Paulo.

Thanks for your patience reviewing this.

Adds new extension SPV_KHR_expect_assume, new capability ExpectAssumeKHR as well as the new instructions: * OpExpectKHR * OpAssumeTrueKHR These are lowered from respectively llvm.expect.<ty> and llvm.assume intrinsics. Differential Revision: https://reviews.llvm.org/D157696

michalpaszkowski

Thank you @pmatos for the patch and thank you @iliya-diyachkov for the comments! The patch LGTM!

One last thing before you merge, please add the missing lines at the end of IntrinsicsSPIRV.td and assume.ll files for consistency.

llvm/include/llvm/IR/IntrinsicsSPIRV.td

llvm/test/CodeGen/SPIRV/assume.ll

pmatos · 2023-09-25T07:16:33Z

Thank you @pmatos for the patch and thank you @iliya-diyachkov for the comments! The patch LGTM!

One last thing before you merge, please add the missing lines at the end of IntrinsicsSPIRV.td and assume.ll files for consistency.

thanks for the remaining comments. just in time before merging. :)

Local branch amd-gfx 850c4ac Merged main:f7bf99fb529f into amd-gfx:d7d51706c803 Remote branch main 0564065 [SPIRV] Implement support for SPV_KHR_expect_assume (llvm#66217)

pmatos requested a review from a team as a code owner September 13, 2023 15:00

llvmbot added the llvm:ir label Sep 13, 2023

pmatos requested review from michalpaszkowski and iliya-diyachkov September 13, 2023 15:01

pmatos self-assigned this Sep 13, 2023

pmatos added the backend:SPIR-V label Sep 13, 2023

iliya-diyachkov reviewed Sep 17, 2023

View reviewed changes

llvm/lib/Target/SPIRV/SPIRVBuiltins.td Outdated Show resolved Hide resolved

llvm/lib/Target/SPIRV/SPIRVInstrInfo.td Outdated Show resolved Hide resolved

llvm/lib/Target/SPIRV/SPIRVModuleAnalysis.cpp Outdated Show resolved Hide resolved

pmatos requested a review from iliya-diyachkov September 20, 2023 11:14

pmatos force-pushed the SPV_KHR_expect_assume branch from dfe5eda to 596c10b Compare September 20, 2023 11:24

iliya-diyachkov reviewed Sep 21, 2023

View reviewed changes

llvm/lib/Target/SPIRV/SPIRVInstrInfo.td Outdated Show resolved Hide resolved

llvm/test/CodeGen/SPIRV/expect.ll Outdated Show resolved Hide resolved

llvm/test/CodeGen/SPIRV/assume.ll Outdated Show resolved Hide resolved

llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp Outdated Show resolved Hide resolved

iliya-diyachkov reviewed Sep 22, 2023

View reviewed changes

llvm/test/CodeGen/SPIRV/assume.ll Outdated Show resolved Hide resolved

llvm/test/CodeGen/SPIRV/expect.ll Outdated Show resolved Hide resolved

pmatos requested a review from iliya-diyachkov September 22, 2023 12:44

pmatos force-pushed the SPV_KHR_expect_assume branch from 9991145 to c08a3bf Compare September 22, 2023 12:50

iliya-diyachkov approved these changes Sep 23, 2023

View reviewed changes

pmatos added 5 commits September 25, 2023 08:55

Address comments from @iliya-diyachkov.

8f92dc8

Address comments.

d8e28ff

Rerun clang-format

ea957ff

left align

ca15e29

michalpaszkowski approved these changes Sep 25, 2023

View reviewed changes

llvm/include/llvm/IR/IntrinsicsSPIRV.td Outdated Show resolved Hide resolved

llvm/test/CodeGen/SPIRV/assume.ll Outdated Show resolved Hide resolved

Add line at end of file.

af5ff04

pmatos force-pushed the SPV_KHR_expect_assume branch from f2762a4 to af5ff04 Compare September 25, 2023 07:16

pmatos merged commit 0564065 into llvm:main Sep 25, 2023

pmatos deleted the SPV_KHR_expect_assume branch September 25, 2023 07:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPIRV] Implement support for SPV_KHR_expect_assume #66217

[SPIRV] Implement support for SPV_KHR_expect_assume #66217

Uh oh!

pmatos commented Sep 13, 2023

Uh oh!

pmatos commented Sep 13, 2023

Uh oh!

llvmbot commented Sep 13, 2023

Previously https://reviews.llvm.org/D157696

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iliya-diyachkov left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 22, 2023 •

edited

Loading

Uh oh!

pmatos commented Sep 22, 2023

Uh oh!

iliya-diyachkov left a comment

Uh oh!

pmatos commented Sep 25, 2023

Uh oh!

michalpaszkowski left a comment

Uh oh!

Uh oh!

Uh oh!

pmatos commented Sep 25, 2023

Uh oh!

Uh oh!

[SPIRV] Implement support for SPV_KHR_expect_assume #66217

[SPIRV] Implement support for SPV_KHR_expect_assume #66217

Uh oh!

Conversation

pmatos commented Sep 13, 2023

Uh oh!

pmatos commented Sep 13, 2023

Uh oh!

llvmbot commented Sep 13, 2023

Previously https://reviews.llvm.org/D157696

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iliya-diyachkov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmatos commented Sep 22, 2023

Uh oh!

iliya-diyachkov left a comment

Choose a reason for hiding this comment

Uh oh!

pmatos commented Sep 25, 2023

Uh oh!

michalpaszkowski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pmatos commented Sep 25, 2023

Uh oh!

Uh oh!

github-actions bot commented Sep 22, 2023 •

edited

Loading