[RISCVInsertVSETVLI] Allow PRE with non-immediate AVLs #71728
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Extend our PRE logic to cover non-immediate AVL values. This covers large constant AVLs (which must be materialized in registers), and may help some code written explicitly with intrinsics. Looking at the existing code, I can't entirely figure out why I thought we needed VL == AVL to perform the PRE. My best guess is that I was worried about the VLMAX < VL < 2 * VLMAX case, but the spec explicitly says that vsetvli must be determinist on any particular AVL value. That case was, possibly by accident, covering another legality precondition. Specifically, by only returning true for immediate and VLMAX AVL values, we didn't encounter the case where the AVL was a register and that register wasn't available in the predecessor (e.g. if AVL is a load in the MBB block itself).
|
@llvm/pr-subscribers-backend-risc-v Author: Philip Reames (preames) ChangesExtend our PRE logic to cover non-immediate AVL values. This covers large constant AVLs (which must be materialized in registers), and may help some code written explicitly with intrinsics. Looking at the existing code, I can't entirely figure out why I thought we needed VL == AVL to perform the PRE. My best guess is that I was worried about the VLMAX < VL < 2 * VLMAX case, but the spec explicitly says that vsetvli must be determinist on any particular AVL value. That case was, possibly by accident, covering another legality precondition. Specifically, by only returning true for immediate and VLMAX AVL values, we didn't encounter the case where the AVL was a register and that register wasn't available in the predecessor (e.g. if AVL is a load in the MBB block itself). Full diff: https://github.com/llvm/llvm-project/pull/71728.diff 3 Files Affected:
diff --git a/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp b/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
index f6d8b1f0a70e13d..4d665e1047637b9 100644
--- a/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
+++ b/llvm/lib/Target/RISCV/RISCVInsertVSETVLI.cpp
@@ -1308,29 +1308,6 @@ void RISCVInsertVSETVLI::emitVSETVLIs(MachineBasicBlock &MBB) {
}
}
-/// Return true if the VL value configured by a vset(i)vli with the
-/// provided Info must be equal to the requested AVL. That is, that
-/// AVL <= VLMAX.
-static bool willVLBeAVL(const VSETVLIInfo &Info, const RISCVSubtarget &ST) {
- if (!Info.hasAVLImm())
- // VLMAX is always the same value.
- // TODO: Could extend to other registers by looking at the associated vreg
- // def placement.
- return RISCV::X0 == Info.getAVLReg();
-
- unsigned AVL = Info.getAVLImm();
- unsigned SEW = Info.getSEW();
- unsigned AVLInBits = AVL * SEW;
-
- unsigned LMul;
- bool Fractional;
- std::tie(LMul, Fractional) = RISCVVType::decodeVLMUL(Info.getVLMUL());
-
- if (Fractional)
- return ST.getRealMinVLen() / LMul >= AVLInBits;
- return ST.getRealMinVLen() * LMul >= AVLInBits;
-}
-
/// Perform simple partial redundancy elimination of the VSETVLI instructions
/// we're about to insert by looking for cases where we can PRE from the
/// beginning of one block to the end of one of its predecessors. Specifically,
@@ -1364,9 +1341,21 @@ void RISCVInsertVSETVLI::doPRE(MachineBasicBlock &MBB) {
if (UnavailablePred->succ_size() != 1)
return;
- // If VL can be less than AVL, then we can't reduce the frequency of exec.
- if (!willVLBeAVL(AvailableInfo, *ST))
- return;
+ // If the AVL value is a register (other than our VLMAX sentinel),
+ // we need to prove the value is available at the point we're going
+ // to insert the vsetvli at.
+ if (AvailableInfo.hasAVLReg() && RISCV::X0 != AvailableInfo.getAVLReg()) {
+ MachineInstr *AVLDefMI = MRI->getVRegDef(AvailableInfo.getAVLReg());
+ if (!AVLDefMI)
+ return;
+ // This is an inline dominance check which covers the case of
+ // UnavailablePrede being the preheader of a loop.
+ if (AVLDefMI->getParent() != UnavailablePred)
+ return;
+ for (auto &TermMI : UnavailablePred->terminators())
+ if (&TermMI == AVLDefMI)
+ return;
+ }
// Model the effect of changing the input state of the block MBB to
// AvailableInfo. We're looking for two issues here; one legality,
diff --git a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-store-asm.ll b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-store-asm.ll
index ef970ad63ae77a8..846295b3ead27dc 100644
--- a/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-store-asm.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/fixed-vectors-strided-load-store-asm.ll
@@ -14,12 +14,12 @@ define void @gather(ptr noalias nocapture %A, ptr noalias nocapture readonly %B)
; CHECK-LABEL: gather:
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
-; CHECK-NEXT: li a3, 32
-; CHECK-NEXT: li a4, 5
+; CHECK-NEXT: li a4, 32
+; CHECK-NEXT: li a3, 5
+; CHECK-NEXT: vsetvli zero, a4, e8, m1, ta, ma
; CHECK-NEXT: .LBB0_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e8, m1, ta, ma
-; CHECK-NEXT: vlse8.v v8, (a1), a4
+; CHECK-NEXT: vlse8.v v8, (a1), a3
; CHECK-NEXT: vle8.v v9, (a0)
; CHECK-NEXT: vadd.vv v8, v9, v8
; CHECK-NEXT: vse8.v v8, (a0)
@@ -126,12 +126,12 @@ define void @gather_negative_stride(ptr noalias nocapture %A, ptr noalias nocapt
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: addi a1, a1, 155
; CHECK-NEXT: li a2, 1024
-; CHECK-NEXT: li a3, 32
-; CHECK-NEXT: li a4, -5
+; CHECK-NEXT: li a4, 32
+; CHECK-NEXT: li a3, -5
+; CHECK-NEXT: vsetvli zero, a4, e8, m1, ta, ma
; CHECK-NEXT: .LBB2_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e8, m1, ta, ma
-; CHECK-NEXT: vlse8.v v8, (a1), a4
+; CHECK-NEXT: vlse8.v v8, (a1), a3
; CHECK-NEXT: vle8.v v9, (a0)
; CHECK-NEXT: vadd.vv v8, v9, v8
; CHECK-NEXT: vse8.v v8, (a0)
@@ -168,12 +168,12 @@ define void @gather_zero_stride(ptr noalias nocapture %A, ptr noalias nocapture
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; CHECK-NEXT: .LBB3_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: lbu a4, 0(a1)
-; CHECK-NEXT: vsetvli zero, a3, e8, m1, ta, ma
+; CHECK-NEXT: lbu a3, 0(a1)
; CHECK-NEXT: vle8.v v8, (a0)
-; CHECK-NEXT: vadd.vx v8, v8, a4
+; CHECK-NEXT: vadd.vx v8, v8, a3
; CHECK-NEXT: vse8.v v8, (a0)
; CHECK-NEXT: addi a2, a2, -32
; CHECK-NEXT: addi a0, a0, 32
@@ -208,9 +208,9 @@ define void @gather_zero_stride_unfold(ptr noalias nocapture %A, ptr noalias noc
; V: # %bb.0: # %entry
; V-NEXT: li a2, 1024
; V-NEXT: li a3, 32
+; V-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; V-NEXT: .LBB4_1: # %vector.body
; V-NEXT: # =>This Inner Loop Header: Depth=1
-; V-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; V-NEXT: vlse8.v v8, (a1), zero
; V-NEXT: vle8.v v9, (a0)
; V-NEXT: vdivu.vv v8, v8, v9
@@ -226,9 +226,9 @@ define void @gather_zero_stride_unfold(ptr noalias nocapture %A, ptr noalias noc
; ZVE32F: # %bb.0: # %entry
; ZVE32F-NEXT: li a2, 1024
; ZVE32F-NEXT: li a3, 32
+; ZVE32F-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; ZVE32F-NEXT: .LBB4_1: # %vector.body
; ZVE32F-NEXT: # =>This Inner Loop Header: Depth=1
-; ZVE32F-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; ZVE32F-NEXT: vlse8.v v8, (a1), zero
; ZVE32F-NEXT: vle8.v v9, (a0)
; ZVE32F-NEXT: vdivu.vv v8, v8, v9
@@ -244,12 +244,12 @@ define void @gather_zero_stride_unfold(ptr noalias nocapture %A, ptr noalias noc
; NOT-OPTIMIZED: # %bb.0: # %entry
; NOT-OPTIMIZED-NEXT: li a2, 1024
; NOT-OPTIMIZED-NEXT: li a3, 32
+; NOT-OPTIMIZED-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; NOT-OPTIMIZED-NEXT: .LBB4_1: # %vector.body
; NOT-OPTIMIZED-NEXT: # =>This Inner Loop Header: Depth=1
-; NOT-OPTIMIZED-NEXT: lbu a4, 0(a1)
-; NOT-OPTIMIZED-NEXT: vsetvli zero, a3, e8, m1, ta, ma
+; NOT-OPTIMIZED-NEXT: lbu a3, 0(a1)
; NOT-OPTIMIZED-NEXT: vle8.v v8, (a0)
-; NOT-OPTIMIZED-NEXT: vmv.v.x v9, a4
+; NOT-OPTIMIZED-NEXT: vmv.v.x v9, a3
; NOT-OPTIMIZED-NEXT: vdivu.vv v8, v9, v8
; NOT-OPTIMIZED-NEXT: vse8.v v8, (a0)
; NOT-OPTIMIZED-NEXT: addi a2, a2, -32
@@ -288,15 +288,15 @@ define void @scatter(ptr noalias nocapture %A, ptr noalias nocapture readonly %B
; CHECK-LABEL: scatter:
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
-; CHECK-NEXT: li a3, 32
-; CHECK-NEXT: li a4, 5
+; CHECK-NEXT: li a4, 32
+; CHECK-NEXT: li a3, 5
+; CHECK-NEXT: vsetvli zero, a4, e8, m1, ta, ma
; CHECK-NEXT: .LBB5_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e8, m1, ta, ma
; CHECK-NEXT: vle8.v v8, (a1)
-; CHECK-NEXT: vlse8.v v9, (a0), a4
+; CHECK-NEXT: vlse8.v v9, (a0), a3
; CHECK-NEXT: vadd.vv v8, v9, v8
-; CHECK-NEXT: vsse8.v v8, (a0), a4
+; CHECK-NEXT: vsse8.v v8, (a0), a3
; CHECK-NEXT: addi a2, a2, -32
; CHECK-NEXT: addi a1, a1, 32
; CHECK-NEXT: addi a0, a0, 160
@@ -821,20 +821,20 @@ define void @strided_load_startval_add_with_splat(ptr noalias nocapture %arg, pt
; CHECK-NEXT: add a6, a0, a2
; CHECK-NEXT: add a2, a1, a2
; CHECK-NEXT: add a2, a2, a7
-; CHECK-NEXT: li a7, 32
-; CHECK-NEXT: li t0, 5
-; CHECK-NEXT: mv t1, a5
+; CHECK-NEXT: li t0, 32
+; CHECK-NEXT: li a7, 5
+; CHECK-NEXT: vsetvli zero, t0, e8, m1, ta, ma
+; CHECK-NEXT: mv t0, a5
; CHECK-NEXT: .LBB13_3: # %bb15
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a7, e8, m1, ta, ma
-; CHECK-NEXT: vlse8.v v8, (a2), t0
+; CHECK-NEXT: vlse8.v v8, (a2), a7
; CHECK-NEXT: vle8.v v9, (a6)
; CHECK-NEXT: vadd.vv v8, v9, v8
; CHECK-NEXT: vse8.v v8, (a6)
-; CHECK-NEXT: addi t1, t1, -32
+; CHECK-NEXT: addi t0, t0, -32
; CHECK-NEXT: addi a6, a6, 32
; CHECK-NEXT: addi a2, a2, 160
-; CHECK-NEXT: bnez t1, .LBB13_3
+; CHECK-NEXT: bnez t0, .LBB13_3
; CHECK-NEXT: # %bb.4: # %bb30
; CHECK-NEXT: beq a4, a5, .LBB13_7
; CHECK-NEXT: .LBB13_5: # %bb32
diff --git a/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll b/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
index f08bfce409305c8..9b083fc286e7c02 100644
--- a/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/sink-splat-operands.ll
@@ -3892,9 +3892,9 @@ define void @sink_splat_mul_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB74_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vmul.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -3927,9 +3927,9 @@ define void @sink_splat_add_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB75_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vadd.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -3962,9 +3962,9 @@ define void @sink_splat_sub_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB76_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vsub.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -3997,9 +3997,9 @@ define void @sink_splat_rsub_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB77_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vrsub.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -4032,9 +4032,9 @@ define void @sink_splat_and_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB78_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vand.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -4067,9 +4067,9 @@ define void @sink_splat_or_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB79_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vor.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
@@ -4102,9 +4102,9 @@ define void @sink_splat_xor_lmul8(ptr nocapture %a, i32 signext %x) {
; CHECK: # %bb.0: # %entry
; CHECK-NEXT: li a2, 1024
; CHECK-NEXT: li a3, 32
+; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: .LBB80_1: # %vector.body
; CHECK-NEXT: # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: vsetvli zero, a3, e32, m8, ta, ma
; CHECK-NEXT: vle32.v v8, (a0)
; CHECK-NEXT: vxor.vx v8, v8, a1
; CHECK-NEXT: vse32.v v8, (a0)
|
wangpc-pp
requested review from
wangpc-pp
and removed request for
pcwang-thead
Co-authored-by: Luke Lau <luke_lau@icloud.com>
zahiraam
pushed a commit
to zahiraam/llvm-project
that referenced
this pull request
Nov 20, 2023
Extend our PRE logic to cover non-immediate AVL values. This covers large constant AVLs (which must be materialized in registers), and may help some code written explicitly with intrinsics. Looking at the existing code, I can't entirely figure out why I thought we needed VL == AVL to perform the PRE. My best guess is that I was worried about the VLMAX < VL < 2 * VLMAX case, but the spec explicitly says that vsetvli must be determinist on any particular AVL value. That case was, possibly by accident, covering another legality precondition. Specifically, by only returning true for immediate and VLMAX AVL values, we didn't encounter the case where the AVL was a register and that register wasn't available in the predecessor (e.g. if AVL is a load in the MBB block itself). --------- Co-authored-by: Luke Lau <luke_lau@icloud.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Extend our PRE logic to cover non-immediate AVL values. This covers large constant AVLs (which must be materialized in registers), and may help some code written explicitly with intrinsics.
Looking at the existing code, I can't entirely figure out why I thought we needed VL == AVL to perform the PRE. My best guess is that I was worried about the VLMAX < VL < 2 * VLMAX case, but the spec explicitly says that vsetvli must be determinist on any particular AVL value.
That case was, possibly by accident, covering another legality precondition. Specifically, by only returning true for immediate and VLMAX AVL values, we didn't encounter the case where the AVL was a register and that register wasn't available in the predecessor (e.g. if AVL is a load in the MBB block itself).