[AMDGPU] Omit umin on ctlz/cttz if operand is non-zero.

[PeddleSpam]

No description provided.

[llvmbot]

@llvm/pr-subscribers-backend-amdgpu

Author: Leon Clark (PeddleSpam)

Changes

---

Full diff: https://github.com/llvm/llvm-project/pull/79127.diff

3 Files Affected:

• (modified) llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp (+6-1)
• (modified) llvm/test/CodeGen/AMDGPU/cttz.ll (-2)
• (modified) llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll (-1)

diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
index 55d95154c75878b..a6ec514b09c66d3 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
@@ -3114,8 +3114,13 @@ SDValue AMDGPUTargetLowering::LowerCTLZ_CTTZ(SDValue Op, SelectionDAG &DAG) cons
     // (cttz hi:lo) -> (umin (S_FF1_I32_B64 src), 64)
     // (ctlz_zero_undef src) -> (S_FLBIT_I32_B64 src)
     // (cttz_zero_undef src) -> (S_FF1_I32_B64 src)
+
+    // umin can be omitted if the operand is known to be non-zero.
+    auto KB = DAG.computeKnownBits(Src);
+    auto const IsNonZero = KB.countMinPopulation() > 0u;
+
     SDValue NewOpr = DAG.getNode(NewOpc, SL, MVT::i32, Src);
-    if (!ZeroUndef) {
+    if (!ZeroUndef && !IsNonZero) {
       const SDValue ConstVal = DAG.getConstant(
           Op.getValueType().getScalarSizeInBits(), SL, MVT::i32);
       NewOpr = DAG.getNode(ISD::UMIN, SL, MVT::i32, NewOpr, ConstVal);
diff --git a/llvm/test/CodeGen/AMDGPU/cttz.ll b/llvm/test/CodeGen/AMDGPU/cttz.ll
index 118d6c123046b79..ee2894a66fbfcc0 100644
--- a/llvm/test/CodeGen/AMDGPU/cttz.ll
+++ b/llvm/test/CodeGen/AMDGPU/cttz.ll
@@ -1408,7 +1408,6 @@ define amdgpu_kernel void @v_cttz_i32_sel_ne_bitwidth(ptr addrspace(1) noalias %
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_or_b32_e32 v2, 0x10000, v0
 ; VI-NEXT:    v_ffbl_b32_e32 v2, v2
-; VI-NEXT:    v_min_u32_e32 v2, 32, v2
 ; VI-NEXT:    v_cmp_ne_u16_e32 vcc, 0, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v0, v1, v2, vcc
 ; VI-NEXT:    buffer_store_short v0, off, s[4:7], 0
@@ -1451,7 +1450,6 @@ define amdgpu_kernel void @v_cttz_i32_sel_ne_bitwidth(ptr addrspace(1) noalias %
 ; GFX10-NEXT:    v_or_b32_e32 v2, 0x10000, v1
 ; GFX10-NEXT:    v_cmp_ne_u16_e32 vcc_lo, 0, v1
 ; GFX10-NEXT:    v_ffbl_b32_e32 v2, v2
-; GFX10-NEXT:    v_min_u32_e32 v2, 32, v2
 ; GFX10-NEXT:    v_cndmask_b32_e32 v1, 0xffff, v2, vcc_lo
 ; GFX10-NEXT:    global_store_short v0, v1, s[0:1]
 ; GFX10-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll b/llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll
index 71f1cd54d705c83..392a44318b0a5bb 100644
--- a/llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll
+++ b/llvm/test/CodeGen/AMDGPU/cttz_zero_undef.ll
@@ -1561,7 +1561,6 @@ define amdgpu_kernel void @v_cttz_i32_sel_ne_bitwidth(ptr addrspace(1) noalias %
 ; VI-NEXT:    v_or_b32_e32 v0, v2, v0
 ; VI-NEXT:    v_or_b32_e32 v2, 0x10000, v0
 ; VI-NEXT:    v_ffbl_b32_e32 v2, v2
-; VI-NEXT:    v_min_u32_e32 v2, 32, v2
 ; VI-NEXT:    v_cmp_ne_u16_e32 vcc, 0, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v2, v1, v2, vcc
 ; VI-NEXT:    v_mov_b32_e32 v0, s0

[arsenm]

can you do the equivalent for globalisel too?

[arsenm]

Should do the equivalent for globalisel too

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[arsenm]

Instead of doing this during the lowering, should the combine on CTLZ/CTTZ transform the non-undef version into the undef version if the input is known non-zero? I thought it was already doing that (it is https://github.com/llvm/llvm-project/blob/9731b77e80261c627d79980f8c275700bdaf6591/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L11005C6-L11005C7)

[PeddleSpam]

Instead of doing this during the lowering, should the combine on CTLZ/CTTZ transform the non-undef version into the undef version if the input is known non-zero? I thought it was already doing that (it is https://github.com/llvm/llvm-project/blob/9731b77e80261c627d79980f8c275700bdaf6591/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L11005C6-L11005C7)

It must be missing some cases. Otherwise the tests wouldn't change.

[arsenm]

Instead of doing this during the lowering, should the combine on CTLZ/CTTZ transform the non-undef version into the undef version if the input is known non-zero? I thought it was already doing that (it is https://github.com/llvm/llvm-project/blob/9731b77e80261c627d79980f8c275700bdaf6591/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L11005C6-L11005C7)

It must be missing some cases. Otherwise the tests wouldn't change.

Yes, so should debug why that happened. We shouldn't need to reinvent optimizations during the lowering

[PeddleSpam]

Instead of doing this during the lowering, should the combine on CTLZ/CTTZ transform the non-undef version into the undef version if the input is known non-zero? I thought it was already doing that (it is https://github.com/llvm/llvm-project/blob/9731b77e80261c627d79980f8c275700bdaf6591/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L11005C6-L11005C7)

It must be missing some cases. Otherwise the tests wouldn't change.

Yes, so should debug why that happened. We shouldn't need to reinvent optimizations during the lowering

At the point where DAGCombiner is called we only have a CTLZ/CTTZ op of a load, so SelectionDAG::isKnownNeverZero correctly returns false. However, during legalisation we introduce an OR of the loaded value with a constant.

Legalizing: t18: i16 = cttz t45
Trying to promote node
Creating new node: t51: i32 = any_extend t45
Creating constant: t52: i32 = Constant<65536>
Creating new node: t53: i32 = or t51, Constant:i32<65536>
Creating new node: t54: i32 = cttz t53
Creating new node: t55: i16 = truncate t54
Successfully promoted node
 ... replacing: t18: i16 = cttz t45
     with:      t55: i16 = truncate t54

Custom legalisation happens after this so we're able to determine that the operand is non-zero. It's an optimisation that can only happen after promotion.

[jayfoad]

However, during legalisation we introduce an OR of the loaded value with a constant.

At that point the cttz should have been replaced with cttz_zero_undef. Does #92514 work for you?