[IndVars] Generate zext nneg when locally obvious #70823
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zext nneg was recently added to the IR in llvm#67982. This patch teaches SimplifyIndVars to prefer zext nneg over *both* sext and plain zext, when a local SCEV query indicates the source is non-negative. The choice to prefer zext nneg over sext looks slightly aggressive here, but probably isn't so much in practice. For cases where we'd "remember" the range fact, instcombine would convert the sext into a zext nneg anyways. The only cases where this produces a different result overall are when SCEV knows a non-local fact, and it doesn't get materialized into the IR. Those are exactly the cases where using zext nneg are most useful. We do run the risk of e.g. a missing combine - since we haven't updated most of them yet - but that seems like a manageable risk. Note that there are much deeper algorithmic changes we could make to this code to exploit zext nneg, but this seemed like a reasonable and low risk starting point.
|
@llvm/pr-subscribers-llvm-analysis @llvm/pr-subscribers-llvm-transforms Author: Philip Reames (preames) Changeszext nneg was recently added to the IR in #67982. This patch teaches SimplifyIndVars to prefer zext nneg over both sext and plain zext, when a local SCEV query indicates the source is non-negative. The choice to prefer zext nneg over sext looks slightly aggressive here, but probably isn't so much in practice. For cases where we'd "remember" the range fact, instcombine would convert the sext into a zext nneg anyways. The only cases where this produces a different result overall are when SCEV knows a non-local fact, and it doesn't get materialized into the IR. Those are exactly the cases where using zext nneg are most useful. We do run the risk of e.g. a missing combine - since we haven't updated most of them yet - but that seems like a manageable risk. Note that there are much deeper algorithmic changes we could make to this code to exploit zext nneg, but this seemed like a reasonable and low risk starting point. Full diff: https://github.com/llvm/llvm-project/pull/70823.diff 7 Files Affected:
diff --git a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
index a23ac41acaa58aa..7834b83c238a5a9 100644
--- a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
@@ -1200,6 +1200,16 @@ Value *WidenIV::createExtendInst(Value *NarrowOper, Type *WideType,
L = L->getParentLoop())
Builder.SetInsertPoint(L->getLoopPreheader()->getTerminator());
+ // If we know the operand is never negative, prefer zext nneg.
+ // For constant expressions, fall back to plain sext or zext.
+ if (SE->isKnownNonNegative(SE->getSCEV(NarrowOper))) {
+ auto *Res = Builder.CreateZExt(NarrowOper, WideType);
+ if (auto I = dyn_cast<Instruction>(Res)) {
+ I->setNonNeg(true);
+ return I;
+ }
+ }
+
return IsSigned ? Builder.CreateSExt(NarrowOper, WideType) :
Builder.CreateZExt(NarrowOper, WideType);
}
@@ -1685,6 +1695,17 @@ bool WidenIV::widenWithVariantUse(WidenIV::NarrowIVDefUse DU) {
auto ExtendedOp = [&](Value * V)->Value * {
if (V == NarrowUse)
return WideBO;
+
+ // If we know the operand is never negative, prefer zext nneg.
+ // For constant expressions, fall back to plain sext or zext.
+ if (SE->isKnownNonNegative(SE->getSCEV(V))) {
+ auto *Res = Builder.CreateZExt(V, WideBO->getType());
+ if (auto I = dyn_cast<Instruction>(Res)) {
+ I->setNonNeg(true);
+ return I;
+ }
+ }
+
if (ExtKind == ExtendKind::Zero)
return Builder.CreateZExt(V, WideBO->getType());
else
diff --git a/llvm/test/Analysis/ScalarEvolution/guards.ll b/llvm/test/Analysis/ScalarEvolution/guards.ll
index ea17c5840067afb..137630cd25e6873 100644
--- a/llvm/test/Analysis/ScalarEvolution/guards.ll
+++ b/llvm/test/Analysis/ScalarEvolution/guards.ll
@@ -57,7 +57,7 @@ define void @test_2(i32 %n, ptr %len_buf) {
; CHECK-SAME: (i32 [[N:%.*]], ptr [[LEN_BUF:%.*]]) {
; CHECK-NEXT: entry:
; CHECK-NEXT: [[LEN:%.*]] = load i32, ptr [[LEN_BUF]], align 4, !range [[RNG1:![0-9]+]]
-; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[LEN]] to i64
+; CHECK-NEXT: [[TMP0:%.*]] = zext nneg i32 [[LEN]] to i64
; CHECK-NEXT: [[TMP1:%.*]] = zext i32 [[N]] to i64
; CHECK-NEXT: br label [[LOOP:%.*]]
; CHECK: loop:
diff --git a/llvm/test/Transforms/IndVarSimplify/X86/pr59615.ll b/llvm/test/Transforms/IndVarSimplify/X86/pr59615.ll
index 17b7b9d40b07a53..4fe7f7fd01a0660 100644
--- a/llvm/test/Transforms/IndVarSimplify/X86/pr59615.ll
+++ b/llvm/test/Transforms/IndVarSimplify/X86/pr59615.ll
@@ -17,7 +17,7 @@ define void @test() {
; CHECK-NEXT: ret void
; CHECK: bb8:
; CHECK-NEXT: [[VAR9:%.*]] = load atomic i32, ptr addrspace(1) poison unordered, align 8, !range [[RNG0]], !invariant.load !1, !noundef !1
-; CHECK-NEXT: [[TMP0:%.*]] = zext i32 [[VAR9]] to i64
+; CHECK-NEXT: [[TMP0:%.*]] = zext nneg i32 [[VAR9]] to i64
; CHECK-NEXT: [[VAR10:%.*]] = icmp ult i64 [[INDVARS_IV]], [[TMP0]]
; CHECK-NEXT: br i1 [[VAR10]], label [[BB12]], label [[BB11:%.*]]
; CHECK: bb11:
diff --git a/llvm/test/Transforms/IndVarSimplify/post-inc-range.ll b/llvm/test/Transforms/IndVarSimplify/post-inc-range.ll
index 5c22ba1044b60af..1df0d62168af24e 100644
--- a/llvm/test/Transforms/IndVarSimplify/post-inc-range.ll
+++ b/llvm/test/Transforms/IndVarSimplify/post-inc-range.ll
@@ -120,7 +120,7 @@ define void @test_range_metadata(ptr %array_length_ptr, ptr %base,
; CHECK: for.body:
; CHECK-NEXT: [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT:%.*]], [[FOR_INC:%.*]] ], [ [[TMP0]], [[FOR_BODY_LR_PH:%.*]] ]
; CHECK-NEXT: [[ARRAY_LENGTH:%.*]] = load i32, ptr [[ARRAY_LENGTH_PTR:%.*]], align 4, !range [[RNG0:![0-9]+]]
-; CHECK-NEXT: [[TMP2:%.*]] = zext i32 [[ARRAY_LENGTH]] to i64
+; CHECK-NEXT: [[TMP2:%.*]] = zext nneg i32 [[ARRAY_LENGTH]] to i64
; CHECK-NEXT: [[WITHIN_LIMITS:%.*]] = icmp ult i64 [[INDVARS_IV]], [[TMP2]]
; CHECK-NEXT: br i1 [[WITHIN_LIMITS]], label [[CONTINUE:%.*]], label [[FOR_END:%.*]]
; CHECK: continue:
diff --git a/llvm/test/Transforms/LoopFlatten/widen-iv.ll b/llvm/test/Transforms/LoopFlatten/widen-iv.ll
index ef5f9d78cd04934..65f2e834b4eac52 100644
--- a/llvm/test/Transforms/LoopFlatten/widen-iv.ll
+++ b/llvm/test/Transforms/LoopFlatten/widen-iv.ll
@@ -156,7 +156,7 @@ define void @foo2_sext(i32* nocapture readonly %A, i32 %N, i32 %M) {
; CHECK-NEXT: [[INDVAR:%.*]] = phi i64 [ 0, [[FOR_COND1_PREHEADER_US]] ]
; CHECK-NEXT: [[J_016_US:%.*]] = phi i32 [ 0, [[FOR_COND1_PREHEADER_US]] ]
; CHECK-NEXT: [[TMP5:%.*]] = add nsw i64 [[INDVAR]], [[TMP3]]
-; CHECK-NEXT: [[TMP6:%.*]] = sext i32 [[J_016_US]] to i64
+; CHECK-NEXT: [[TMP6:%.*]] = zext nneg i32 [[J_016_US]] to i64
; CHECK-NEXT: [[TMP7:%.*]] = add nsw i64 [[TMP6]], [[TMP3]]
; CHECK-NEXT: [[ADD_US:%.*]] = add nsw i32 [[J_016_US]], [[MUL_US]]
; CHECK-NEXT: [[IDXPROM_US:%.*]] = sext i32 [[FLATTEN_TRUNCIV]] to i64
diff --git a/llvm/test/Transforms/LoopFlatten/widen-iv2.ll b/llvm/test/Transforms/LoopFlatten/widen-iv2.ll
index 946b98420249e2f..f4c8b90d4bc27b0 100644
--- a/llvm/test/Transforms/LoopFlatten/widen-iv2.ll
+++ b/llvm/test/Transforms/LoopFlatten/widen-iv2.ll
@@ -39,7 +39,7 @@ define dso_local i32 @fn1() local_unnamed_addr #0 {
; CHECK-NEXT: [[INDVAR:%.*]] = phi i64 [ [[INDVAR_NEXT:%.*]], [[FOR_BODY3_US]] ], [ 0, [[FOR_COND1_PREHEADER_US]] ]
; CHECK-NEXT: [[J_014_US:%.*]] = phi i32 [ 0, [[FOR_COND1_PREHEADER_US]] ], [ [[INC_US:%.*]], [[FOR_BODY3_US]] ]
; CHECK-NEXT: [[TMP7:%.*]] = add nsw i64 [[INDVAR]], [[TMP5]]
-; CHECK-NEXT: [[TMP8:%.*]] = sext i32 [[J_014_US]] to i64
+; CHECK-NEXT: [[TMP8:%.*]] = zext nneg i32 [[J_014_US]] to i64
; CHECK-NEXT: [[TMP9:%.*]] = add nsw i64 [[TMP8]], [[TMP5]]
; CHECK-NEXT: [[ADD_US:%.*]] = add nsw i32 [[J_014_US]], [[MUL_US]]
; CHECK-NEXT: [[IDXPROM_US:%.*]] = sext i32 [[ADD_US]] to i64
diff --git a/llvm/test/Transforms/LoopFlatten/widen-iv3.ll b/llvm/test/Transforms/LoopFlatten/widen-iv3.ll
index df8ee6ff0750574..b3a9ac823fd2df7 100644
--- a/llvm/test/Transforms/LoopFlatten/widen-iv3.ll
+++ b/llvm/test/Transforms/LoopFlatten/widen-iv3.ll
@@ -18,7 +18,7 @@ define i16 @foo() {
; CHECK-NEXT: [[SUM_012:%.*]] = phi i16 [ 0, [[ENTRY]] ], [ [[ADD5_LCSSA:%.*]], [[FOR_COND_CLEANUP3]] ]
; CHECK-NEXT: [[TMP0:%.*]] = mul nsw i32 [[INDVAR2]], 16
; CHECK-NEXT: [[MUL:%.*]] = mul nsw i16 [[I_013]], 16
-; CHECK-NEXT: [[TMP1:%.*]] = zext i16 [[MUL]] to i32
+; CHECK-NEXT: [[TMP1:%.*]] = zext nneg i16 [[MUL]] to i32
; CHECK-NEXT: br label [[FOR_BODY4:%.*]]
; CHECK: for.cond.cleanup:
; CHECK-NEXT: [[ADD5_LCSSA_LCSSA:%.*]] = phi i16 [ [[ADD5_LCSSA]], [[FOR_COND_CLEANUP3]] ]
|
You can test this locally with the following command:git-clang-format --diff 6be0e979896f7dd610abf263f845c532f1be3762 0d658e39e60826af720f0468eb03aa24d847f0e1 -- llvm/lib/Transforms/Utils/SimplifyIndVar.cppView the diff from clang-format here.diff --git a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
index 7834b83c238a..7f8c8a45af62 100644
--- a/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyIndVar.cpp
@@ -1705,7 +1705,7 @@ bool WidenIV::widenWithVariantUse(WidenIV::NarrowIVDefUse DU) {
return I;
}
}
-
+
if (ExtKind == ExtendKind::Zero)
return Builder.CreateZExt(V, WideBO->getType());
else
|
|
I went ahead and landed this as c4b8f1e2b2aaf54eb74a2f40f999dc4c89b37234. The change itself is pretty straight forward, and when attempting to rebase to resolve the test diff, git/github got into a confused state I couldn't figure out how to undo. It didn't seem worth opening a separate PR and closing this one for something so straight forward. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
zext nneg was recently added to the IR in #67982. This patch teaches SimplifyIndVars to prefer zext nneg over both sext and plain zext, when a local SCEV query indicates the source is non-negative.
The choice to prefer zext nneg over sext looks slightly aggressive here, but probably isn't so much in practice. For cases where we'd "remember" the range fact, instcombine would convert the sext into a zext nneg anyways. The only cases where this produces a different result overall are when SCEV knows a non-local fact, and it doesn't get materialized into the IR. Those are exactly the cases where using zext nneg are most useful. We do run the risk of e.g. a missing combine - since we haven't updated most of them yet - but that seems like a manageable risk.
Note that there are much deeper algorithmic changes we could make to this code to exploit zext nneg, but this seemed like a reasonable and low risk starting point.