[VPlan] Move widening decisions to VPlanWideningDecisions.cpp (NFC) - #209887
Open
fhahn wants to merge 2 commits into
Open
[VPlan] Move widening decisions to VPlanWideningDecisions.cpp (NFC)#209887fhahn wants to merge 2 commits into
fhahn wants to merge 2 commits into
Conversation
|
@llvm/pr-subscribers-llvm-transforms Author: Florian Hahn (fhahn) ChangesFollowing up to #209885, This moves Depends on #209883 (included in PR) Patch is 336.08 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209887.diff 8 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/CMakeLists.txt b/llvm/lib/Transforms/Vectorize/CMakeLists.txt
index 6e26203d957cb..b052300c06d8a 100644
--- a/llvm/lib/Transforms/Vectorize/CMakeLists.txt
+++ b/llvm/lib/Transforms/Vectorize/CMakeLists.txt
@@ -30,12 +30,15 @@ add_llvm_component_library(LLVMVectorize
VPlan.cpp
VPlanAnalysis.cpp
VPlanConstruction.cpp
+ VPlanEVLTransforms.cpp
+ VPlanLowering.cpp
VPlanPredicator.cpp
VPlanRecipes.cpp
VPlanTransforms.cpp
VPlanUnroll.cpp
VPlanVerifier.cpp
VPlanUtils.cpp
+ VPlanWideningDecisions.cpp
ADDITIONAL_HEADER_DIRS
${LLVM_MAIN_INCLUDE_DIR}/llvm/Transforms
diff --git a/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp
new file mode 100644
index 0000000000000..d1ffa88b7eff6
--- /dev/null
+++ b/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp
@@ -0,0 +1,657 @@
+//===- VPlanEVLTransforms.cpp - Explicit Vector Length transforms ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file implements the VPlan-to-VPlan transforms related to explicit
+/// vector length (EVL) support.
+///
+//===----------------------------------------------------------------------===//
+
+#include "VPlanTransforms.h"
+#include "LoopVectorizationPlanner.h"
+#include "VPlan.h"
+#include "VPlanCFG.h"
+#include "VPlanHelpers.h"
+#include "VPlanPatternMatch.h"
+#include "VPlanUtils.h"
+#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/IR/Intrinsics.h"
+
+using namespace llvm;
+using namespace VPlanPatternMatch;
+
+/// From the definition of llvm.experimental.get.vector.length,
+/// VPInstruction::ExplicitVectorLength(%AVL) = %AVL when %AVL <= VF.
+bool VPlanTransforms::simplifyKnownEVL(VPlan &Plan, ElementCount VF,
+ PredicatedScalarEvolution &PSE) {
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ VPValue *AVL;
+ if (!match(&R, m_EVL(m_VPValue(AVL))))
+ continue;
+
+ const SCEV *AVLSCEV = vputils::getSCEVExprForVPValue(AVL, PSE);
+ if (isa<SCEVCouldNotCompute>(AVLSCEV))
+ continue;
+ ScalarEvolution &SE = *PSE.getSE();
+ const SCEV *VFSCEV = SE.getElementCount(AVLSCEV->getType(), VF);
+ if (!SE.isKnownPredicate(CmpInst::ICMP_ULE, AVLSCEV, VFSCEV))
+ continue;
+
+ VPValue *Trunc = VPBuilder(&R).createScalarZExtOrTrunc(
+ AVL, Type::getInt32Ty(Plan.getContext()), AVLSCEV->getType(),
+ R.getDebugLoc());
+ if (Trunc != AVL) {
+ auto *TruncR = cast<VPSingleDefRecipe>(Trunc);
+ const DataLayout &DL = Plan.getDataLayout();
+ if (VPValue *Folded =
+ vputils::tryToFoldLiveIns(*TruncR, TruncR->operands(), DL))
+ Trunc = Folded;
+ }
+ R.getVPSingleValue()->replaceAllUsesWith(Trunc);
+ return true;
+ }
+ }
+ return false;
+}
+
+template <typename Op0_t, typename Op1_t> struct RemoveMask_match {
+ Op0_t In;
+ Op1_t &Out;
+
+ RemoveMask_match(const Op0_t &In, Op1_t &Out) : In(In), Out(Out) {}
+
+ template <typename OpTy> bool match(OpTy *V) const {
+ if (m_Specific(In).match(V)) {
+ Out = nullptr;
+ return true;
+ }
+ return m_LogicalAnd(m_Specific(In), m_VPValue(Out)).match(V);
+ }
+};
+
+/// Match a specific mask \p In, or a combination of it (logical-and In, Out).
+/// Returns the remaining part \p Out if so, or nullptr otherwise.
+template <typename Op0_t, typename Op1_t>
+static inline RemoveMask_match<Op0_t, Op1_t> m_RemoveMask(const Op0_t &In,
+ Op1_t &Out) {
+ return RemoveMask_match<Op0_t, Op1_t>(In, Out);
+}
+
+static std::optional<Intrinsic::ID> getVPDivRemIntrinsic(Intrinsic::ID IntrID) {
+ switch (IntrID) {
+ case Intrinsic::masked_udiv:
+ return Intrinsic::vp_udiv;
+ case Intrinsic::masked_sdiv:
+ return Intrinsic::vp_sdiv;
+ case Intrinsic::masked_urem:
+ return Intrinsic::vp_urem;
+ case Intrinsic::masked_srem:
+ return Intrinsic::vp_srem;
+ default:
+ return std::nullopt;
+ }
+}
+
+/// Try to optimize a \p CurRecipe masked by \p HeaderMask to a corresponding
+/// EVL-based recipe without the header mask. Returns nullptr if no EVL-based
+/// recipe could be created.
+/// \p HeaderMask Header Mask.
+/// \p CurRecipe Recipe to be transform.
+/// \p EVL The explicit vector length parameter of vector-predication
+/// intrinsics.
+static VPRecipeBase *optimizeMaskToEVL(VPValue *HeaderMask,
+ VPRecipeBase &CurRecipe, VPValue &EVL) {
+ VPlan *Plan = CurRecipe.getParent()->getPlan();
+ DebugLoc DL = CurRecipe.getDebugLoc();
+ VPValue *Addr, *Mask, *EndPtr;
+
+ /// Adjust any end pointers so that they point to the end of EVL lanes not VF.
+ auto AdjustEndPtr = [&CurRecipe, &EVL](VPValue *EndPtr) {
+ auto *EVLEndPtr = cast<VPVectorEndPointerRecipe>(EndPtr)->clone();
+ EVLEndPtr->insertBefore(&CurRecipe);
+ // Cast EVL (i32) to match the VF operand's type.
+ VPValue *EVLAsVF = VPBuilder(EVLEndPtr).createScalarZExtOrTrunc(
+ &EVL, EVLEndPtr->getOperand(1)->getScalarType(), EVL.getScalarType(),
+ DebugLoc::getUnknown());
+ EVLEndPtr->setOperand(1, EVLAsVF);
+ return EVLEndPtr;
+ };
+
+ auto GetVPReverse = [&CurRecipe, &EVL, Plan,
+ DL](VPValue *V) -> VPWidenIntrinsicRecipe * {
+ if (!V)
+ return nullptr;
+ auto *Reverse = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_reverse, {V, Plan->getTrue(), &EVL},
+ V->getScalarType(), {}, {}, DL);
+ Reverse->insertBefore(&CurRecipe);
+ return Reverse;
+ };
+
+ if (match(&CurRecipe,
+ m_MaskedLoad(m_VPValue(Addr), m_RemoveMask(HeaderMask, Mask))))
+ return new VPWidenLoadEVLRecipe(cast<VPWidenLoadRecipe>(CurRecipe), Addr,
+ EVL, Mask);
+
+ if (match(&CurRecipe,
+ m_MaskedLoad(m_VPValue(EndPtr),
+ m_Reverse(m_RemoveMask(HeaderMask, Mask)))) &&
+ match(EndPtr, m_VecEndPtr(m_VPValue(), m_Specific(&Plan->getVF())))) {
+ Mask = GetVPReverse(Mask);
+ Addr = AdjustEndPtr(EndPtr);
+ auto *LoadR = new VPWidenLoadEVLRecipe(cast<VPWidenLoadRecipe>(CurRecipe),
+ Addr, EVL, Mask);
+ LoadR->insertBefore(&CurRecipe);
+ VPValue *Poison = Plan->getPoison(LoadR->getScalarType());
+ return new VPWidenIntrinsicRecipe(Intrinsic::vector_splice_left,
+ {Poison, LoadR, &EVL},
+ LoadR->getScalarType(), {}, {}, DL);
+ }
+
+ VPValue *Stride;
+ if (match(&CurRecipe, m_Intrinsic<Intrinsic::experimental_vp_strided_load>(
+ m_VPValue(Addr), m_VPValue(Stride),
+ m_RemoveMask(HeaderMask, Mask),
+ m_TruncOrSelf(m_Specific(&Plan->getVF()))))) {
+ if (!Mask)
+ Mask = Plan->getTrue();
+ auto *NewLoad = cast<VPWidenMemIntrinsicRecipe>(&CurRecipe)->clone();
+ NewLoad->setOperand(2, Mask);
+ NewLoad->setOperand(3, &EVL);
+ return NewLoad;
+ }
+
+ VPValue *StoredVal;
+ if (match(&CurRecipe, m_MaskedStore(m_VPValue(Addr), m_VPValue(StoredVal),
+ m_RemoveMask(HeaderMask, Mask))))
+ return new VPWidenStoreEVLRecipe(cast<VPWidenStoreRecipe>(CurRecipe), Addr,
+ StoredVal, EVL, Mask);
+
+ if (match(&CurRecipe,
+ m_MaskedStore(m_VPValue(EndPtr), m_VPValue(StoredVal),
+ m_Reverse(m_RemoveMask(HeaderMask, Mask)))) &&
+ match(EndPtr, m_VecEndPtr(m_VPValue(), m_Specific(&Plan->getVF())))) {
+ Mask = GetVPReverse(Mask);
+ Addr = AdjustEndPtr(EndPtr);
+ VPValue *Poison = Plan->getPoison(StoredVal->getScalarType());
+ auto *SpliceR = new VPWidenIntrinsicRecipe(
+ Intrinsic::vector_splice_right, {StoredVal, Poison, &EVL},
+ StoredVal->getScalarType(), {}, {}, DL);
+ SpliceR->insertBefore(&CurRecipe);
+ return new VPWidenStoreEVLRecipe(cast<VPWidenStoreRecipe>(CurRecipe), Addr,
+ SpliceR, EVL, Mask);
+ }
+
+ if (auto *Rdx = dyn_cast<VPReductionRecipe>(&CurRecipe))
+ if (Rdx->isConditional() &&
+ match(Rdx->getCondOp(), m_RemoveMask(HeaderMask, Mask)))
+ return new VPReductionEVLRecipe(*Rdx, EVL, Mask);
+
+ if (auto *Interleave = dyn_cast<VPInterleaveRecipe>(&CurRecipe))
+ if (Interleave->getMask() &&
+ match(Interleave->getMask(), m_RemoveMask(HeaderMask, Mask)))
+ return new VPInterleaveEVLRecipe(*Interleave, EVL, Mask);
+
+ VPValue *LHS, *RHS;
+ if (match(&CurRecipe, m_SelectLike(m_RemoveMask(HeaderMask, Mask),
+ m_VPValue(LHS), m_VPValue(RHS))))
+ return new VPWidenIntrinsicRecipe(
+ Intrinsic::vp_merge, {Mask ? Mask : Plan->getTrue(), LHS, RHS, &EVL},
+ LHS->getScalarType(), {}, {}, DL);
+
+ if (match(&CurRecipe, m_LastActiveLane(m_Specific(HeaderMask)))) {
+ Type *Ty = CurRecipe.getVPSingleValue()->getScalarType();
+ VPValue *ZExt =
+ VPBuilder(&CurRecipe)
+ .createScalarZExtOrTrunc(&EVL, Ty, EVL.getScalarType(), DL);
+ return new VPInstruction(
+ Instruction::Sub, {ZExt, Plan->getConstantInt(Ty, 1)},
+ VPIRFlags::getDefaultFlags(Instruction::Sub), {}, DL);
+ }
+
+ // lhs | (headermask && rhs) -> vp.merge rhs, true, lhs, evl
+ if (match(&CurRecipe,
+ m_c_BinaryOr(m_VPValue(LHS),
+ m_LogicalAnd(m_Specific(HeaderMask), m_VPValue(RHS)))))
+ return new VPWidenIntrinsicRecipe(Intrinsic::vp_merge,
+ {RHS, Plan->getTrue(), LHS, &EVL},
+ LHS->getScalarType(), {}, {}, DL);
+
+ if (auto *IntrR = dyn_cast<VPWidenIntrinsicRecipe>(&CurRecipe))
+ if (auto VPID = getVPDivRemIntrinsic(IntrR->getVectorIntrinsicID()))
+ if (match(IntrR->getOperand(2), m_RemoveMask(HeaderMask, Mask)))
+ return new VPWidenIntrinsicRecipe(*VPID,
+ {IntrR->getOperand(0),
+ IntrR->getOperand(1),
+ Mask ? Mask : Plan->getTrue(), &EVL},
+ IntrR->getScalarType(), {}, {}, DL);
+
+ return nullptr;
+}
+
+/// Optimize away any EVL-based header masks to VP intrinsic based recipes.
+/// The transforms here need to preserve the original semantics.
+void VPlanTransforms::optimizeEVLMasks(VPlan &Plan) {
+ // Find the EVL-based header mask if it exists: icmp ult step-vector, EVL
+ VPValue *HeaderMask = nullptr, *EVL = nullptr;
+ for (VPRecipeBase &R : *Plan.getVectorLoopRegion()->getEntryBasicBlock()) {
+ if (match(&R, m_SpecificICmp(CmpInst::ICMP_ULT, m_StepVector(),
+ m_VPValue(EVL))) &&
+ match(EVL, m_EVL(m_VPValue()))) {
+ HeaderMask = R.getVPSingleValue();
+ break;
+ }
+ }
+ if (!HeaderMask)
+ return;
+
+ SmallVector<VPRecipeBase *> OldRecipes;
+ for (VPUser *U : vputils::collectUsersRecursively(HeaderMask)) {
+ VPRecipeBase *R = cast<VPRecipeBase>(U);
+ if (auto *NewR = optimizeMaskToEVL(HeaderMask, *R, *EVL)) {
+ NewR->insertBefore(R);
+ for (auto [Old, New] :
+ zip_equal(R->definedValues(), NewR->definedValues()))
+ Old->replaceAllUsesWith(New);
+ OldRecipes.push_back(R);
+ }
+ }
+
+ // Replace remaining (HeaderMask && Mask) with vp.merge (True, Mask,
+ // False, EVL)
+ for (VPUser *U : vputils::collectUsersRecursively(HeaderMask)) {
+ VPValue *Mask;
+ if (match(U, m_LogicalAnd(m_Specific(HeaderMask), m_VPValue(Mask)))) {
+ auto *LogicalAnd = cast<VPInstruction>(U);
+ auto *Merge = new VPWidenIntrinsicRecipe(
+ Intrinsic::vp_merge, {Plan.getTrue(), Mask, Plan.getFalse(), EVL},
+ Mask->getScalarType(), {}, {}, LogicalAnd->getDebugLoc());
+ Merge->insertBefore(LogicalAnd);
+ LogicalAnd->replaceAllUsesWith(Merge);
+ OldRecipes.push_back(LogicalAnd);
+ }
+ }
+
+ // Pull out left splices from any elementwise op.
+ // binop(splice.left(poison, x, evl), live-in)
+ // -> splice.left(poison, binop(x,live-in), evl)
+ pullOutPermutations(
+ Plan,
+ [&EVL](const auto &X) {
+ return m_Intrinsic<Intrinsic::vector_splice_left>(m_Poison(), X,
+ m_Specific(EVL));
+ },
+ [&Plan, &EVL](auto *X) {
+ return new VPWidenIntrinsicRecipe(
+ Intrinsic::vector_splice_left,
+ {Plan.getPoison(X->getScalarType()), X, EVL}, X->getScalarType(),
+ {}, {}, X->getDebugLoc());
+ });
+
+ // Fold the following splice patterns:
+ // splice.right(splice.left(poison, x, evl), poison, evl) -> x
+ // vector.reverse(splice.left(poison, x, evl)) -> vp.reverse(x, true, evl)
+ // splice.right(vector.reverse(x), poison, evl) -> vp.reverse(x, true, evl)
+ for (VPUser *U : vputils::collectUsersRecursively(EVL)) {
+ auto *R = cast<VPRecipeBase>(U);
+ // Remove potentially dead left splices from the transform above.
+ if (match(U, m_Intrinsic<Intrinsic::vector_splice_left>()) &&
+ R->getVPSingleValue()->getNumUsers() == 0) {
+ OldRecipes.push_back(R);
+ continue;
+ }
+
+ VPValue *X;
+ if (match(U, m_Intrinsic<Intrinsic::vector_splice_right>(
+ m_Intrinsic<Intrinsic::vector_splice_left>(
+ m_Poison(), m_VPValue(X), m_Specific(EVL)),
+ m_Poison(), m_Specific(EVL)))) {
+ R->getVPSingleValue()->replaceAllUsesWith(X);
+ OldRecipes.push_back(R);
+ continue;
+ }
+
+ if (!match(U,
+ m_CombineOr(
+ m_Reverse(m_Intrinsic<Intrinsic::vector_splice_left>(
+ m_Poison(), m_VPValue(X), m_Specific(EVL))),
+ m_Intrinsic<Intrinsic::vector_splice_right>(
+ m_Reverse(m_VPValue(X)), m_Poison(), m_Specific(EVL)))))
+ continue;
+
+ auto *VPReverse = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_reverse, {X, Plan.getTrue(), EVL},
+ X->getScalarType(), {}, {}, R->getDebugLoc());
+ VPReverse->insertBefore(R);
+ R->getVPSingleValue()->replaceAllUsesWith(VPReverse);
+ OldRecipes.push_back(R);
+ }
+
+ for (VPRecipeBase *R : reverse(OldRecipes)) {
+ SmallVector<VPValue *> PossiblyDead(R->operands());
+ R->eraseFromParent();
+ for (VPValue *Op : PossiblyDead)
+ vputils::recursivelyDeleteDeadRecipes(Op);
+ }
+}
+
+/// After replacing the canonical IV with a EVL-based IV, fixup recipes that use
+/// VF to use the EVL instead to avoid incorrect updates on the penultimate
+/// iteration.
+static void fixupVFUsersForEVL(VPlan &Plan, VPValue &EVL) {
+ VPRegionBlock *LoopRegion = Plan.getVectorLoopRegion();
+ VPBasicBlock *Header = LoopRegion->getEntryBasicBlock();
+
+ // EVL is i32 but VF/VFxUF are IdxTy. Convert as needed.
+ VPValue *EVLAsIdx =
+ VPBuilder::getToInsertAfter(EVL.getDefiningRecipe())
+ .createScalarZExtOrTrunc(&EVL, Plan.getVF().getScalarType(),
+ EVL.getScalarType(), DebugLoc::getUnknown());
+
+ assert(all_of(Plan.getVF().users(),
+ [&Plan](VPUser *U) {
+ auto IsAllowedUser =
+ IsaPred<VPVectorEndPointerRecipe, VPScalarIVStepsRecipe,
+ VPWidenIntOrFpInductionRecipe,
+ VPWidenMemIntrinsicRecipe>;
+ if (match(U, m_Trunc(m_Specific(&Plan.getVF()))))
+ return all_of(cast<VPSingleDefRecipe>(U)->users(),
+ IsAllowedUser);
+ return IsAllowedUser(U);
+ }) &&
+ "User of VF that we can't transform to EVL.");
+ Plan.getVF().replaceUsesWithIf(EVLAsIdx, [](VPUser &U, unsigned Idx) {
+ return isa<VPWidenIntOrFpInductionRecipe, VPScalarIVStepsRecipe>(U);
+ });
+
+ assert(all_of(Plan.getVFxUF().users(),
+ match_fn(m_CombineOr(
+ m_c_Add(m_Specific(LoopRegion->getCanonicalIV()),
+ m_Specific(&Plan.getVFxUF())),
+ m_Isa<VPWidenPointerInductionRecipe>()))) &&
+ "Only users of VFxUF should be VPWidenPointerInductionRecipe and the "
+ "increment of the canonical induction.");
+ Plan.getVFxUF().replaceUsesWithIf(EVLAsIdx, [](VPUser &U, unsigned Idx) {
+ // Only replace uses in VPWidenPointerInductionRecipe; The increment of the
+ // canonical induction must not be updated.
+ return isa<VPWidenPointerInductionRecipe>(U);
+ });
+
+ // Create a scalar phi to track the previous EVL if fixed-order recurrence is
+ // contained.
+ bool ContainsFORs =
+ any_of(Header->phis(), IsaPred<VPFirstOrderRecurrencePHIRecipe>);
+ if (ContainsFORs) {
+ // TODO: Use VPInstruction::ExplicitVectorLength to get maximum EVL.
+ VPValue *MaxEVL = &Plan.getVF();
+ // Emit VPScalarCastRecipe in preheader if VF is not a 32 bits integer.
+ VPBuilder Builder(LoopRegion->getPreheaderVPBB());
+ MaxEVL = Builder.createScalarZExtOrTrunc(
+ MaxEVL, Type::getInt32Ty(Plan.getContext()), MaxEVL->getScalarType(),
+ DebugLoc::getUnknown());
+
+ Builder.setInsertPoint(Header, Header->getFirstNonPhi());
+ VPValue *PrevEVL = Builder.createScalarPhi(
+ {MaxEVL, &EVL}, DebugLoc::getUnknown(), "prev.evl");
+
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getVectorLoopRegion()->getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ VPValue *V1, *V2;
+ if (!match(&R,
+ m_VPInstruction<VPInstruction::FirstOrderRecurrenceSplice>(
+ m_VPValue(V1), m_VPValue(V2))))
+ continue;
+ VPValue *Imm = Plan.getOrAddLiveIn(
+ ConstantInt::getSigned(Type::getInt32Ty(Plan.getContext()), -1));
+ VPWidenIntrinsicRecipe *VPSplice = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_splice,
+ {V1, V2, Imm, Plan.getTrue(), PrevEVL, &EVL},
+ R.getVPSingleValue()->getScalarType(), {}, {}, R.getDebugLoc());
+ VPSplice->insertBefore(&R);
+ R.getVPSingleValue()->replaceAllUsesWith(VPSplice);
+ }
+ }
+ }
+
+ VPValue *HeaderMask = LoopRegion->getHeaderMask();
+ if (!HeaderMask)
+ return;
+
+ // Ensure that any reduction that uses a select to mask off tail lanes does so
+ // in the vector loop, not the middle block, since EVL tail folding can have
+ // tail elements in the penultimate iteration.
+ assert(all_of(*Plan.getMiddleBlock(), [&Plan, HeaderMask](VPRecipeBase &R) {
+ if (match(&R, m_ComputeReductionResult(m_Select(m_Specific(HeaderMask),
+ m_VPValue(), m_VPValue()))))
+ return R.getOperand(0)->getDefiningRecipe()->getRegion() ==
+ Plan.getVectorLoopRegion();
+ return true;
+ }));
+
+ // Replace the abstract header mask with a mask equivalent to predicating by
+ // EVL: icmp ult step-vector, EVL
+ VPRecipeBase *EVLR = EVL.getDefiningRecipe();
+ VPBuilder Builder(EVLR->getParent(), std::next(EVLR->getIterator()));
+ Type *EVLType = EVL.getScalarType();
+ VPValue *EVLMask = Builder.createICmp(
+ CmpInst::ICMP_ULT,
+ Builder.createNaryOp(VPInstruction::StepVector, {}, EVLType), &EVL);
+ HeaderMask->replaceAllUsesWith(EVLMask);
+}
+
+/// Converts a tail folded vector loop region to step by
+/// VPInstruction::ExplicitVectorLength elements instead of VF elements each
+/// iteration.
+///
+/// - Add a VPCurrentIterationPHIRecipe and related recipes to \p Plan and
+/// replaces all uses of the canonical IV except for the canonical IV
+/// increment with a VPCurrentIterationPHIRecipe. The canonical IV is used
+/// only for loop iterations co...
[truncated]
|
|
@llvm/pr-subscribers-vectorizers Author: Florian Hahn (fhahn) ChangesFollowing up to #209885, This moves Depends on #209883 (included in PR) Patch is 336.08 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209887.diff 8 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/CMakeLists.txt b/llvm/lib/Transforms/Vectorize/CMakeLists.txt
index 6e26203d957cb..b052300c06d8a 100644
--- a/llvm/lib/Transforms/Vectorize/CMakeLists.txt
+++ b/llvm/lib/Transforms/Vectorize/CMakeLists.txt
@@ -30,12 +30,15 @@ add_llvm_component_library(LLVMVectorize
VPlan.cpp
VPlanAnalysis.cpp
VPlanConstruction.cpp
+ VPlanEVLTransforms.cpp
+ VPlanLowering.cpp
VPlanPredicator.cpp
VPlanRecipes.cpp
VPlanTransforms.cpp
VPlanUnroll.cpp
VPlanVerifier.cpp
VPlanUtils.cpp
+ VPlanWideningDecisions.cpp
ADDITIONAL_HEADER_DIRS
${LLVM_MAIN_INCLUDE_DIR}/llvm/Transforms
diff --git a/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp
new file mode 100644
index 0000000000000..d1ffa88b7eff6
--- /dev/null
+++ b/llvm/lib/Transforms/Vectorize/VPlanEVLTransforms.cpp
@@ -0,0 +1,657 @@
+//===- VPlanEVLTransforms.cpp - Explicit Vector Length transforms ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file implements the VPlan-to-VPlan transforms related to explicit
+/// vector length (EVL) support.
+///
+//===----------------------------------------------------------------------===//
+
+#include "VPlanTransforms.h"
+#include "LoopVectorizationPlanner.h"
+#include "VPlan.h"
+#include "VPlanCFG.h"
+#include "VPlanHelpers.h"
+#include "VPlanPatternMatch.h"
+#include "VPlanUtils.h"
+#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/IR/Intrinsics.h"
+
+using namespace llvm;
+using namespace VPlanPatternMatch;
+
+/// From the definition of llvm.experimental.get.vector.length,
+/// VPInstruction::ExplicitVectorLength(%AVL) = %AVL when %AVL <= VF.
+bool VPlanTransforms::simplifyKnownEVL(VPlan &Plan, ElementCount VF,
+ PredicatedScalarEvolution &PSE) {
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ VPValue *AVL;
+ if (!match(&R, m_EVL(m_VPValue(AVL))))
+ continue;
+
+ const SCEV *AVLSCEV = vputils::getSCEVExprForVPValue(AVL, PSE);
+ if (isa<SCEVCouldNotCompute>(AVLSCEV))
+ continue;
+ ScalarEvolution &SE = *PSE.getSE();
+ const SCEV *VFSCEV = SE.getElementCount(AVLSCEV->getType(), VF);
+ if (!SE.isKnownPredicate(CmpInst::ICMP_ULE, AVLSCEV, VFSCEV))
+ continue;
+
+ VPValue *Trunc = VPBuilder(&R).createScalarZExtOrTrunc(
+ AVL, Type::getInt32Ty(Plan.getContext()), AVLSCEV->getType(),
+ R.getDebugLoc());
+ if (Trunc != AVL) {
+ auto *TruncR = cast<VPSingleDefRecipe>(Trunc);
+ const DataLayout &DL = Plan.getDataLayout();
+ if (VPValue *Folded =
+ vputils::tryToFoldLiveIns(*TruncR, TruncR->operands(), DL))
+ Trunc = Folded;
+ }
+ R.getVPSingleValue()->replaceAllUsesWith(Trunc);
+ return true;
+ }
+ }
+ return false;
+}
+
+template <typename Op0_t, typename Op1_t> struct RemoveMask_match {
+ Op0_t In;
+ Op1_t &Out;
+
+ RemoveMask_match(const Op0_t &In, Op1_t &Out) : In(In), Out(Out) {}
+
+ template <typename OpTy> bool match(OpTy *V) const {
+ if (m_Specific(In).match(V)) {
+ Out = nullptr;
+ return true;
+ }
+ return m_LogicalAnd(m_Specific(In), m_VPValue(Out)).match(V);
+ }
+};
+
+/// Match a specific mask \p In, or a combination of it (logical-and In, Out).
+/// Returns the remaining part \p Out if so, or nullptr otherwise.
+template <typename Op0_t, typename Op1_t>
+static inline RemoveMask_match<Op0_t, Op1_t> m_RemoveMask(const Op0_t &In,
+ Op1_t &Out) {
+ return RemoveMask_match<Op0_t, Op1_t>(In, Out);
+}
+
+static std::optional<Intrinsic::ID> getVPDivRemIntrinsic(Intrinsic::ID IntrID) {
+ switch (IntrID) {
+ case Intrinsic::masked_udiv:
+ return Intrinsic::vp_udiv;
+ case Intrinsic::masked_sdiv:
+ return Intrinsic::vp_sdiv;
+ case Intrinsic::masked_urem:
+ return Intrinsic::vp_urem;
+ case Intrinsic::masked_srem:
+ return Intrinsic::vp_srem;
+ default:
+ return std::nullopt;
+ }
+}
+
+/// Try to optimize a \p CurRecipe masked by \p HeaderMask to a corresponding
+/// EVL-based recipe without the header mask. Returns nullptr if no EVL-based
+/// recipe could be created.
+/// \p HeaderMask Header Mask.
+/// \p CurRecipe Recipe to be transform.
+/// \p EVL The explicit vector length parameter of vector-predication
+/// intrinsics.
+static VPRecipeBase *optimizeMaskToEVL(VPValue *HeaderMask,
+ VPRecipeBase &CurRecipe, VPValue &EVL) {
+ VPlan *Plan = CurRecipe.getParent()->getPlan();
+ DebugLoc DL = CurRecipe.getDebugLoc();
+ VPValue *Addr, *Mask, *EndPtr;
+
+ /// Adjust any end pointers so that they point to the end of EVL lanes not VF.
+ auto AdjustEndPtr = [&CurRecipe, &EVL](VPValue *EndPtr) {
+ auto *EVLEndPtr = cast<VPVectorEndPointerRecipe>(EndPtr)->clone();
+ EVLEndPtr->insertBefore(&CurRecipe);
+ // Cast EVL (i32) to match the VF operand's type.
+ VPValue *EVLAsVF = VPBuilder(EVLEndPtr).createScalarZExtOrTrunc(
+ &EVL, EVLEndPtr->getOperand(1)->getScalarType(), EVL.getScalarType(),
+ DebugLoc::getUnknown());
+ EVLEndPtr->setOperand(1, EVLAsVF);
+ return EVLEndPtr;
+ };
+
+ auto GetVPReverse = [&CurRecipe, &EVL, Plan,
+ DL](VPValue *V) -> VPWidenIntrinsicRecipe * {
+ if (!V)
+ return nullptr;
+ auto *Reverse = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_reverse, {V, Plan->getTrue(), &EVL},
+ V->getScalarType(), {}, {}, DL);
+ Reverse->insertBefore(&CurRecipe);
+ return Reverse;
+ };
+
+ if (match(&CurRecipe,
+ m_MaskedLoad(m_VPValue(Addr), m_RemoveMask(HeaderMask, Mask))))
+ return new VPWidenLoadEVLRecipe(cast<VPWidenLoadRecipe>(CurRecipe), Addr,
+ EVL, Mask);
+
+ if (match(&CurRecipe,
+ m_MaskedLoad(m_VPValue(EndPtr),
+ m_Reverse(m_RemoveMask(HeaderMask, Mask)))) &&
+ match(EndPtr, m_VecEndPtr(m_VPValue(), m_Specific(&Plan->getVF())))) {
+ Mask = GetVPReverse(Mask);
+ Addr = AdjustEndPtr(EndPtr);
+ auto *LoadR = new VPWidenLoadEVLRecipe(cast<VPWidenLoadRecipe>(CurRecipe),
+ Addr, EVL, Mask);
+ LoadR->insertBefore(&CurRecipe);
+ VPValue *Poison = Plan->getPoison(LoadR->getScalarType());
+ return new VPWidenIntrinsicRecipe(Intrinsic::vector_splice_left,
+ {Poison, LoadR, &EVL},
+ LoadR->getScalarType(), {}, {}, DL);
+ }
+
+ VPValue *Stride;
+ if (match(&CurRecipe, m_Intrinsic<Intrinsic::experimental_vp_strided_load>(
+ m_VPValue(Addr), m_VPValue(Stride),
+ m_RemoveMask(HeaderMask, Mask),
+ m_TruncOrSelf(m_Specific(&Plan->getVF()))))) {
+ if (!Mask)
+ Mask = Plan->getTrue();
+ auto *NewLoad = cast<VPWidenMemIntrinsicRecipe>(&CurRecipe)->clone();
+ NewLoad->setOperand(2, Mask);
+ NewLoad->setOperand(3, &EVL);
+ return NewLoad;
+ }
+
+ VPValue *StoredVal;
+ if (match(&CurRecipe, m_MaskedStore(m_VPValue(Addr), m_VPValue(StoredVal),
+ m_RemoveMask(HeaderMask, Mask))))
+ return new VPWidenStoreEVLRecipe(cast<VPWidenStoreRecipe>(CurRecipe), Addr,
+ StoredVal, EVL, Mask);
+
+ if (match(&CurRecipe,
+ m_MaskedStore(m_VPValue(EndPtr), m_VPValue(StoredVal),
+ m_Reverse(m_RemoveMask(HeaderMask, Mask)))) &&
+ match(EndPtr, m_VecEndPtr(m_VPValue(), m_Specific(&Plan->getVF())))) {
+ Mask = GetVPReverse(Mask);
+ Addr = AdjustEndPtr(EndPtr);
+ VPValue *Poison = Plan->getPoison(StoredVal->getScalarType());
+ auto *SpliceR = new VPWidenIntrinsicRecipe(
+ Intrinsic::vector_splice_right, {StoredVal, Poison, &EVL},
+ StoredVal->getScalarType(), {}, {}, DL);
+ SpliceR->insertBefore(&CurRecipe);
+ return new VPWidenStoreEVLRecipe(cast<VPWidenStoreRecipe>(CurRecipe), Addr,
+ SpliceR, EVL, Mask);
+ }
+
+ if (auto *Rdx = dyn_cast<VPReductionRecipe>(&CurRecipe))
+ if (Rdx->isConditional() &&
+ match(Rdx->getCondOp(), m_RemoveMask(HeaderMask, Mask)))
+ return new VPReductionEVLRecipe(*Rdx, EVL, Mask);
+
+ if (auto *Interleave = dyn_cast<VPInterleaveRecipe>(&CurRecipe))
+ if (Interleave->getMask() &&
+ match(Interleave->getMask(), m_RemoveMask(HeaderMask, Mask)))
+ return new VPInterleaveEVLRecipe(*Interleave, EVL, Mask);
+
+ VPValue *LHS, *RHS;
+ if (match(&CurRecipe, m_SelectLike(m_RemoveMask(HeaderMask, Mask),
+ m_VPValue(LHS), m_VPValue(RHS))))
+ return new VPWidenIntrinsicRecipe(
+ Intrinsic::vp_merge, {Mask ? Mask : Plan->getTrue(), LHS, RHS, &EVL},
+ LHS->getScalarType(), {}, {}, DL);
+
+ if (match(&CurRecipe, m_LastActiveLane(m_Specific(HeaderMask)))) {
+ Type *Ty = CurRecipe.getVPSingleValue()->getScalarType();
+ VPValue *ZExt =
+ VPBuilder(&CurRecipe)
+ .createScalarZExtOrTrunc(&EVL, Ty, EVL.getScalarType(), DL);
+ return new VPInstruction(
+ Instruction::Sub, {ZExt, Plan->getConstantInt(Ty, 1)},
+ VPIRFlags::getDefaultFlags(Instruction::Sub), {}, DL);
+ }
+
+ // lhs | (headermask && rhs) -> vp.merge rhs, true, lhs, evl
+ if (match(&CurRecipe,
+ m_c_BinaryOr(m_VPValue(LHS),
+ m_LogicalAnd(m_Specific(HeaderMask), m_VPValue(RHS)))))
+ return new VPWidenIntrinsicRecipe(Intrinsic::vp_merge,
+ {RHS, Plan->getTrue(), LHS, &EVL},
+ LHS->getScalarType(), {}, {}, DL);
+
+ if (auto *IntrR = dyn_cast<VPWidenIntrinsicRecipe>(&CurRecipe))
+ if (auto VPID = getVPDivRemIntrinsic(IntrR->getVectorIntrinsicID()))
+ if (match(IntrR->getOperand(2), m_RemoveMask(HeaderMask, Mask)))
+ return new VPWidenIntrinsicRecipe(*VPID,
+ {IntrR->getOperand(0),
+ IntrR->getOperand(1),
+ Mask ? Mask : Plan->getTrue(), &EVL},
+ IntrR->getScalarType(), {}, {}, DL);
+
+ return nullptr;
+}
+
+/// Optimize away any EVL-based header masks to VP intrinsic based recipes.
+/// The transforms here need to preserve the original semantics.
+void VPlanTransforms::optimizeEVLMasks(VPlan &Plan) {
+ // Find the EVL-based header mask if it exists: icmp ult step-vector, EVL
+ VPValue *HeaderMask = nullptr, *EVL = nullptr;
+ for (VPRecipeBase &R : *Plan.getVectorLoopRegion()->getEntryBasicBlock()) {
+ if (match(&R, m_SpecificICmp(CmpInst::ICMP_ULT, m_StepVector(),
+ m_VPValue(EVL))) &&
+ match(EVL, m_EVL(m_VPValue()))) {
+ HeaderMask = R.getVPSingleValue();
+ break;
+ }
+ }
+ if (!HeaderMask)
+ return;
+
+ SmallVector<VPRecipeBase *> OldRecipes;
+ for (VPUser *U : vputils::collectUsersRecursively(HeaderMask)) {
+ VPRecipeBase *R = cast<VPRecipeBase>(U);
+ if (auto *NewR = optimizeMaskToEVL(HeaderMask, *R, *EVL)) {
+ NewR->insertBefore(R);
+ for (auto [Old, New] :
+ zip_equal(R->definedValues(), NewR->definedValues()))
+ Old->replaceAllUsesWith(New);
+ OldRecipes.push_back(R);
+ }
+ }
+
+ // Replace remaining (HeaderMask && Mask) with vp.merge (True, Mask,
+ // False, EVL)
+ for (VPUser *U : vputils::collectUsersRecursively(HeaderMask)) {
+ VPValue *Mask;
+ if (match(U, m_LogicalAnd(m_Specific(HeaderMask), m_VPValue(Mask)))) {
+ auto *LogicalAnd = cast<VPInstruction>(U);
+ auto *Merge = new VPWidenIntrinsicRecipe(
+ Intrinsic::vp_merge, {Plan.getTrue(), Mask, Plan.getFalse(), EVL},
+ Mask->getScalarType(), {}, {}, LogicalAnd->getDebugLoc());
+ Merge->insertBefore(LogicalAnd);
+ LogicalAnd->replaceAllUsesWith(Merge);
+ OldRecipes.push_back(LogicalAnd);
+ }
+ }
+
+ // Pull out left splices from any elementwise op.
+ // binop(splice.left(poison, x, evl), live-in)
+ // -> splice.left(poison, binop(x,live-in), evl)
+ pullOutPermutations(
+ Plan,
+ [&EVL](const auto &X) {
+ return m_Intrinsic<Intrinsic::vector_splice_left>(m_Poison(), X,
+ m_Specific(EVL));
+ },
+ [&Plan, &EVL](auto *X) {
+ return new VPWidenIntrinsicRecipe(
+ Intrinsic::vector_splice_left,
+ {Plan.getPoison(X->getScalarType()), X, EVL}, X->getScalarType(),
+ {}, {}, X->getDebugLoc());
+ });
+
+ // Fold the following splice patterns:
+ // splice.right(splice.left(poison, x, evl), poison, evl) -> x
+ // vector.reverse(splice.left(poison, x, evl)) -> vp.reverse(x, true, evl)
+ // splice.right(vector.reverse(x), poison, evl) -> vp.reverse(x, true, evl)
+ for (VPUser *U : vputils::collectUsersRecursively(EVL)) {
+ auto *R = cast<VPRecipeBase>(U);
+ // Remove potentially dead left splices from the transform above.
+ if (match(U, m_Intrinsic<Intrinsic::vector_splice_left>()) &&
+ R->getVPSingleValue()->getNumUsers() == 0) {
+ OldRecipes.push_back(R);
+ continue;
+ }
+
+ VPValue *X;
+ if (match(U, m_Intrinsic<Intrinsic::vector_splice_right>(
+ m_Intrinsic<Intrinsic::vector_splice_left>(
+ m_Poison(), m_VPValue(X), m_Specific(EVL)),
+ m_Poison(), m_Specific(EVL)))) {
+ R->getVPSingleValue()->replaceAllUsesWith(X);
+ OldRecipes.push_back(R);
+ continue;
+ }
+
+ if (!match(U,
+ m_CombineOr(
+ m_Reverse(m_Intrinsic<Intrinsic::vector_splice_left>(
+ m_Poison(), m_VPValue(X), m_Specific(EVL))),
+ m_Intrinsic<Intrinsic::vector_splice_right>(
+ m_Reverse(m_VPValue(X)), m_Poison(), m_Specific(EVL)))))
+ continue;
+
+ auto *VPReverse = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_reverse, {X, Plan.getTrue(), EVL},
+ X->getScalarType(), {}, {}, R->getDebugLoc());
+ VPReverse->insertBefore(R);
+ R->getVPSingleValue()->replaceAllUsesWith(VPReverse);
+ OldRecipes.push_back(R);
+ }
+
+ for (VPRecipeBase *R : reverse(OldRecipes)) {
+ SmallVector<VPValue *> PossiblyDead(R->operands());
+ R->eraseFromParent();
+ for (VPValue *Op : PossiblyDead)
+ vputils::recursivelyDeleteDeadRecipes(Op);
+ }
+}
+
+/// After replacing the canonical IV with a EVL-based IV, fixup recipes that use
+/// VF to use the EVL instead to avoid incorrect updates on the penultimate
+/// iteration.
+static void fixupVFUsersForEVL(VPlan &Plan, VPValue &EVL) {
+ VPRegionBlock *LoopRegion = Plan.getVectorLoopRegion();
+ VPBasicBlock *Header = LoopRegion->getEntryBasicBlock();
+
+ // EVL is i32 but VF/VFxUF are IdxTy. Convert as needed.
+ VPValue *EVLAsIdx =
+ VPBuilder::getToInsertAfter(EVL.getDefiningRecipe())
+ .createScalarZExtOrTrunc(&EVL, Plan.getVF().getScalarType(),
+ EVL.getScalarType(), DebugLoc::getUnknown());
+
+ assert(all_of(Plan.getVF().users(),
+ [&Plan](VPUser *U) {
+ auto IsAllowedUser =
+ IsaPred<VPVectorEndPointerRecipe, VPScalarIVStepsRecipe,
+ VPWidenIntOrFpInductionRecipe,
+ VPWidenMemIntrinsicRecipe>;
+ if (match(U, m_Trunc(m_Specific(&Plan.getVF()))))
+ return all_of(cast<VPSingleDefRecipe>(U)->users(),
+ IsAllowedUser);
+ return IsAllowedUser(U);
+ }) &&
+ "User of VF that we can't transform to EVL.");
+ Plan.getVF().replaceUsesWithIf(EVLAsIdx, [](VPUser &U, unsigned Idx) {
+ return isa<VPWidenIntOrFpInductionRecipe, VPScalarIVStepsRecipe>(U);
+ });
+
+ assert(all_of(Plan.getVFxUF().users(),
+ match_fn(m_CombineOr(
+ m_c_Add(m_Specific(LoopRegion->getCanonicalIV()),
+ m_Specific(&Plan.getVFxUF())),
+ m_Isa<VPWidenPointerInductionRecipe>()))) &&
+ "Only users of VFxUF should be VPWidenPointerInductionRecipe and the "
+ "increment of the canonical induction.");
+ Plan.getVFxUF().replaceUsesWithIf(EVLAsIdx, [](VPUser &U, unsigned Idx) {
+ // Only replace uses in VPWidenPointerInductionRecipe; The increment of the
+ // canonical induction must not be updated.
+ return isa<VPWidenPointerInductionRecipe>(U);
+ });
+
+ // Create a scalar phi to track the previous EVL if fixed-order recurrence is
+ // contained.
+ bool ContainsFORs =
+ any_of(Header->phis(), IsaPred<VPFirstOrderRecurrencePHIRecipe>);
+ if (ContainsFORs) {
+ // TODO: Use VPInstruction::ExplicitVectorLength to get maximum EVL.
+ VPValue *MaxEVL = &Plan.getVF();
+ // Emit VPScalarCastRecipe in preheader if VF is not a 32 bits integer.
+ VPBuilder Builder(LoopRegion->getPreheaderVPBB());
+ MaxEVL = Builder.createScalarZExtOrTrunc(
+ MaxEVL, Type::getInt32Ty(Plan.getContext()), MaxEVL->getScalarType(),
+ DebugLoc::getUnknown());
+
+ Builder.setInsertPoint(Header, Header->getFirstNonPhi());
+ VPValue *PrevEVL = Builder.createScalarPhi(
+ {MaxEVL, &EVL}, DebugLoc::getUnknown(), "prev.evl");
+
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getVectorLoopRegion()->getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ VPValue *V1, *V2;
+ if (!match(&R,
+ m_VPInstruction<VPInstruction::FirstOrderRecurrenceSplice>(
+ m_VPValue(V1), m_VPValue(V2))))
+ continue;
+ VPValue *Imm = Plan.getOrAddLiveIn(
+ ConstantInt::getSigned(Type::getInt32Ty(Plan.getContext()), -1));
+ VPWidenIntrinsicRecipe *VPSplice = new VPWidenIntrinsicRecipe(
+ Intrinsic::experimental_vp_splice,
+ {V1, V2, Imm, Plan.getTrue(), PrevEVL, &EVL},
+ R.getVPSingleValue()->getScalarType(), {}, {}, R.getDebugLoc());
+ VPSplice->insertBefore(&R);
+ R.getVPSingleValue()->replaceAllUsesWith(VPSplice);
+ }
+ }
+ }
+
+ VPValue *HeaderMask = LoopRegion->getHeaderMask();
+ if (!HeaderMask)
+ return;
+
+ // Ensure that any reduction that uses a select to mask off tail lanes does so
+ // in the vector loop, not the middle block, since EVL tail folding can have
+ // tail elements in the penultimate iteration.
+ assert(all_of(*Plan.getMiddleBlock(), [&Plan, HeaderMask](VPRecipeBase &R) {
+ if (match(&R, m_ComputeReductionResult(m_Select(m_Specific(HeaderMask),
+ m_VPValue(), m_VPValue()))))
+ return R.getOperand(0)->getDefiningRecipe()->getRegion() ==
+ Plan.getVectorLoopRegion();
+ return true;
+ }));
+
+ // Replace the abstract header mask with a mask equivalent to predicating by
+ // EVL: icmp ult step-vector, EVL
+ VPRecipeBase *EVLR = EVL.getDefiningRecipe();
+ VPBuilder Builder(EVLR->getParent(), std::next(EVLR->getIterator()));
+ Type *EVLType = EVL.getScalarType();
+ VPValue *EVLMask = Builder.createICmp(
+ CmpInst::ICMP_ULT,
+ Builder.createNaryOp(VPInstruction::StepVector, {}, EVLType), &EVL);
+ HeaderMask->replaceAllUsesWith(EVLMask);
+}
+
+/// Converts a tail folded vector loop region to step by
+/// VPInstruction::ExplicitVectorLength elements instead of VF elements each
+/// iteration.
+///
+/// - Add a VPCurrentIterationPHIRecipe and related recipes to \p Plan and
+/// replaces all uses of the canonical IV except for the canonical IV
+/// increment with a VPCurrentIterationPHIRecipe. The canonical IV is used
+/// only for loop iterations co...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
Contributor
|
Does this need rebasing now? |
fhahn
force-pushed
the
vplan-move-widening
branch
from
July 27, 2026 13:17
27bd32c to
ea1d737
Compare
fhahn
commented
Jul 27, 2026
fhahn
left a comment
fhahn
left a comment
Contributor
Author
There was a problem hiding this comment.
Does this need rebasing now?
Yep, should be updated now that all dependencies have landed, thanks
david-arm
reviewed
Jul 27, 2026
| } | ||
| } | ||
|
|
||
| void VPlanTransforms::narrowToSingleScalarRecipes(VPlan &Plan) { |
Contributor
There was a problem hiding this comment.
Surely this is a narrowing action?
Contributor
Author
There was a problem hiding this comment.
Right, arguably narrowing is deciding not to widen. I think this should be consistent with the existing use of the terminology in LV, e.g. here https://github.com/llvm/llvm-project/blob/main/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp#L875.
Not sure if there's a good more generic alternative without making to name of the file substantially longer?
Contributor
There was a problem hiding this comment.
It looks like VPRecipeBuilder::tryToWidenMemory in LoopVectorize.cpp also calls Builder.createWidenLoad. Given that tryToWidenMemory is also making widening decisions I assume this will be moved here too eventually? Otherwise the code will remain as fragmented as before. If the intention is to leave some widening decisions outside of this file, it would be good to add comments in the file header explaining what the rules are for which bits of code should live here. For example, if this file is strictly for vplan recipe to recipe transformations then it would be good to explain why it cannot live in VPlanTransforms.cpp.
Contributor
Author
There was a problem hiding this comment.
. If the intention is to leave some widening decisions outside of this file, it would be good to add comments in the file header explaining what the rules are for which bits of code should live here. For example, if this file is strictly for vplan recipe to recipe transformations then it would be good to explain why it cannot live in VPlanTransforms.cpp.
The intention is merely to split up VPlanTransforms.cpp to reduce single file compile-time as it used to be among the top 10 longest compilation units for a full clang build.
VPRecipeBuilder::tryToWidenMemory cannot be moved because it is part of the legacy decision making instead of making the decision VPlan-based (it is using legacy cost model which is only visible in LoopVectorie.cpp), which is gradually being replaced by the VPlan-based code.
artagnon
reviewed
Jul 28, 2026
artagnon
left a comment
artagnon
left a comment
Contributor
There was a problem hiding this comment.
Hm, not sure about this patch, because widening decisions are all over the place really: take the simple example of how legalizeAndOptimizeInductions narrows to single scalars?
Following up to, move transformations making decisions about widening to VPlanWideningDecisions.cpp. This moves tryToConvertVPInstructionsToVPRecipes, convertToAbstractRecipes, createPartialReductions, makeMemOpWideningDecisions, makeScalarizationDecisions, makeCallWideningDecisions and convertToStridedAccesses together with their static helpers.
…rrowInterleaveGroups.
fhahn
force-pushed
the
vplan-move-widening
branch
from
July 28, 2026 13:22
ea1d737 to
8cdec1c
Compare
Contributor
Author
Originally I only planned to move memory widening decision logic, but I don't think there's any reason we cannot consolidate most (all?) relevant VPlan code in a single file? Moved createInterleaveGroups, legalizeAndOptimizeInductions and narrowInterleaveGroups as well. |
lukel97
approved these changes
Jul 28, 2026
lukel97
left a comment
lukel97
left a comment
Contributor
There was a problem hiding this comment.
LGTM, I think the most important thing here is to trim down VPlanTransforms. "Widening" to me kind of means any decision to change the width of a recipe, either making it narrower or wider.
ayalz
reviewed
Aug 1, 2026
| VPlanUnroll.cpp | ||
| VPlanVerifier.cpp | ||
| VPlanUtils.cpp | ||
| VPlanWideningDecisions.cpp |
There was a problem hiding this comment.
Suggested change
| VPlanWideningDecisions.cpp | |
| VPlanWideningTransforms.cpp |
as this involves both making decisions and taking them - consistent with VPlanEVLTransforms.cpp, or perhaps VPlanWidening.cpp suffices - consistent with VPlanLowering.cpp and VPlanUnroll.cpp?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following up to #209885,
move transformations making decisions about widening to
VPlanWideningDecisions.cpp.
This moves
tryToConvertVPInstructionsToVPRecipes, convertToAbstractRecipes,
createPartialReductions, makeMemOpWideningDecisions, makeScalarizationDecisions,
makeCallWideningDecisions and convertToStridedAccesses together with their
static helpers.
Depends on #209883
Depends on #209885