[TTI] Introduce utilities for target costing of build & explode vector [NFC] #85455
+52
−50
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce utilities for costing build vector and explode vector operations inside the TTI target implementation logic. As can be seen these are by far the most common operations actually performed. In case the goal isn't clear here, I plan to eliminate getScalarizationOverhead from the TTI interface layer. All of our targets cost a combined insert and extract as equivalent to a explode vector followed by a build vector so the combined interface can be killed off. This is the inverse of llvm#85421. Once both patches land, only the actual meat of the change remains. One subtlety here - we have to be very careful to make sure we're calling the directly analogous cover function. We've got a base class and subclass involved here, and it's important at times whether we call a method on the subclass or the base class. This is harder to follow since we have multiple getScalarizationOverhead variants with different signatures - most of which only exist on the base class, but some (not all) of which proxy back to the sub-class.
|
@llvm/pr-subscribers-backend-arm @llvm/pr-subscribers-backend-x86 Author: Philip Reames (preames) ChangesIntroduce utilities for costing build vector and explode vector operations inside the TTI target implementation logic. As can be seen these are by far the most common operations actually performed. In case the goal isn't clear here, I plan to eliminate getScalarizationOverhead from the TTI interface layer. All of our targets cost a combined insert and extract as equivalent to a explode vector followed by a build vector so the combined interface can be killed off. This is the inverse of #85421. Once both patches land, only the actual meat of the change remains. One subtlety here - we have to be very careful to make sure we're calling the directly analogous cover function. We've got a base class and subclass involved here, and it's important at times whether we call a method on the subclass or the base class. This is harder to follow since we have multiple getScalarizationOverhead variants with different signatures - most of which only exist on the base class, but some (not all) of which proxy back to the sub-class. Full diff: https://github.com/llvm/llvm-project/pull/85455.diff 4 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 61f6564e8cd79b..18e0896650f0dd 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -782,6 +782,20 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return Cost;
}
+ InstructionCost getBuildVectorCost(VectorType *InTy,
+ const APInt &DemandedElts,
+ TTI::TargetCostKind CostKind) {
+ return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/ true,
+ /*Extract=*/ false, CostKind);
+ }
+
+ InstructionCost getExplodeVectorCost(VectorType *InTy,
+ const APInt &DemandedElts,
+ TTI::TargetCostKind CostKind) {
+ return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/ false,
+ /*Extract=*/ true, CostKind);
+ }
+
/// Helper wrapper for the DemandedElts variant of getScalarizationOverhead.
InstructionCost getScalarizationOverhead(VectorType *InTy, bool Insert,
bool Extract,
@@ -795,6 +809,18 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
CostKind);
}
+ InstructionCost getBuildVectorCost(VectorType *InTy,
+ TTI::TargetCostKind CostKind) {
+ return getScalarizationOverhead(InTy, /*Insert=*/ true, /*Extract=*/ false,
+ CostKind);
+ }
+
+ InstructionCost getExplodeVectorCost(VectorType *InTy,
+ TTI::TargetCostKind CostKind) {
+ return getScalarizationOverhead(InTy, /*Insert=*/ false, /*Extract=*/ true,
+ CostKind);
+ }
+
/// Estimate the overhead of scalarizing an instructions unique
/// non-constant operands. The (potentially vector) types to use for each of
/// argument are passes via Tys.
@@ -816,8 +842,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
if (!isa<Constant>(A) && UniqueOperands.insert(A).second) {
if (auto *VecTy = dyn_cast<VectorType>(Ty))
- Cost += getScalarizationOverhead(VecTy, /*Insert*/ false,
- /*Extract*/ true, CostKind);
+ Cost += getExplodeVectorCost(VecTy, CostKind);
}
}
@@ -1186,12 +1211,8 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
// that the conversion is scalarized in one way or another.
if (Opcode == Instruction::BitCast) {
// Illegal bitcasts are done by storing and loading from a stack slot.
- return (SrcVTy ? getScalarizationOverhead(SrcVTy, /*Insert*/ false,
- /*Extract*/ true, CostKind)
- : 0) +
- (DstVTy ? getScalarizationOverhead(DstVTy, /*Insert*/ true,
- /*Extract*/ false, CostKind)
- : 0);
+ return (SrcVTy ? getExplodeVectorCost(SrcVTy, CostKind) : 0) +
+ (DstVTy ? getBuildVectorCost(DstVTy, CostKind) : 0);
}
llvm_unreachable("Unhandled cast");
@@ -1254,9 +1275,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
// Return the cost of multiple scalar invocation plus the cost of
// inserting and extracting the values.
- return getScalarizationOverhead(ValVTy, /*Insert*/ true,
- /*Extract*/ false, CostKind) +
- Num * Cost;
+ return getBuildVectorCost(ValVTy, CostKind) + Num * Cost;
}
// Unknown scalar opcode.
@@ -1821,9 +1840,8 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
if (RetVF.isVector() && !RetVF.isScalable()) {
ScalarizationCost = 0;
if (!RetTy->isVoidTy())
- ScalarizationCost += getScalarizationOverhead(
- cast<VectorType>(RetTy),
- /*Insert*/ true, /*Extract*/ false, CostKind);
+ ScalarizationCost += getBuildVectorCost(
+ cast<VectorType>(RetTy), CostKind);
ScalarizationCost +=
getOperandsScalarizationOverhead(Args, ICA.getArgTypes(), CostKind);
}
@@ -1877,8 +1895,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
Type *ScalarRetTy = RetTy;
if (auto *RetVTy = dyn_cast<VectorType>(RetTy)) {
if (!SkipScalarizationCost)
- ScalarizationCost = getScalarizationOverhead(
- RetVTy, /*Insert*/ true, /*Extract*/ false, CostKind);
+ ScalarizationCost = getBuildVectorCost(RetVTy, CostKind);
ScalarCalls = std::max(ScalarCalls,
cast<FixedVectorType>(RetVTy)->getNumElements());
ScalarRetTy = RetTy->getScalarType();
@@ -1888,8 +1905,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
Type *Ty = Tys[i];
if (auto *VTy = dyn_cast<VectorType>(Ty)) {
if (!SkipScalarizationCost)
- ScalarizationCost += getScalarizationOverhead(
- VTy, /*Insert*/ false, /*Extract*/ true, CostKind);
+ ScalarizationCost += getExplodeVectorCost(VTy, CostKind);
ScalarCalls = std::max(ScalarCalls,
cast<FixedVectorType>(VTy)->getNumElements());
Ty = Ty->getScalarType();
@@ -2299,8 +2315,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
InstructionCost ScalarizationCost =
SkipScalarizationCost
? ScalarizationCostPassed
- : getScalarizationOverhead(RetVTy, /*Insert*/ true,
- /*Extract*/ false, CostKind);
+ : getBuildVectorCost(RetVTy, CostKind);
unsigned ScalarCalls = cast<FixedVectorType>(RetVTy)->getNumElements();
SmallVector<Type *, 4> ScalarTys;
@@ -2316,8 +2331,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
for (unsigned i = 0, ie = Tys.size(); i != ie; ++i) {
if (auto *VTy = dyn_cast<VectorType>(Tys[i])) {
if (!ICA.skipScalarizationCost())
- ScalarizationCost += getScalarizationOverhead(
- VTy, /*Insert*/ false, /*Extract*/ true, CostKind);
+ ScalarizationCost += getExplodeVectorCost(VTy, CostKind);
ScalarCalls = std::max(ScalarCalls,
cast<FixedVectorType>(VTy)->getNumElements());
}
@@ -2462,8 +2476,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return InstructionCost::getInvalid();
auto *VTy = cast<FixedVectorType>(Ty);
- InstructionCost ExtractCost = getScalarizationOverhead(
- VTy, /*Insert=*/false, /*Extract=*/true, CostKind);
+ InstructionCost ExtractCost = getExplodeVectorCost(VTy, CostKind);
InstructionCost ArithCost = thisT()->getArithmeticInstrCost(
Opcode, VTy->getElementType(), CostKind);
ArithCost *= VTy->getNumElements();
diff --git a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
index 3be894ad3bef2c..1e0bac8e540c2e 100644
--- a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
+++ b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
@@ -1025,10 +1025,8 @@ InstructionCost ARMTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
if (Opcode == Instruction::FCmp && !ST->hasMVEFloatOps()) {
// One scalaization insert, one scalarization extract and the cost of the
// fcmps.
- return BaseT::getScalarizationOverhead(VecValTy, /*Insert*/ false,
- /*Extract*/ true, CostKind) +
- BaseT::getScalarizationOverhead(VecCondTy, /*Insert*/ true,
- /*Extract*/ false, CostKind) +
+ return BaseT::getExplodeVectorCost(VecValTy, CostKind) +
+ BaseT::getBuildVectorCost(VecCondTy, CostKind) +
VecValTy->getNumElements() *
getCmpSelInstrCost(Opcode, ValTy->getScalarType(),
VecCondTy->getScalarType(), VecPred,
@@ -1045,8 +1043,7 @@ InstructionCost ARMTTIImpl::getCmpSelInstrCost(unsigned Opcode, Type *ValTy,
if (LT.second.isVector() && LT.second.getVectorNumElements() > 2) {
if (LT.first > 1)
return LT.first * BaseCost +
- BaseT::getScalarizationOverhead(VecCondTy, /*Insert*/ true,
- /*Extract*/ false, CostKind);
+ BaseT::getBuildVectorCost(VecCondTy, CostKind);
return BaseCost;
}
}
@@ -1599,10 +1596,8 @@ InstructionCost ARMTTIImpl::getGatherScatterOpCost(
// greatly increasing the cost.
InstructionCost ScalarCost =
NumElems * LT.first + (VariableMask ? NumElems * 5 : 0) +
- BaseT::getScalarizationOverhead(VTy, /*Insert*/ true, /*Extract*/ false,
- CostKind) +
- BaseT::getScalarizationOverhead(VTy, /*Insert*/ false, /*Extract*/ true,
- CostKind);
+ BaseT::getBuildVectorCost(VTy, CostKind) +
+ BaseT::getExplodeVectorCost(VTy, CostKind);
if (EltSize < 8 || Alignment < EltSize / 8)
return ScalarCost;
diff --git a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
index e4adb7be564952..69ed6ae8d33597 100644
--- a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
+++ b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
@@ -910,10 +910,10 @@ InstructionCost SystemZTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
(Opcode == Instruction::FPToSI || Opcode == Instruction::FPToUI))
NeedsExtracts = false;
- TotCost += getScalarizationOverhead(SrcVecTy, /*Insert*/ false,
- NeedsExtracts, CostKind);
- TotCost += getScalarizationOverhead(DstVecTy, NeedsInserts,
- /*Extract*/ false, CostKind);
+ if (NeedsExtracts)
+ TotCost += getExplodeVectorCost(SrcVecTy, CostKind);
+ if (NeedsInserts)
+ TotCost += getBuildVectorCost(DstVecTy, CostKind);
// FIXME: VF 2 for float<->i32 is currently just as expensive as for VF 4.
if (VF == 2 && SrcScalarBits == 32 && DstScalarBits == 32)
@@ -925,8 +925,7 @@ InstructionCost SystemZTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
if (Opcode == Instruction::FPTrunc) {
if (SrcScalarBits == 128) // fp128 -> double/float + inserts of elements.
return VF /*ldxbr/lexbr*/ +
- getScalarizationOverhead(DstVecTy, /*Insert*/ true,
- /*Extract*/ false, CostKind);
+ getBuildVectorCost(DstVecTy, CostKind);
else // double -> float
return VF / 2 /*vledb*/ + std::max(1U, VF / 4 /*vperm*/);
}
@@ -939,8 +938,7 @@ InstructionCost SystemZTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
return VF * 2;
}
// -> fp128. VF * lxdb/lxeb + extraction of elements.
- return VF + getScalarizationOverhead(SrcVecTy, /*Insert*/ false,
- /*Extract*/ true, CostKind);
+ return VF + getExplodeVectorCost(SrcVecTy, CostKind);
}
}
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index d336ab9d309c4e..e0ab99ae729166 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -4530,8 +4530,7 @@ X86TTIImpl::getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
// For types we can insert directly, insertion into 128-bit sub vectors is
// cheap, followed by a cheap chain of concatenations.
if (LegalVectorBitWidth <= LaneBitWidth) {
- Cost += BaseT::getScalarizationOverhead(Ty, DemandedElts, Insert,
- /*Extract*/ false, CostKind);
+ Cost += BaseT::getBuildVectorCost(Ty, DemandedElts, CostKind);
} else {
// In each 128-lane, if at least one index is demanded but not all
// indices are demanded and this 128-lane is not the first 128-lane of
@@ -4570,8 +4569,7 @@ X86TTIImpl::getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
if (!LaneEltMask.isAllOnes())
Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, std::nullopt,
CostKind, I * NumEltsPerLane, LaneTy);
- Cost += BaseT::getScalarizationOverhead(LaneTy, LaneEltMask, Insert,
- /*Extract*/ false, CostKind);
+ Cost += BaseT::getBuildVectorCost(LaneTy, LaneEltMask, CostKind);
}
APInt AffectedLanes =
@@ -4648,8 +4646,7 @@ X86TTIImpl::getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
continue;
Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, std::nullopt,
CostKind, I * NumEltsPerLane, LaneTy);
- Cost += BaseT::getScalarizationOverhead(
- LaneTy, LaneEltMask, /*Insert*/ false, Extract, CostKind);
+ Cost += BaseT::getExplodeVectorCost(LaneTy, LaneEltMask, CostKind);
}
return Cost;
@@ -4657,8 +4654,7 @@ X86TTIImpl::getScalarizationOverhead(VectorType *Ty, const APInt &DemandedElts,
}
// Fallback to default extraction.
- Cost += BaseT::getScalarizationOverhead(Ty, DemandedElts, /*Insert*/ false,
- Extract, CostKind);
+ Cost += BaseT::getExplodeVectorCost(Ty, DemandedElts, CostKind);
}
return Cost;
|
You can test this locally with the following command:git-clang-format --diff bb9ca8afc0bd5a1abb8ef3d0f5c8e4928c2f7f60 4628b586b398b863bb78f5acde48fcb719f5d41b -- llvm/include/llvm/CodeGen/BasicTTIImpl.h llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp llvm/lib/Target/X86/X86TargetTransformInfo.cppView the diff from clang-format here.diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 18e0896650..310be56c77 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -785,15 +785,15 @@ public:
InstructionCost getBuildVectorCost(VectorType *InTy,
const APInt &DemandedElts,
TTI::TargetCostKind CostKind) {
- return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/ true,
- /*Extract=*/ false, CostKind);
+ return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/true,
+ /*Extract=*/false, CostKind);
}
InstructionCost getExplodeVectorCost(VectorType *InTy,
const APInt &DemandedElts,
TTI::TargetCostKind CostKind) {
- return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/ false,
- /*Extract=*/ true, CostKind);
+ return getScalarizationOverhead(InTy, DemandedElts, /*Insert=*/false,
+ /*Extract=*/true, CostKind);
}
/// Helper wrapper for the DemandedElts variant of getScalarizationOverhead.
@@ -811,13 +811,13 @@ public:
InstructionCost getBuildVectorCost(VectorType *InTy,
TTI::TargetCostKind CostKind) {
- return getScalarizationOverhead(InTy, /*Insert=*/ true, /*Extract=*/ false,
+ return getScalarizationOverhead(InTy, /*Insert=*/true, /*Extract=*/false,
CostKind);
}
InstructionCost getExplodeVectorCost(VectorType *InTy,
TTI::TargetCostKind CostKind) {
- return getScalarizationOverhead(InTy, /*Insert=*/ false, /*Extract=*/ true,
+ return getScalarizationOverhead(InTy, /*Insert=*/false, /*Extract=*/true,
CostKind);
}
@@ -1840,8 +1840,8 @@ public:
if (RetVF.isVector() && !RetVF.isScalable()) {
ScalarizationCost = 0;
if (!RetTy->isVoidTy())
- ScalarizationCost += getBuildVectorCost(
- cast<VectorType>(RetTy), CostKind);
+ ScalarizationCost +=
+ getBuildVectorCost(cast<VectorType>(RetTy), CostKind);
ScalarizationCost +=
getOperandsScalarizationOverhead(Args, ICA.getArgTypes(), CostKind);
}
@@ -2313,9 +2313,8 @@ public:
return InstructionCost::getInvalid();
InstructionCost ScalarizationCost =
- SkipScalarizationCost
- ? ScalarizationCostPassed
- : getBuildVectorCost(RetVTy, CostKind);
+ SkipScalarizationCost ? ScalarizationCostPassed
+ : getBuildVectorCost(RetVTy, CostKind);
unsigned ScalarCalls = cast<FixedVectorType>(RetVTy)->getNumElements();
SmallVector<Type *, 4> ScalarTys;
diff --git a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
index 1e0bac8e54..4235b4fd79 100644
--- a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
+++ b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
@@ -1594,10 +1594,10 @@ InstructionCost ARMTTIImpl::getGatherScatterOpCost(
// elements plus the scalarization overhead. If masking is required then a lot
// of little blocks will be needed and potentially a scalarized p0 mask,
// greatly increasing the cost.
- InstructionCost ScalarCost =
- NumElems * LT.first + (VariableMask ? NumElems * 5 : 0) +
- BaseT::getBuildVectorCost(VTy, CostKind) +
- BaseT::getExplodeVectorCost(VTy, CostKind);
+ InstructionCost ScalarCost = NumElems * LT.first +
+ (VariableMask ? NumElems * 5 : 0) +
+ BaseT::getBuildVectorCost(VTy, CostKind) +
+ BaseT::getExplodeVectorCost(VTy, CostKind);
if (EltSize < 8 || Alignment < EltSize / 8)
return ScalarCost;
diff --git a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
index 69ed6ae8d3..e2e9da67d4 100644
--- a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
+++ b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
@@ -924,8 +924,7 @@ InstructionCost SystemZTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
if (Opcode == Instruction::FPTrunc) {
if (SrcScalarBits == 128) // fp128 -> double/float + inserts of elements.
- return VF /*ldxbr/lexbr*/ +
- getBuildVectorCost(DstVecTy, CostKind);
+ return VF /*ldxbr/lexbr*/ + getBuildVectorCost(DstVecTy, CostKind);
else // double -> float
return VF / 2 /*vledb*/ + std::max(1U, VF / 4 /*vperm*/);
}
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce utilities for costing build vector and explode vector operations inside the TTI target implementation logic. As can be seen these are by far the most common operations actually performed.
In case the goal isn't clear here, I plan to eliminate getScalarizationOverhead from the TTI interface layer. All of our targets cost a combined insert and extract as equivalent to a explode vector followed by a build vector so the combined interface can be killed off.
This is the inverse of #85421. Once both patches land, only the actual meat of the change remains.
One subtlety here - we have to be very careful to make sure we're calling the directly analogous cover function. We've got a base class and subclass involved here, and it's important at times whether we call a method on the subclass or the base class. This is harder to follow since we have multiple getScalarizationOverhead variants with different signatures - most of which only exist on the base class, but some (not all) of which proxy back to the sub-class.