CodeGen: Add LibcallLoweringInfo analysis pass #168622
Open
+303
−61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The libcall lowering decisions should be program dependent, depending on the current module's RuntimeLibcallInfo. We need another related analysis derived from that plus the current function's subtarget to provide concrete lowering decisions. This takes on a somewhat unusual form. It's a Module analysis, with a lookup keyed on the subtarget. This is a separate module analysis from RuntimeLibraryAnalysis to avoid that depending on codegen. It's not a function pass to avoid depending on any particular function, to avoid repeated subtarget map lookups in most of the use passes, and to avoid any recomputation in the common case of one subtarget (and keeps it reusable across repeated compilations). This also switches ExpandFp and PreISelIntrinsicLowering as a sample function and module pass. Note this is not yet wired up to SelectionDAG, which is still using the LibcallLoweringInfo constructed inside of TargetLowering.
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Nov 18, 2025
llvmbot
added
clang
Member
|
@llvm/pr-subscribers-backend-spir-v @llvm/pr-subscribers-backend-amdgpu Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-backend-loongarch Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-backend-x86 Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-backend-risc-v Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-backend-powerpc Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-llvm-analysis Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-llvm-transforms Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Member
|
@llvm/pr-subscribers-backend-aarch64 Author: Matt Arsenault (arsenm) ChangesThe libcall lowering decisions should be program dependent, This takes on a somewhat unusual form. It's a Module analysis, This also switches ExpandFp and PreISelIntrinsicLowering as Patch is 46.21 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168622.diff 36 Files Affected:
diff --git a/clang/lib/CodeGen/BackendUtil.cpp b/clang/lib/CodeGen/BackendUtil.cpp
index 6f63e6470270e..fe700607cc2d7 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -19,6 +19,7 @@
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Bitcode/BitcodeReader.h"
@@ -667,6 +668,11 @@ bool EmitAssemblyHelper::AddEmitPasses(legacy::PassManager &CodeGenPasses,
llvm::driver::createTLII(TargetTriple, CodeGenOpts.getVecLib()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(*TLII));
+ const llvm::TargetOptions &Options = TM->Options;
+ CodeGenPasses.add(new RuntimeLibraryInfoWrapper(
+ TargetTriple, Options.ExceptionModel, Options.FloatABIType,
+ Options.EABIVersion, Options.MCOptions.ABIName, Options.VecLib));
+
// Normal mode, emit a .s or .o file by running the code generator. Note,
// this also adds codegenerator level optimization passes.
CodeGenFileType CGFT = getCodeGenFileType(Action);
diff --git a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
index 28a2ec47f81ad..38fa83b58cbef 100644
--- a/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
+++ b/llvm/include/llvm/Analysis/RuntimeLibcallInfo.h
@@ -22,7 +22,12 @@ class LLVM_ABI RuntimeLibraryAnalysis
RuntimeLibraryAnalysis() = default;
RuntimeLibraryAnalysis(RTLIB::RuntimeLibcallsInfo &&BaselineInfoImpl)
: LibcallsInfo(std::move(BaselineInfoImpl)) {}
- explicit RuntimeLibraryAnalysis(const Triple &T) : LibcallsInfo(T) {}
+ RuntimeLibraryAnalysis(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
LLVM_ABI RTLIB::RuntimeLibcallsInfo run(const Module &M,
ModuleAnalysisManager &);
@@ -41,8 +46,12 @@ class LLVM_ABI RuntimeLibraryInfoWrapper : public ImmutablePass {
public:
static char ID;
RuntimeLibraryInfoWrapper();
- explicit RuntimeLibraryInfoWrapper(const Triple &T);
- explicit RuntimeLibraryInfoWrapper(const RTLIB::RuntimeLibcallsInfo &RTLCI);
+ RuntimeLibraryInfoWrapper(
+ const Triple &TT,
+ ExceptionHandling ExceptionModel = ExceptionHandling::None,
+ FloatABI::ABIType FloatABI = FloatABI::Default,
+ EABI EABIVersion = EABI::Default, StringRef ABIName = "",
+ VectorLibrary VecLib = VectorLibrary::NoLibrary);
const RTLIB::RuntimeLibcallsInfo &getRTLCI(const Module &M) {
ModuleAnalysisManager DummyMAM;
diff --git a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
index 8624fd2403a12..13225c072cf78 100644
--- a/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
+++ b/llvm/include/llvm/CodeGen/LibcallLoweringInfo.h
@@ -9,12 +9,16 @@
#ifndef LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
#define LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
+#include "llvm/ADT/DenseMap.h"
#include "llvm/IR/RuntimeLibcalls.h"
+#include "llvm/Pass.h"
namespace llvm {
class TargetSubtargetInfo;
+class TargetMachine;
+/// Tracks which library functions to use for a particular subtarget.
class LibcallLoweringInfo {
private:
const RTLIB::RuntimeLibcallsInfo &RTLCI;
@@ -73,6 +77,70 @@ class LibcallLoweringInfo {
}
};
+/// Record a mapping from subtarget to LibcallLoweringInfo.
+class LibcallLoweringModuleAnalysisResult {
+private:
+ using LibcallLoweringMap =
+ DenseMap<const TargetSubtargetInfo *, LibcallLoweringInfo>;
+ mutable LibcallLoweringMap LoweringMap;
+ const RTLIB::RuntimeLibcallsInfo *RTLCI = nullptr;
+
+public:
+ LibcallLoweringModuleAnalysisResult() = default;
+ LibcallLoweringModuleAnalysisResult(RTLIB::RuntimeLibcallsInfo &RTLCI)
+ : RTLCI(&RTLCI) {}
+
+ void init(const RTLIB::RuntimeLibcallsInfo *RT) { RTLCI = RT; }
+
+ void clear() {
+ RTLCI = nullptr;
+ LoweringMap.clear();
+ }
+
+ LLVM_ABI bool invalidate(Module &, const PreservedAnalyses &,
+ ModuleAnalysisManager::Invalidator &);
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return LoweringMap.try_emplace(&Subtarget, *RTLCI, Subtarget).first->second;
+ }
+};
+
+class LibcallLoweringModuleAnalysis
+ : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> {
+private:
+ friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>;
+ LLVM_ABI static AnalysisKey Key;
+
+ LibcallLoweringModuleAnalysisResult LibcallLoweringMap;
+
+public:
+ using Result = LibcallLoweringModuleAnalysisResult;
+
+ LLVM_ABI Result run(Module &M, ModuleAnalysisManager &);
+};
+
+class LLVM_ABI LibcallLoweringInfoWrapper : public ImmutablePass {
+ LibcallLoweringModuleAnalysisResult Result;
+
+public:
+ static char ID;
+ LibcallLoweringInfoWrapper();
+
+ const LibcallLoweringInfo &
+ getLibcallLowering(const TargetSubtargetInfo &Subtarget) const {
+ return Result.getLibcallLowering(Subtarget);
+ }
+
+ const LibcallLoweringModuleAnalysisResult &getResult() const {
+ return Result;
+ }
+
+ bool doInitialization(Module &M) override;
+ void getAnalysisUsage(AnalysisUsage &AU) const override;
+ void releaseMemory() override;
+};
+
} // end namespace llvm
#endif // LLVM_CODEGEN_LIBCALLLOWERINGINFO_H
diff --git a/llvm/include/llvm/InitializePasses.h b/llvm/include/llvm/InitializePasses.h
index 10a4d8525a9e8..c718e29b99ff4 100644
--- a/llvm/include/llvm/InitializePasses.h
+++ b/llvm/include/llvm/InitializePasses.h
@@ -133,6 +133,7 @@ LLVM_ABI void initializeGlobalMergeFuncPassWrapperPass(PassRegistry &);
LLVM_ABI void initializeGlobalMergePass(PassRegistry &);
LLVM_ABI void initializeGlobalsAAWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeHardwareLoopsLegacyPass(PassRegistry &);
+LLVM_ABI void initializeLibcallLoweringInfoWrapperPass(PassRegistry &);
LLVM_ABI void initializeMIRProfileLoaderPassPass(PassRegistry &);
LLVM_ABI void initializeIRSimilarityIdentifierWrapperPassPass(PassRegistry &);
LLVM_ABI void initializeIRTranslatorPass(PassRegistry &);
diff --git a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
index 9ea789a4ee45a..1c5a1cc75b7bd 100644
--- a/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
+++ b/llvm/lib/Analysis/RuntimeLibcallInfo.cpp
@@ -13,6 +13,15 @@ using namespace llvm;
AnalysisKey RuntimeLibraryAnalysis::Key;
+RuntimeLibraryAnalysis::RuntimeLibraryAnalysis(const Triple &TT,
+ ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI,
+ EABI EABIVersion,
+ StringRef ABIName,
+ VectorLibrary VecLib)
+ : LibcallsInfo(std::in_place, TT, ExceptionModel, FloatABI, EABIVersion,
+ ABIName, VecLib) {}
+
RTLIB::RuntimeLibcallsInfo
RuntimeLibraryAnalysis::run(const Module &M, ModuleAnalysisManager &) {
if (!LibcallsInfo)
@@ -26,6 +35,13 @@ INITIALIZE_PASS(RuntimeLibraryInfoWrapper, "runtime-library-info",
RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper()
: ImmutablePass(ID), RTLA(RTLIB::RuntimeLibcallsInfo(Triple())) {}
+RuntimeLibraryInfoWrapper::RuntimeLibraryInfoWrapper(
+ const Triple &TT, ExceptionHandling ExceptionModel,
+ FloatABI::ABIType FloatABI, EABI EABIVersion, StringRef ABIName,
+ VectorLibrary VecLib)
+ : ImmutablePass(ID), RTLCI(std::in_place, TT, ExceptionModel, FloatABI,
+ EABIVersion, ABIName, VecLib) {}
+
char RuntimeLibraryInfoWrapper::ID = 0;
ModulePass *llvm::createRuntimeLibraryInfoWrapperPass() {
diff --git a/llvm/lib/CodeGen/CodeGen.cpp b/llvm/lib/CodeGen/CodeGen.cpp
index 9795a0b707fd3..fe293c63fa762 100644
--- a/llvm/lib/CodeGen/CodeGen.cpp
+++ b/llvm/lib/CodeGen/CodeGen.cpp
@@ -57,6 +57,7 @@ void llvm::initializeCodeGen(PassRegistry &Registry) {
initializeInterleavedLoadCombinePass(Registry);
initializeInterleavedAccessPass(Registry);
initializeJMCInstrumenterPass(Registry);
+ initializeLibcallLoweringInfoWrapperPass(Registry);
initializeLiveDebugValuesLegacyPass(Registry);
initializeLiveDebugVariablesWrapperLegacyPass(Registry);
initializeLiveIntervalsWrapperPassPass(Registry);
diff --git a/llvm/lib/CodeGen/ExpandFp.cpp b/llvm/lib/CodeGen/ExpandFp.cpp
index f44eb227133ae..8ef926c2d2a34 100644
--- a/llvm/lib/CodeGen/ExpandFp.cpp
+++ b/llvm/lib/CodeGen/ExpandFp.cpp
@@ -991,7 +991,7 @@ static void addToWorklist(Instruction &I,
}
static bool runImpl(Function &F, const TargetLowering &TLI,
- AssumptionCache *AC) {
+ const LibcallLoweringInfo &Libcalls, AssumptionCache *AC) {
SmallVector<Instruction *, 4> Worklist;
unsigned MaxLegalFpConvertBitWidth =
@@ -1090,12 +1090,17 @@ class ExpandFpLegacyPass : public FunctionPass {
bool runOnFunction(Function &F) override {
auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- auto *TLI = TM->getSubtargetImpl(F)->getTargetLowering();
+ const TargetSubtargetInfo *Subtarget = TM->getSubtargetImpl(F);
+ auto *TLI = Subtarget->getTargetLowering();
AssumptionCache *AC = nullptr;
+ const LibcallLoweringInfo &Libcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getLibcallLowering(
+ *Subtarget);
+
if (OptLevel != CodeGenOptLevel::None && !F.hasOptNone())
AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
- return runImpl(F, *TLI, AC);
+ return runImpl(F, *TLI, Libcalls, AC);
}
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -1104,6 +1109,7 @@ class ExpandFpLegacyPass : public FunctionPass {
AU.addRequired<AssumptionCacheTracker>();
AU.addPreserved<AAResultsWrapperPass>();
AU.addPreserved<GlobalsAAWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
}
};
} // namespace
@@ -1126,8 +1132,23 @@ PreservedAnalyses ExpandFpPass::run(Function &F, FunctionAnalysisManager &FAM) {
AssumptionCache *AC = nullptr;
if (OptLevel != CodeGenOptLevel::None)
AC = &FAM.getResult<AssumptionAnalysis>(F);
- return runImpl(F, TLI, AC) ? PreservedAnalyses::none()
- : PreservedAnalyses::all();
+
+ auto &MAMProxy = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
+
+ const LibcallLoweringModuleAnalysisResult *LibcallLowering =
+ MAMProxy.getCachedResult<LibcallLoweringModuleAnalysis>(*F.getParent());
+
+ if (!LibcallLowering) {
+ F.getContext().emitError("'" + LibcallLoweringModuleAnalysis::name() +
+ "' analysis required");
+ return PreservedAnalyses::all();
+ }
+
+ const LibcallLoweringInfo &Libcalls =
+ LibcallLowering->getLibcallLowering(*STI);
+
+ return runImpl(F, TLI, Libcalls, AC) ? PreservedAnalyses::none()
+ : PreservedAnalyses::all();
}
char ExpandFpLegacyPass::ID = 0;
diff --git a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
index 6f3607e8db824..0d54fac2422e2 100644
--- a/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
+++ b/llvm/lib/CodeGen/LibcallLoweringInfo.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/LibcallLoweringInfo.h"
+#include "llvm/Analysis/RuntimeLibcallInfo.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Target/TargetMachine.h"
using namespace llvm;
@@ -28,3 +31,42 @@ LibcallLoweringInfo::LibcallLoweringInfo(
Subtarget.initLibcallLoweringInfo(*this);
}
+
+AnalysisKey LibcallLoweringModuleAnalysis::Key;
+
+bool LibcallLoweringModuleAnalysisResult::invalidate(
+ Module &, const PreservedAnalyses &PA,
+ ModuleAnalysisManager::Invalidator &) {
+ // Passes that change the runtime libcall set must explicitly invalidate this
+ // pass.
+ auto PAC = PA.getChecker<LibcallLoweringModuleAnalysis>();
+ return !PAC.preservedWhenStateless();
+}
+
+LibcallLoweringModuleAnalysisResult
+LibcallLoweringModuleAnalysis::run(Module &M, ModuleAnalysisManager &MAM) {
+ LibcallLoweringMap.init(&MAM.getResult<RuntimeLibraryAnalysis>(M));
+ return LibcallLoweringMap;
+}
+
+INITIALIZE_PASS_BEGIN(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
+INITIALIZE_PASS_END(LibcallLoweringInfoWrapper, "libcall-lowering-info",
+ "Library Function Lowering Analysis", false, true)
+
+char LibcallLoweringInfoWrapper::ID = 0;
+
+LibcallLoweringInfoWrapper::LibcallLoweringInfoWrapper() : ImmutablePass(ID) {}
+
+bool LibcallLoweringInfoWrapper::doInitialization(Module &M) {
+ Result.init(&getAnalysis<RuntimeLibraryInfoWrapper>().getRTLCI(M));
+ return false;
+}
+
+void LibcallLoweringInfoWrapper::getAnalysisUsage(AnalysisUsage &AU) const {
+ AU.addRequired<RuntimeLibraryInfoWrapper>();
+ AU.setPreservesAll();
+}
+
+void LibcallLoweringInfoWrapper::releaseMemory() { Result.clear(); }
diff --git a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
index d738dc4eea36d..72c3c566163e2 100644
--- a/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
+++ b/llvm/lib/CodeGen/PreISelIntrinsicLowering.cpp
@@ -17,6 +17,7 @@
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/CodeGen/ExpandVectorPredication.h"
+#include "llvm/CodeGen/LibcallLoweringInfo.h"
#include "llvm/CodeGen/Passes.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetPassConfig.h"
@@ -51,6 +52,7 @@ namespace {
struct PreISelIntrinsicLowering {
const TargetMachine *TM;
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls;
const function_ref<TargetTransformInfo &(Function &)> LookupTTI;
const function_ref<TargetLibraryInfo &(Function &)> LookupTLI;
@@ -61,11 +63,13 @@ struct PreISelIntrinsicLowering {
explicit PreISelIntrinsicLowering(
const TargetMachine *TM_,
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls_,
function_ref<TargetTransformInfo &(Function &)> LookupTTI_,
function_ref<TargetLibraryInfo &(Function &)> LookupTLI_,
bool UseMemIntrinsicLibFunc_ = true)
- : TM(TM_), LookupTTI(LookupTTI_), LookupTLI(LookupTLI_),
- UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {}
+ : TM(TM_), ModuleLibcalls(ModuleLibcalls_), LookupTTI(LookupTTI_),
+ LookupTLI(LookupTLI_), UseMemIntrinsicLibFunc(UseMemIntrinsicLibFunc_) {
+ }
static bool shouldExpandMemIntrinsicWithSize(Value *Size,
const TargetTransformInfo &TTI);
@@ -230,21 +234,26 @@ bool PreISelIntrinsicLowering::shouldExpandMemIntrinsicWithSize(
return SizeVal > Threshold || Threshold == 0;
}
-static bool canEmitLibcall(const TargetMachine *TM, Function *F,
- RTLIB::Libcall LC) {
+static bool
+canEmitLibcall(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F, RTLIB::Libcall LC) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getLibcallName(LC) != nullptr;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getLibcallImpl(LC) != RTLIB::Unsupported;
}
-static bool canEmitMemcpy(const TargetMachine *TM, Function *F) {
+static bool
+canEmitMemcpy(const LibcallLoweringModuleAnalysisResult &ModuleLowering,
+ const TargetMachine *TM, Function *F) {
// TODO: Should this consider the address space of the memcpy?
if (!TM)
return true;
- const TargetLowering *TLI = TM->getSubtargetImpl(*F)->getTargetLowering();
- return TLI->getMemcpyImpl() != RTLIB::Unsupported;
+ const LibcallLoweringInfo &Lowering =
+ ModuleLowering.getLibcallLowering(*TM->getSubtargetImpl(*F));
+ return Lowering.getMemcpyImpl() != RTLIB::Unsupported;
}
// Return a value appropriate for use with the memset_pattern16 libcall, if
@@ -317,7 +326,8 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
Function *ParentFunc = Memcpy->getFunction();
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memcpy->getLength(), TTI)) {
- if (UseMemIntrinsicLibFunc && canEmitMemcpy(TM, ParentFunc))
+ if (UseMemIntrinsicLibFunc &&
+ canEmitMemcpy(ModuleLibcalls, TM, ParentFunc))
break;
// TODO: For optsize, emit the loop into a separate function
@@ -349,7 +359,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memmove->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMMOVE))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMMOVE))
break;
if (expandMemMoveAsLoop(Memmove, TTI)) {
@@ -366,7 +376,7 @@ bool PreISelIntrinsicLowering::expandMemIntrinsicUses(
const TargetTransformInfo &TTI = LookupTTI(*ParentFunc);
if (shouldExpandMemIntrinsicWithSize(Memset->getLength(), TTI)) {
if (UseMemIntrinsicLibFunc &&
- canEmitLibcall(TM, ParentFunc, RTLIB::MEMSET))
+ canEmitLibcall(ModuleLibcalls, TM, ParentFunc, RTLIB::MEMSET))
break;
expandMemSetAsLoop(Memset);
@@ -619,10 +629,14 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
void getAnalysisUsage(AnalysisUsage &AU) const override {
AU.addRequired<TargetTransformInfoWrapperPass>();
AU.addRequired<TargetLibraryInfoWrapperPass>();
+ AU.addRequired<LibcallLoweringInfoWrapper>();
AU.addRequired<TargetPassConfig>();
}
bool runOnModule(Module &M) override {
+ const LibcallLoweringModuleAnalysisResult &ModuleLibcalls =
+ getAnalysis<LibcallLoweringInfoWrapper>().getResult();
+
auto LookupTTI = [this](Function &F) -> TargetTransformInfo & {
return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
};
@@ -631,7 +645,7 @@ class PreISelIntrinsicLoweringLegacyPass : public ModulePass {
};
const auto *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
- PreISelIntrinsicLowering Lowering(TM, LookupTTI, LookupTLI);
+ PreISelIntrinsicLowering Lowering(TM, ModuleLibcalls, LookupTTI, LookupTLI);
return Lowering.lowerIntrinsics(M);
}
};
@@ -643,6 +657,8 @@ char PreISelIntrinsicLoweringLegacyPass::ID;
INITIALIZE_PASS_BEGIN(PreISelIntrinsicLoweringLegacyPass,
"pre-isel-intrinsic-lowering",
"Pre-ISel Intrinsic Lowering", false, false)
+INITIALIZE_PASS_DEPENDENCY(LibcallLoweringInfoWrapper)
+INITIALIZE_PASS_DEPENDENCY(RuntimeLibraryInfoWrapper)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
@@ -654,9 +670,12 @@ ModulePass *llvm::createPreISelIntrinsicLoweringPass() {
return new PreISelIntrinsicLoweringLegacyPass();
}
-PreservedAnalyses PreISelIntrinsicLoweringPass::run(Module &M,
- ModuleAnalysisManager &AM) {
- auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
+PreservedAnalyses
+PreISelIntrinsicLoweringPass::run(Module &M, ModuleAnalysisManager &MAM) {
+ const LibcallLoweringModuleAnalysisResult &LibcallLowering =
+ MAM.getResult<LibcallLoweringModuleAnalysis>(M);
+
+ auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
auto LookupTTI = [&FAM](Function &F) -> TargetTransformInfo & {
return FAM.getResult<TargetIRAnalysis>(F);
@@ ...
[truncated]
|
Contributor
|
What happens if there are two functions, one with soft-fp, the other with hard-fp attributes, and the target has different libcall names for the two modes? Is it forbidden, and if so, is it diagnosed? |
Contributor
Author
The targets that do this are already modifying the subtarget construction based on the soft-float attribute (e.g., arm) |
| ExceptionHandling ExceptionModel = ExceptionHandling::None, | ||
| FloatABI::ABIType FloatABI = FloatABI::Default, | ||
| EABI EABIVersion = EABI::Default, StringRef ABIName = "", | ||
| VectorLibrary VecLib = VectorLibrary::NoLibrary); |
Contributor
There was a problem hiding this comment.
Are the default values required? AFAICS they are always explicitly specified
| AU.addRequired<AssumptionCacheTracker>(); | ||
| AU.addPreserved<AAResultsWrapperPass>(); | ||
| AU.addPreserved<GlobalsAAWrapperPass>(); | ||
| AU.addRequired<LibcallLoweringInfoWrapper>(); |
Contributor
There was a problem hiding this comment.
Should it be added as INITIALIZE_PASS_DEPENDENCY as well? I still haven't figured out what's the difference between the macro and addRequired() call.
| : public AnalysisInfoMixin<LibcallLoweringModuleAnalysis> { | ||
| private: | ||
| friend AnalysisInfoMixin<LibcallLoweringModuleAnalysis>; | ||
| LLVM_ABI static AnalysisKey Key; |
Contributor
There was a problem hiding this comment.
private members shouldn't be LLVM_ABI
🐧 Linux x64 Test Results
Failed Tests(click on a test name to see its output) lldlld.wasm/lto/stub-library-libcall.sIf these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The libcall lowering decisions should be program dependent,
depending on the current module's RuntimeLibcallInfo. We need
another related analysis derived from that plus the current
function's subtarget to provide concrete lowering decisions.
This takes on a somewhat unusual form. It's a Module analysis,
with a lookup keyed on the subtarget. This is a separate module
analysis from RuntimeLibraryAnalysis to avoid that depending on
codegen. It's not a function pass to avoid depending on any
particular function, to avoid repeated subtarget map lookups in
most of the use passes, and to avoid any recomputation in the
common case of one subtarget (and keeps it reusable across
repeated compilations).
This also switches ExpandFp and PreISelIntrinsicLowering as
a sample function and module pass. Note this is not yet wired
up to SelectionDAG, which is still using the LibcallLoweringInfo
constructed inside of TargetLowering.