Reapply "[Polly] Update ScopInliner for NPM (#125427)" #164601
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
An assertion failed when Polly was registering for the pass manager which assumed that there would be only Polly passes. Since the this does not need to be the case, re-apply with the assert removed.
|
@llvm/pr-subscribers-vectorizers Author: Michael Kruse (Meinersbur) ChangesAn assertion failed when Polly was registering for the pass manager which assumed that there would be only Polly passes. Since this does not need to be the case, re-apply with the assert removed. Includes a non-Polly change to trigger the premerge CI to trigger check-llvm failed for 0b9a7b8. Full diff: https://github.com/llvm/llvm-project/pull/164601.diff 10 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanValue.h b/llvm/lib/Transforms/Vectorize/VPlanValue.h
index 0678bc90ef4b5..a33391e8484f8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanValue.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanValue.h
@@ -41,10 +41,10 @@ class VPRecipeBase;
class VPInterleaveBase;
class VPPhiAccessors;
-// This is the base class of the VPlan Def/Use graph, used for modeling the data
-// flow into, within and out of the VPlan. VPValues can stand for live-ins
-// coming from the input IR and instructions which VPlan will generate if
-// executed.
+/// This is the base class of the VPlan Def/Use graph, used for modeling the data
+/// flow into, within and out of the VPlan. VPValues can stand for live-ins
+/// coming from the input IR and instructions which VPlan will generate if
+/// executed.
class LLVM_ABI_FOR_TEST VPValue {
friend class VPDef;
friend struct VPDoubleValueDef;
@@ -57,7 +57,7 @@ class LLVM_ABI_FOR_TEST VPValue {
SmallVector<VPUser *, 1> Users;
protected:
- // Hold the underlying Value, if any, attached to this VPValue.
+ /// Hold the underlying Value, if any, attached to this VPValue.
Value *UnderlyingVal;
/// Pointer to the VPDef that defines this VPValue. If it is nullptr, the
diff --git a/polly/docs/ReleaseNotes.rst b/polly/docs/ReleaseNotes.rst
index f7c9689089be2..f5ea47b69cf02 100644
--- a/polly/docs/ReleaseNotes.rst
+++ b/polly/docs/ReleaseNotes.rst
@@ -11,3 +11,5 @@ In Polly |version| the following important changes have been incorporated.
the new features that have recently been committed to our development
branch.
+ * ScopInliner has been updated for the New Pass Manager.
+
diff --git a/polly/include/polly/LinkAllPasses.h b/polly/include/polly/LinkAllPasses.h
index c3b68a74056ac..9978344c73e9f 100644
--- a/polly/include/polly/LinkAllPasses.h
+++ b/polly/include/polly/LinkAllPasses.h
@@ -119,7 +119,7 @@ struct PollyForcePassLinking {
namespace llvm {
void initializeCodePreparationPass(llvm::PassRegistry &);
-void initializeScopInlinerPass(llvm::PassRegistry &);
+void initializeScopInlinerWrapperPassPass(llvm::PassRegistry &);
void initializeScopDetectionWrapperPassPass(llvm::PassRegistry &);
void initializeScopDetectionPrinterLegacyPassPass(llvm::PassRegistry &);
void initializeScopInfoRegionPassPass(PassRegistry &);
diff --git a/polly/include/polly/ScopInliner.h b/polly/include/polly/ScopInliner.h
new file mode 100644
index 0000000000000..014667804330f
--- /dev/null
+++ b/polly/include/polly/ScopInliner.h
@@ -0,0 +1,34 @@
+//===------ ScopInliner.h ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef POLLY_POLLYINLINER_H
+#define POLLY_POLLYINLINER_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace polly {
+class ScopInlinerPass : public llvm::PassInfoMixin<ScopInlinerPass> {
+public:
+ ScopInlinerPass();
+
+ llvm::PreservedAnalyses run(llvm::LazyCallGraph::SCC &C,
+ llvm::CGSCCAnalysisManager &AM,
+ llvm::LazyCallGraph &CG,
+ llvm::CGSCCUpdateResult &UR);
+};
+
+llvm::Pass *createScopInlinerWrapperPass();
+} // namespace polly
+
+namespace llvm {
+void initializeScopInlinerWrapperPassPass(llvm::PassRegistry &);
+}
+
+#endif /* POLLY_POLLYINLINER_H */
diff --git a/polly/lib/Support/PollyPasses.def b/polly/lib/Support/PollyPasses.def
index e068f31fdb703..2c792a5867100 100644
--- a/polly/lib/Support/PollyPasses.def
+++ b/polly/lib/Support/PollyPasses.def
@@ -1,3 +1,9 @@
+#ifndef CGSCC_PASS
+#define CGSCC_PASS(NAME, CREATE_PASS, PARSER)
+#endif
+CGSCC_PASS("polly-inline", ScopInlinerPass(), parseNoOptions)
+#undef CGSCC_PASS
+
#ifndef FUNCTION_ANALYSIS
#define FUNCTION_ANALYSIS(NAME, CREATE_PASS)
#endif
diff --git a/polly/lib/Support/RegisterPasses.cpp b/polly/lib/Support/RegisterPasses.cpp
index 0420dff944f62..04f8715502c38 100644
--- a/polly/lib/Support/RegisterPasses.cpp
+++ b/polly/lib/Support/RegisterPasses.cpp
@@ -35,6 +35,7 @@
#include "polly/ScopDetection.h"
#include "polly/ScopGraphPrinter.h"
#include "polly/ScopInfo.h"
+#include "polly/ScopInliner.h"
#include "polly/Simplify.h"
#include "polly/Support/DumpFunctionPass.h"
#include "polly/Support/DumpModulePass.h"
@@ -46,10 +47,13 @@
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"
#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
#include "llvm/Support/TargetSelect.h"
#include "llvm/Transforms/IPO.h"
+using namespace llvm;
namespace cl = llvm::cl;
+using namespace polly;
using llvm::FunctionPassManager;
using llvm::OptimizationLevel;
@@ -233,7 +237,7 @@ void initializePollyPasses(llvm::PassRegistry &Registry) {
initializePollyCanonicalizePass(Registry);
initializeScopDetectionWrapperPassPass(Registry);
initializeScopDetectionPrinterLegacyPassPass(Registry);
- initializeScopInlinerPass(Registry);
+ initializeScopInlinerWrapperPassPass(Registry);
initializeScopInfoRegionPassPass(Registry);
initializeScopInfoPrinterLegacyRegionPassPass(Registry);
initializeScopInfoWrapperPassPass(Registry);
@@ -434,6 +438,16 @@ static void buildLatePollyPipeline(FunctionPassManager &PM,
false);
}
+static llvm::Expected<std::monostate> parseNoOptions(StringRef Params) {
+ if (!Params.empty())
+ return make_error<StringError>(
+ formatv("'{0}' passed to pass that does not take any options", Params)
+ .str(),
+ inconvertibleErrorCode());
+
+ return std::monostate{};
+}
+
static OwningScopAnalysisManagerFunctionProxy
createScopAnalyses(FunctionAnalysisManager &FAM,
PassInstrumentationCallbacks *PIC) {
@@ -461,6 +475,23 @@ static void registerFunctionAnalyses(FunctionAnalysisManager &FAM,
FAM.registerPass([&FAM, PIC] { return createScopAnalyses(FAM, PIC); });
}
+static llvm::Expected<bool>
+parseCGPipeline(StringRef Name, llvm::CGSCCPassManager &CGPM,
+ PassInstrumentationCallbacks *PIC,
+ ArrayRef<PassBuilder::PipelineElement> Pipeline) {
+#define CGSCC_PASS(NAME, CREATE_PASS, PARSER) \
+ if (PassBuilder::checkParametrizedPassName(Name, NAME)) { \
+ auto Params = PassBuilder::parsePassParameters(PARSER, Name, NAME); \
+ if (!Params) \
+ return Params.takeError(); \
+ CGPM.addPass(CREATE_PASS); \
+ return true; \
+ }
+#include "PollyPasses.def"
+
+ return false;
+}
+
static bool
parseFunctionPipeline(StringRef Name, FunctionPassManager &FPM,
ArrayRef<PassBuilder::PipelineElement> Pipeline) {
@@ -598,6 +629,12 @@ void registerPollyPasses(PassBuilder &PB) {
ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
return parseScopPipeline(Name, FPM, PIC, Pipeline);
});
+ PB.registerPipelineParsingCallback(
+ [PIC](StringRef Name, CGSCCPassManager &CGPM,
+ ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
+ ExitOnError Err("Unable to parse Polly call graph pass: ");
+ return Err(parseCGPipeline(Name, CGPM, PIC, Pipeline));
+ });
PB.registerParseTopLevelPipelineCallback(
[PIC](llvm::ModulePassManager &MPM,
ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
diff --git a/polly/lib/Transform/ScopInliner.cpp b/polly/lib/Transform/ScopInliner.cpp
index b78206c1e40ba..c04ba3498339e 100644
--- a/polly/lib/Transform/ScopInliner.cpp
+++ b/polly/lib/Transform/ScopInliner.cpp
@@ -13,10 +13,14 @@
//
//===----------------------------------------------------------------------===//
-#include "polly/LinkAllPasses.h"
+#include "polly/ScopInliner.h"
#include "polly/ScopDetection.h"
+#include "polly/ScopInliner.h"
#include "llvm/Analysis/CallGraph.h"
#include "llvm/Analysis/CallGraphSCCPass.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/RegionInfo.h"
+#include "llvm/IR/Dominators.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Transforms/IPO/AlwaysInliner.h"
@@ -28,13 +32,77 @@ using namespace llvm;
using namespace polly;
namespace {
-class ScopInliner final : public CallGraphSCCPass {
+
+/// Inliner implementation that works with both, LPM (using SCC_t=CallGraph) and
+/// NPM (using SCC_t=LazyCallGraph::SCC)
+template <typename SCC_t> bool runScopInlinerImpl(Function *F, SCC_t &SCC) {
+ // We do not try to inline non-trivial SCCs because this would lead to
+ // "infinite" inlining if we are not careful.
+ if (SCC.size() > 1)
+ return false;
+ assert(SCC.size() == 1 && "found empty SCC");
+
+ // If the function is a nullptr, or the function is a declaration.
+ if (!F)
+ return false;
+ if (F->isDeclaration()) {
+ POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
+ << "because it is a declaration.\n");
+ return false;
+ }
+
+ PassBuilder PB;
+ // Populate analysis managers and register Polly-specific analyses.
+ LoopAnalysisManager LAM;
+ FunctionAnalysisManager FAM;
+ CGSCCAnalysisManager CGAM;
+ ModuleAnalysisManager MAM;
+ PB.registerModuleAnalyses(MAM);
+ PB.registerCGSCCAnalyses(CGAM);
+ PB.registerFunctionAnalyses(FAM);
+ PB.registerLoopAnalyses(LAM);
+ PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+
+ auto &DT = FAM.getResult<DominatorTreeAnalysis>(*F);
+ auto &SE = FAM.getResult<ScalarEvolutionAnalysis>(*F);
+ auto &LI = FAM.getResult<LoopAnalysis>(*F);
+ auto &RI = FAM.getResult<RegionInfoAnalysis>(*F);
+ auto &AA = FAM.getResult<AAManager>(*F);
+ auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(*F);
+ ScopDetection SD(DT, SE, LI, RI, AA, ORE);
+ SD.detect(*F);
+
+ const bool HasScopAsTopLevelRegion =
+ SD.ValidRegions.contains(RI.getTopLevelRegion());
+
+ bool Changed = false;
+ if (HasScopAsTopLevelRegion) {
+ POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
+ << " has scop as top level region");
+ F->addFnAttr(llvm::Attribute::AlwaysInline);
+
+ ModulePassManager MPM;
+ MPM.addPass(AlwaysInlinerPass());
+ Module *M = F->getParent();
+ assert(M && "Function has illegal module");
+ PreservedAnalyses PA = MPM.run(*M, MAM);
+ if (!PA.areAllPreserved())
+ Changed = true;
+ } else {
+ POLLY_DEBUG(dbgs() << F->getName()
+ << " does NOT have scop as top level region\n");
+ }
+
+ return Changed;
+}
+
+class ScopInlinerWrapperPass final : public CallGraphSCCPass {
using llvm::Pass::doInitialization;
public:
static char ID;
- ScopInliner() : CallGraphSCCPass(ID) {}
+ ScopInlinerWrapperPass() : CallGraphSCCPass(ID) {}
bool doInitialization(CallGraph &CG) override {
if (!polly::PollyAllowFullFunction) {
@@ -50,60 +118,8 @@ class ScopInliner final : public CallGraphSCCPass {
}
bool runOnSCC(CallGraphSCC &SCC) override {
- // We do not try to inline non-trivial SCCs because this would lead to
- // "infinite" inlining if we are not careful.
- if (SCC.size() > 1)
- return false;
- assert(SCC.size() == 1 && "found empty SCC");
Function *F = (*SCC.begin())->getFunction();
-
- // If the function is a nullptr, or the function is a declaration.
- if (!F)
- return false;
- if (F->isDeclaration()) {
- POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
- << "because it is a declaration.\n");
- return false;
- }
-
- PassBuilder PB;
- // Populate analysis managers and register Polly-specific analyses.
- LoopAnalysisManager LAM;
- FunctionAnalysisManager FAM;
- CGSCCAnalysisManager CGAM;
- ModuleAnalysisManager MAM;
- FAM.registerPass([] { return ScopAnalysis(); });
- PB.registerModuleAnalyses(MAM);
- PB.registerCGSCCAnalyses(CGAM);
- PB.registerFunctionAnalyses(FAM);
- PB.registerLoopAnalyses(LAM);
- PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
-
- RegionInfo &RI = FAM.getResult<RegionInfoAnalysis>(*F);
- ScopDetection &SD = FAM.getResult<ScopAnalysis>(*F);
-
- const bool HasScopAsTopLevelRegion =
- SD.ValidRegions.contains(RI.getTopLevelRegion());
-
- bool Changed = false;
- if (HasScopAsTopLevelRegion) {
- POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
- << " has scop as top level region");
- F->addFnAttr(llvm::Attribute::AlwaysInline);
-
- ModulePassManager MPM;
- MPM.addPass(AlwaysInlinerPass());
- Module *M = F->getParent();
- assert(M && "Function has illegal module");
- PreservedAnalyses PA = MPM.run(*M, MAM);
- if (!PA.areAllPreserved())
- Changed = true;
- } else {
- POLLY_DEBUG(dbgs() << F->getName()
- << " does NOT have scop as top level region\n");
- }
-
- return Changed;
+ return runScopInlinerImpl(F, SCC);
};
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -111,18 +127,39 @@ class ScopInliner final : public CallGraphSCCPass {
}
};
} // namespace
-char ScopInliner::ID;
+char ScopInlinerWrapperPass::ID;
-Pass *polly::createScopInlinerPass() {
- ScopInliner *pass = new ScopInliner();
+Pass *polly::createScopInlinerWrapperPass() {
+ ScopInlinerWrapperPass *pass = new ScopInlinerWrapperPass();
return pass;
}
INITIALIZE_PASS_BEGIN(
- ScopInliner, "polly-scop-inliner",
+ ScopInlinerWrapperPass, "polly-scop-inliner",
"inline functions based on how much of the function is a scop.", false,
false)
INITIALIZE_PASS_END(
- ScopInliner, "polly-scop-inliner",
+ ScopInlinerWrapperPass, "polly-scop-inliner",
"inline functions based on how much of the function is a scop.", false,
false)
+
+polly::ScopInlinerPass::ScopInlinerPass() {
+ if (!polly::PollyAllowFullFunction) {
+ report_fatal_error(
+ "Aborting from ScopInliner because it only makes sense to run with "
+ "-polly-allow-full-function. "
+ "The heurtistic for ScopInliner checks that the full function is a "
+ "Scop, which happens if and only if polly-allow-full-function is "
+ " enabled. "
+ " If not, the entry block is not included in the Scop");
+ }
+}
+
+PreservedAnalyses polly::ScopInlinerPass::run(llvm::LazyCallGraph::SCC &SCC,
+ llvm::CGSCCAnalysisManager &AM,
+ llvm::LazyCallGraph &CG,
+ llvm::CGSCCUpdateResult &UR) {
+ Function *F = &SCC.begin()->getFunction();
+ bool Changed = runScopInlinerImpl(F, SCC);
+ return Changed ? PreservedAnalyses::none() : PreservedAnalyses::all();
+}
diff --git a/polly/test/ScopInliner/ignore-declares.ll b/polly/test/ScopInliner/ignore-declares.ll
index 11722dcb32166..5c0cfa103f0bf 100644
--- a/polly/test/ScopInliner/ignore-declares.ll
+++ b/polly/test/ScopInliner/ignore-declares.ll
@@ -1,5 +1,4 @@
-; RUN: opt %loadPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-scops -disable-output < %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s
; Check that we do not crash if there are declares. We should skip function
; declarations and not try to query for domtree.
diff --git a/polly/test/ScopInliner/invariant-load-func.ll b/polly/test/ScopInliner/invariant-load-func.ll
index ffd2ec9cdb60f..58c556a455fb9 100644
--- a/polly/test/ScopInliner/invariant-load-func.ll
+++ b/polly/test/ScopInliner/invariant-load-func.ll
@@ -1,12 +1,9 @@
-; RUN: opt %loadNPMPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-invariant-load-hoisting '-passes=print<polly-function-scops>' -disable-output < %s | FileCheck %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions -polly-invariant-load-hoisting '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s 2>&1 | FileCheck %s
; Check that we inline a function that requires invariant load hoisting
; correctly.
; CHECK: Max Loop Depth: 2
-; REQUIRES: pollyacc
-
; void to_be_inlined(int A[], int *begin, int *end) {
; for(int i = *begin; i < *end; i++) {
diff --git a/polly/test/ScopInliner/simple-inline-loop.ll b/polly/test/ScopInliner/simple-inline-loop.ll
index a5e3483edad05..f12798a3d831a 100644
--- a/polly/test/ScopInliner/simple-inline-loop.ll
+++ b/polly/test/ScopInliner/simple-inline-loop.ll
@@ -1,5 +1,4 @@
-; RUN: opt %loadPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-print-scops -disable-output < %s | FileCheck %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s 2>&1 | FileCheck %s
; Check that we get the 2 nested loops by inlining `to_be_inlined` into
; `inline_site`.
|
|
@llvm/pr-subscribers-llvm-transforms Author: Michael Kruse (Meinersbur) ChangesAn assertion failed when Polly was registering for the pass manager which assumed that there would be only Polly passes. Since this does not need to be the case, re-apply with the assert removed. Includes a non-Polly change to trigger the premerge CI to trigger check-llvm failed for 0b9a7b8. Full diff: https://github.com/llvm/llvm-project/pull/164601.diff 10 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanValue.h b/llvm/lib/Transforms/Vectorize/VPlanValue.h
index 0678bc90ef4b5..a33391e8484f8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanValue.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanValue.h
@@ -41,10 +41,10 @@ class VPRecipeBase;
class VPInterleaveBase;
class VPPhiAccessors;
-// This is the base class of the VPlan Def/Use graph, used for modeling the data
-// flow into, within and out of the VPlan. VPValues can stand for live-ins
-// coming from the input IR and instructions which VPlan will generate if
-// executed.
+/// This is the base class of the VPlan Def/Use graph, used for modeling the data
+/// flow into, within and out of the VPlan. VPValues can stand for live-ins
+/// coming from the input IR and instructions which VPlan will generate if
+/// executed.
class LLVM_ABI_FOR_TEST VPValue {
friend class VPDef;
friend struct VPDoubleValueDef;
@@ -57,7 +57,7 @@ class LLVM_ABI_FOR_TEST VPValue {
SmallVector<VPUser *, 1> Users;
protected:
- // Hold the underlying Value, if any, attached to this VPValue.
+ /// Hold the underlying Value, if any, attached to this VPValue.
Value *UnderlyingVal;
/// Pointer to the VPDef that defines this VPValue. If it is nullptr, the
diff --git a/polly/docs/ReleaseNotes.rst b/polly/docs/ReleaseNotes.rst
index f7c9689089be2..f5ea47b69cf02 100644
--- a/polly/docs/ReleaseNotes.rst
+++ b/polly/docs/ReleaseNotes.rst
@@ -11,3 +11,5 @@ In Polly |version| the following important changes have been incorporated.
the new features that have recently been committed to our development
branch.
+ * ScopInliner has been updated for the New Pass Manager.
+
diff --git a/polly/include/polly/LinkAllPasses.h b/polly/include/polly/LinkAllPasses.h
index c3b68a74056ac..9978344c73e9f 100644
--- a/polly/include/polly/LinkAllPasses.h
+++ b/polly/include/polly/LinkAllPasses.h
@@ -119,7 +119,7 @@ struct PollyForcePassLinking {
namespace llvm {
void initializeCodePreparationPass(llvm::PassRegistry &);
-void initializeScopInlinerPass(llvm::PassRegistry &);
+void initializeScopInlinerWrapperPassPass(llvm::PassRegistry &);
void initializeScopDetectionWrapperPassPass(llvm::PassRegistry &);
void initializeScopDetectionPrinterLegacyPassPass(llvm::PassRegistry &);
void initializeScopInfoRegionPassPass(PassRegistry &);
diff --git a/polly/include/polly/ScopInliner.h b/polly/include/polly/ScopInliner.h
new file mode 100644
index 0000000000000..014667804330f
--- /dev/null
+++ b/polly/include/polly/ScopInliner.h
@@ -0,0 +1,34 @@
+//===------ ScopInliner.h ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef POLLY_POLLYINLINER_H
+#define POLLY_POLLYINLINER_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace polly {
+class ScopInlinerPass : public llvm::PassInfoMixin<ScopInlinerPass> {
+public:
+ ScopInlinerPass();
+
+ llvm::PreservedAnalyses run(llvm::LazyCallGraph::SCC &C,
+ llvm::CGSCCAnalysisManager &AM,
+ llvm::LazyCallGraph &CG,
+ llvm::CGSCCUpdateResult &UR);
+};
+
+llvm::Pass *createScopInlinerWrapperPass();
+} // namespace polly
+
+namespace llvm {
+void initializeScopInlinerWrapperPassPass(llvm::PassRegistry &);
+}
+
+#endif /* POLLY_POLLYINLINER_H */
diff --git a/polly/lib/Support/PollyPasses.def b/polly/lib/Support/PollyPasses.def
index e068f31fdb703..2c792a5867100 100644
--- a/polly/lib/Support/PollyPasses.def
+++ b/polly/lib/Support/PollyPasses.def
@@ -1,3 +1,9 @@
+#ifndef CGSCC_PASS
+#define CGSCC_PASS(NAME, CREATE_PASS, PARSER)
+#endif
+CGSCC_PASS("polly-inline", ScopInlinerPass(), parseNoOptions)
+#undef CGSCC_PASS
+
#ifndef FUNCTION_ANALYSIS
#define FUNCTION_ANALYSIS(NAME, CREATE_PASS)
#endif
diff --git a/polly/lib/Support/RegisterPasses.cpp b/polly/lib/Support/RegisterPasses.cpp
index 0420dff944f62..04f8715502c38 100644
--- a/polly/lib/Support/RegisterPasses.cpp
+++ b/polly/lib/Support/RegisterPasses.cpp
@@ -35,6 +35,7 @@
#include "polly/ScopDetection.h"
#include "polly/ScopGraphPrinter.h"
#include "polly/ScopInfo.h"
+#include "polly/ScopInliner.h"
#include "polly/Simplify.h"
#include "polly/Support/DumpFunctionPass.h"
#include "polly/Support/DumpModulePass.h"
@@ -46,10 +47,13 @@
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"
#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
#include "llvm/Support/TargetSelect.h"
#include "llvm/Transforms/IPO.h"
+using namespace llvm;
namespace cl = llvm::cl;
+using namespace polly;
using llvm::FunctionPassManager;
using llvm::OptimizationLevel;
@@ -233,7 +237,7 @@ void initializePollyPasses(llvm::PassRegistry &Registry) {
initializePollyCanonicalizePass(Registry);
initializeScopDetectionWrapperPassPass(Registry);
initializeScopDetectionPrinterLegacyPassPass(Registry);
- initializeScopInlinerPass(Registry);
+ initializeScopInlinerWrapperPassPass(Registry);
initializeScopInfoRegionPassPass(Registry);
initializeScopInfoPrinterLegacyRegionPassPass(Registry);
initializeScopInfoWrapperPassPass(Registry);
@@ -434,6 +438,16 @@ static void buildLatePollyPipeline(FunctionPassManager &PM,
false);
}
+static llvm::Expected<std::monostate> parseNoOptions(StringRef Params) {
+ if (!Params.empty())
+ return make_error<StringError>(
+ formatv("'{0}' passed to pass that does not take any options", Params)
+ .str(),
+ inconvertibleErrorCode());
+
+ return std::monostate{};
+}
+
static OwningScopAnalysisManagerFunctionProxy
createScopAnalyses(FunctionAnalysisManager &FAM,
PassInstrumentationCallbacks *PIC) {
@@ -461,6 +475,23 @@ static void registerFunctionAnalyses(FunctionAnalysisManager &FAM,
FAM.registerPass([&FAM, PIC] { return createScopAnalyses(FAM, PIC); });
}
+static llvm::Expected<bool>
+parseCGPipeline(StringRef Name, llvm::CGSCCPassManager &CGPM,
+ PassInstrumentationCallbacks *PIC,
+ ArrayRef<PassBuilder::PipelineElement> Pipeline) {
+#define CGSCC_PASS(NAME, CREATE_PASS, PARSER) \
+ if (PassBuilder::checkParametrizedPassName(Name, NAME)) { \
+ auto Params = PassBuilder::parsePassParameters(PARSER, Name, NAME); \
+ if (!Params) \
+ return Params.takeError(); \
+ CGPM.addPass(CREATE_PASS); \
+ return true; \
+ }
+#include "PollyPasses.def"
+
+ return false;
+}
+
static bool
parseFunctionPipeline(StringRef Name, FunctionPassManager &FPM,
ArrayRef<PassBuilder::PipelineElement> Pipeline) {
@@ -598,6 +629,12 @@ void registerPollyPasses(PassBuilder &PB) {
ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
return parseScopPipeline(Name, FPM, PIC, Pipeline);
});
+ PB.registerPipelineParsingCallback(
+ [PIC](StringRef Name, CGSCCPassManager &CGPM,
+ ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
+ ExitOnError Err("Unable to parse Polly call graph pass: ");
+ return Err(parseCGPipeline(Name, CGPM, PIC, Pipeline));
+ });
PB.registerParseTopLevelPipelineCallback(
[PIC](llvm::ModulePassManager &MPM,
ArrayRef<PassBuilder::PipelineElement> Pipeline) -> bool {
diff --git a/polly/lib/Transform/ScopInliner.cpp b/polly/lib/Transform/ScopInliner.cpp
index b78206c1e40ba..c04ba3498339e 100644
--- a/polly/lib/Transform/ScopInliner.cpp
+++ b/polly/lib/Transform/ScopInliner.cpp
@@ -13,10 +13,14 @@
//
//===----------------------------------------------------------------------===//
-#include "polly/LinkAllPasses.h"
+#include "polly/ScopInliner.h"
#include "polly/ScopDetection.h"
+#include "polly/ScopInliner.h"
#include "llvm/Analysis/CallGraph.h"
#include "llvm/Analysis/CallGraphSCCPass.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/RegionInfo.h"
+#include "llvm/IR/Dominators.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Transforms/IPO/AlwaysInliner.h"
@@ -28,13 +32,77 @@ using namespace llvm;
using namespace polly;
namespace {
-class ScopInliner final : public CallGraphSCCPass {
+
+/// Inliner implementation that works with both, LPM (using SCC_t=CallGraph) and
+/// NPM (using SCC_t=LazyCallGraph::SCC)
+template <typename SCC_t> bool runScopInlinerImpl(Function *F, SCC_t &SCC) {
+ // We do not try to inline non-trivial SCCs because this would lead to
+ // "infinite" inlining if we are not careful.
+ if (SCC.size() > 1)
+ return false;
+ assert(SCC.size() == 1 && "found empty SCC");
+
+ // If the function is a nullptr, or the function is a declaration.
+ if (!F)
+ return false;
+ if (F->isDeclaration()) {
+ POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
+ << "because it is a declaration.\n");
+ return false;
+ }
+
+ PassBuilder PB;
+ // Populate analysis managers and register Polly-specific analyses.
+ LoopAnalysisManager LAM;
+ FunctionAnalysisManager FAM;
+ CGSCCAnalysisManager CGAM;
+ ModuleAnalysisManager MAM;
+ PB.registerModuleAnalyses(MAM);
+ PB.registerCGSCCAnalyses(CGAM);
+ PB.registerFunctionAnalyses(FAM);
+ PB.registerLoopAnalyses(LAM);
+ PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+
+ auto &DT = FAM.getResult<DominatorTreeAnalysis>(*F);
+ auto &SE = FAM.getResult<ScalarEvolutionAnalysis>(*F);
+ auto &LI = FAM.getResult<LoopAnalysis>(*F);
+ auto &RI = FAM.getResult<RegionInfoAnalysis>(*F);
+ auto &AA = FAM.getResult<AAManager>(*F);
+ auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(*F);
+ ScopDetection SD(DT, SE, LI, RI, AA, ORE);
+ SD.detect(*F);
+
+ const bool HasScopAsTopLevelRegion =
+ SD.ValidRegions.contains(RI.getTopLevelRegion());
+
+ bool Changed = false;
+ if (HasScopAsTopLevelRegion) {
+ POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
+ << " has scop as top level region");
+ F->addFnAttr(llvm::Attribute::AlwaysInline);
+
+ ModulePassManager MPM;
+ MPM.addPass(AlwaysInlinerPass());
+ Module *M = F->getParent();
+ assert(M && "Function has illegal module");
+ PreservedAnalyses PA = MPM.run(*M, MAM);
+ if (!PA.areAllPreserved())
+ Changed = true;
+ } else {
+ POLLY_DEBUG(dbgs() << F->getName()
+ << " does NOT have scop as top level region\n");
+ }
+
+ return Changed;
+}
+
+class ScopInlinerWrapperPass final : public CallGraphSCCPass {
using llvm::Pass::doInitialization;
public:
static char ID;
- ScopInliner() : CallGraphSCCPass(ID) {}
+ ScopInlinerWrapperPass() : CallGraphSCCPass(ID) {}
bool doInitialization(CallGraph &CG) override {
if (!polly::PollyAllowFullFunction) {
@@ -50,60 +118,8 @@ class ScopInliner final : public CallGraphSCCPass {
}
bool runOnSCC(CallGraphSCC &SCC) override {
- // We do not try to inline non-trivial SCCs because this would lead to
- // "infinite" inlining if we are not careful.
- if (SCC.size() > 1)
- return false;
- assert(SCC.size() == 1 && "found empty SCC");
Function *F = (*SCC.begin())->getFunction();
-
- // If the function is a nullptr, or the function is a declaration.
- if (!F)
- return false;
- if (F->isDeclaration()) {
- POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
- << "because it is a declaration.\n");
- return false;
- }
-
- PassBuilder PB;
- // Populate analysis managers and register Polly-specific analyses.
- LoopAnalysisManager LAM;
- FunctionAnalysisManager FAM;
- CGSCCAnalysisManager CGAM;
- ModuleAnalysisManager MAM;
- FAM.registerPass([] { return ScopAnalysis(); });
- PB.registerModuleAnalyses(MAM);
- PB.registerCGSCCAnalyses(CGAM);
- PB.registerFunctionAnalyses(FAM);
- PB.registerLoopAnalyses(LAM);
- PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
-
- RegionInfo &RI = FAM.getResult<RegionInfoAnalysis>(*F);
- ScopDetection &SD = FAM.getResult<ScopAnalysis>(*F);
-
- const bool HasScopAsTopLevelRegion =
- SD.ValidRegions.contains(RI.getTopLevelRegion());
-
- bool Changed = false;
- if (HasScopAsTopLevelRegion) {
- POLLY_DEBUG(dbgs() << "Skipping " << F->getName()
- << " has scop as top level region");
- F->addFnAttr(llvm::Attribute::AlwaysInline);
-
- ModulePassManager MPM;
- MPM.addPass(AlwaysInlinerPass());
- Module *M = F->getParent();
- assert(M && "Function has illegal module");
- PreservedAnalyses PA = MPM.run(*M, MAM);
- if (!PA.areAllPreserved())
- Changed = true;
- } else {
- POLLY_DEBUG(dbgs() << F->getName()
- << " does NOT have scop as top level region\n");
- }
-
- return Changed;
+ return runScopInlinerImpl(F, SCC);
};
void getAnalysisUsage(AnalysisUsage &AU) const override {
@@ -111,18 +127,39 @@ class ScopInliner final : public CallGraphSCCPass {
}
};
} // namespace
-char ScopInliner::ID;
+char ScopInlinerWrapperPass::ID;
-Pass *polly::createScopInlinerPass() {
- ScopInliner *pass = new ScopInliner();
+Pass *polly::createScopInlinerWrapperPass() {
+ ScopInlinerWrapperPass *pass = new ScopInlinerWrapperPass();
return pass;
}
INITIALIZE_PASS_BEGIN(
- ScopInliner, "polly-scop-inliner",
+ ScopInlinerWrapperPass, "polly-scop-inliner",
"inline functions based on how much of the function is a scop.", false,
false)
INITIALIZE_PASS_END(
- ScopInliner, "polly-scop-inliner",
+ ScopInlinerWrapperPass, "polly-scop-inliner",
"inline functions based on how much of the function is a scop.", false,
false)
+
+polly::ScopInlinerPass::ScopInlinerPass() {
+ if (!polly::PollyAllowFullFunction) {
+ report_fatal_error(
+ "Aborting from ScopInliner because it only makes sense to run with "
+ "-polly-allow-full-function. "
+ "The heurtistic for ScopInliner checks that the full function is a "
+ "Scop, which happens if and only if polly-allow-full-function is "
+ " enabled. "
+ " If not, the entry block is not included in the Scop");
+ }
+}
+
+PreservedAnalyses polly::ScopInlinerPass::run(llvm::LazyCallGraph::SCC &SCC,
+ llvm::CGSCCAnalysisManager &AM,
+ llvm::LazyCallGraph &CG,
+ llvm::CGSCCUpdateResult &UR) {
+ Function *F = &SCC.begin()->getFunction();
+ bool Changed = runScopInlinerImpl(F, SCC);
+ return Changed ? PreservedAnalyses::none() : PreservedAnalyses::all();
+}
diff --git a/polly/test/ScopInliner/ignore-declares.ll b/polly/test/ScopInliner/ignore-declares.ll
index 11722dcb32166..5c0cfa103f0bf 100644
--- a/polly/test/ScopInliner/ignore-declares.ll
+++ b/polly/test/ScopInliner/ignore-declares.ll
@@ -1,5 +1,4 @@
-; RUN: opt %loadPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-scops -disable-output < %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s
; Check that we do not crash if there are declares. We should skip function
; declarations and not try to query for domtree.
diff --git a/polly/test/ScopInliner/invariant-load-func.ll b/polly/test/ScopInliner/invariant-load-func.ll
index ffd2ec9cdb60f..58c556a455fb9 100644
--- a/polly/test/ScopInliner/invariant-load-func.ll
+++ b/polly/test/ScopInliner/invariant-load-func.ll
@@ -1,12 +1,9 @@
-; RUN: opt %loadNPMPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-invariant-load-hoisting '-passes=print<polly-function-scops>' -disable-output < %s | FileCheck %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions -polly-invariant-load-hoisting '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s 2>&1 | FileCheck %s
; Check that we inline a function that requires invariant load hoisting
; correctly.
; CHECK: Max Loop Depth: 2
-; REQUIRES: pollyacc
-
; void to_be_inlined(int A[], int *begin, int *end) {
; for(int i = *begin; i < *end; i++) {
diff --git a/polly/test/ScopInliner/simple-inline-loop.ll b/polly/test/ScopInliner/simple-inline-loop.ll
index a5e3483edad05..f12798a3d831a 100644
--- a/polly/test/ScopInliner/simple-inline-loop.ll
+++ b/polly/test/ScopInliner/simple-inline-loop.ll
@@ -1,5 +1,4 @@
-; RUN: opt %loadPolly -polly-detect-full-functions -polly-scop-inliner \
-; RUN: -polly-print-scops -disable-output < %s | FileCheck %s
+; RUN: opt %loadNPMPolly -polly-detect-full-functions '-passes=cgscc(polly-inline),function(print<polly-function-scops>)' -disable-output < %s 2>&1 | FileCheck %s
; Check that we get the 2 nested loops by inlining `to_be_inlined` into
; `inline_site`.
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An assertion failed when Polly was registering for the pass manager which assumed that there would be only Polly passes. Since this does not need to be the case, re-apply with the assert removed.
Includes a non-Polly change to trigger the premerge CI to trigger check-llvm which failed for 0b9a7b8, but pre-merge did not catch.