[X86] Delete Profile Guided Prefetch Passes #167317
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As the PGPF effort has been turned down, there is no current way to generate profiles that will be used by these passes. Current efforts are also focused around inserting prefetches in PLO optimizers, which have a more accurate view of how the code looks.
Member
|
@llvm/pr-subscribers-backend-x86 Author: Aiden Grossman (boomanaiden154) ChangesAs the PGPF effort has been turned down, there is no current way to generate profiles that will be used by these passes. Current efforts are also focused around inserting prefetches in PLO optimizers, which have a more accurate view of how the code looks. Patch is 40.80 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/167317.diff 17 Files Affected:
diff --git a/llvm/lib/Target/X86/CMakeLists.txt b/llvm/lib/Target/X86/CMakeLists.txt
index f9bd233cf8ecf..434a6d2c3553f 100644
--- a/llvm/lib/Target/X86/CMakeLists.txt
+++ b/llvm/lib/Target/X86/CMakeLists.txt
@@ -31,7 +31,6 @@ set(sources
X86CmovConversion.cpp
X86CodeGenPassBuilder.cpp
X86DomainReassignment.cpp
- X86DiscriminateMemOps.cpp
X86LowerTileCopy.cpp
X86LowerAMXType.cpp
X86LowerAMXIntrinsics.cpp
@@ -57,7 +56,6 @@ set(sources
X86IndirectBranchTracking.cpp
X86IndirectThunks.cpp
X86InterleavedAccess.cpp
- X86InsertPrefetch.cpp
X86InstCombineIntrinsic.cpp
X86InstrFMA3Info.cpp
X86InstrFoldTables.cpp
diff --git a/llvm/lib/Target/X86/X86.h b/llvm/lib/Target/X86/X86.h
index 2b83d575ace91..39251f816b9cb 100644
--- a/llvm/lib/Target/X86/X86.h
+++ b/llvm/lib/Target/X86/X86.h
@@ -154,13 +154,6 @@ FunctionPass *createX86IndirectThunksPass();
/// This pass replaces ret instructions with jmp's to __x86_return thunk.
FunctionPass *createX86ReturnThunksPass();
-/// This pass ensures instructions featuring a memory operand
-/// have distinctive <LineNumber, Discriminator> (with respect to each other)
-FunctionPass *createX86DiscriminateMemOpsPass();
-
-/// This pass applies profiling information to insert cache prefetches.
-FunctionPass *createX86InsertPrefetchPass();
-
/// This pass insert wait instruction after X87 instructions which could raise
/// fp exceptions when strict-fp enabled.
FunctionPass *createX86InsertX87waitPass();
diff --git a/llvm/lib/Target/X86/X86DiscriminateMemOps.cpp b/llvm/lib/Target/X86/X86DiscriminateMemOps.cpp
deleted file mode 100644
index bd151a450394a..0000000000000
--- a/llvm/lib/Target/X86/X86DiscriminateMemOps.cpp
+++ /dev/null
@@ -1,184 +0,0 @@
-//===- X86DiscriminateMemOps.cpp - Unique IDs for Mem Ops -----------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-///
-/// This pass aids profile-driven cache prefetch insertion by ensuring all
-/// instructions that have a memory operand are distinguishible from each other.
-///
-//===----------------------------------------------------------------------===//
-
-#include "X86.h"
-#include "X86Subtarget.h"
-#include "llvm/CodeGen/MachineFunctionPass.h"
-#include "llvm/CodeGen/MachineModuleInfo.h"
-#include "llvm/IR/DebugInfoMetadata.h"
-#include "llvm/ProfileData/SampleProf.h"
-#include "llvm/ProfileData/SampleProfReader.h"
-#include "llvm/Support/Debug.h"
-#include <optional>
-using namespace llvm;
-
-#define DEBUG_TYPE "x86-discriminate-memops"
-
-static cl::opt<bool> EnableDiscriminateMemops(
- DEBUG_TYPE, cl::init(false),
- cl::desc("Generate unique debug info for each instruction with a memory "
- "operand. Should be enabled for profile-driven cache prefetching, "
- "both in the build of the binary being profiled, as well as in "
- "the build of the binary consuming the profile."),
- cl::Hidden);
-
-static cl::opt<bool> BypassPrefetchInstructions(
- "x86-bypass-prefetch-instructions", cl::init(true),
- cl::desc("When discriminating instructions with memory operands, ignore "
- "prefetch instructions. This ensures the other memory operand "
- "instructions have the same identifiers after inserting "
- "prefetches, allowing for successive insertions."),
- cl::Hidden);
-
-namespace {
-
-using Location = std::pair<StringRef, unsigned>;
-
-Location diToLocation(const DILocation *Loc) {
- return std::make_pair(Loc->getFilename(), Loc->getLine());
-}
-
-/// Ensure each instruction having a memory operand has a distinct <LineNumber,
-/// Discriminator> pair.
-void updateDebugInfo(MachineInstr *MI, const DILocation *Loc) {
- DebugLoc DL(Loc);
- MI->setDebugLoc(DL);
-}
-
-class X86DiscriminateMemOps : public MachineFunctionPass {
- bool runOnMachineFunction(MachineFunction &MF) override;
- StringRef getPassName() const override {
- return "X86 Discriminate Memory Operands";
- }
-
-public:
- static char ID;
-
- /// Default construct and initialize the pass.
- X86DiscriminateMemOps();
-};
-
-bool IsPrefetchOpcode(unsigned Opcode) {
- return Opcode == X86::PREFETCHNTA || Opcode == X86::PREFETCHT0 ||
- Opcode == X86::PREFETCHT1 || Opcode == X86::PREFETCHT2 ||
- Opcode == X86::PREFETCHIT0 || Opcode == X86::PREFETCHIT1 ||
- Opcode == X86::PREFETCHRST2;
-}
-} // end anonymous namespace
-
-//===----------------------------------------------------------------------===//
-// Implementation
-//===----------------------------------------------------------------------===//
-
-char X86DiscriminateMemOps::ID = 0;
-
-/// Default construct and initialize the pass.
-X86DiscriminateMemOps::X86DiscriminateMemOps() : MachineFunctionPass(ID) {}
-
-bool X86DiscriminateMemOps::runOnMachineFunction(MachineFunction &MF) {
- if (!EnableDiscriminateMemops)
- return false;
-
- DISubprogram *FDI = MF.getFunction().getSubprogram();
- if (!FDI || !FDI->getUnit()->getDebugInfoForProfiling())
- return false;
-
- // Have a default DILocation, if we find instructions with memops that don't
- // have any debug info.
- const DILocation *ReferenceDI =
- DILocation::get(FDI->getContext(), FDI->getLine(), 0, FDI);
- assert(ReferenceDI && "ReferenceDI should not be nullptr");
- DenseMap<Location, unsigned> MemOpDiscriminators;
- MemOpDiscriminators[diToLocation(ReferenceDI)] = 0;
-
- // Figure out the largest discriminator issued for each Location. When we
- // issue new discriminators, we can thus avoid issuing discriminators
- // belonging to instructions that don't have memops. This isn't a requirement
- // for the goals of this pass, however, it avoids unnecessary ambiguity.
- for (auto &MBB : MF) {
- for (auto &MI : MBB) {
- const auto &DI = MI.getDebugLoc();
- if (!DI)
- continue;
- if (BypassPrefetchInstructions && IsPrefetchOpcode(MI.getDesc().Opcode))
- continue;
- Location Loc = diToLocation(DI);
- unsigned &Disc = MemOpDiscriminators[Loc];
- Disc = std::max(Disc, DI->getBaseDiscriminator());
- }
- }
-
- // Keep track of the discriminators seen at each Location. If an instruction's
- // DebugInfo has a Location and discriminator we've already seen, replace its
- // discriminator with a new one, to guarantee uniqueness.
- DenseMap<Location, DenseSet<unsigned>> Seen;
-
- bool Changed = false;
- for (auto &MBB : MF) {
- for (auto &MI : MBB) {
- if (X86II::getMemoryOperandNo(MI.getDesc().TSFlags) < 0)
- continue;
- if (BypassPrefetchInstructions && IsPrefetchOpcode(MI.getDesc().Opcode))
- continue;
- const DILocation *DI = MI.getDebugLoc();
- bool HasDebug = DI;
- if (!HasDebug) {
- DI = ReferenceDI;
- }
- Location L = diToLocation(DI);
- DenseSet<unsigned> &Set = Seen[L];
- const std::pair<DenseSet<unsigned>::iterator, bool> TryInsert =
- Set.insert(DI->getBaseDiscriminator());
- if (!TryInsert.second || !HasDebug) {
- unsigned BF, DF, CI = 0;
- DILocation::decodeDiscriminator(DI->getDiscriminator(), BF, DF, CI);
- std::optional<unsigned> EncodedDiscriminator =
- DILocation::encodeDiscriminator(MemOpDiscriminators[L] + 1, DF, CI);
-
- if (!EncodedDiscriminator) {
- // FIXME(mtrofin): The assumption is that this scenario is infrequent/OK
- // not to support. If evidence points otherwise, we can explore synthesizeing
- // unique DIs by adding fake line numbers, or by constructing 64 bit
- // discriminators.
- LLVM_DEBUG(dbgs() << "Unable to create a unique discriminator "
- "for instruction with memory operand in: "
- << DI->getFilename() << " Line: " << DI->getLine()
- << " Column: " << DI->getColumn()
- << ". This is likely due to a large macro expansion. \n");
- continue;
- }
- // Since we were able to encode, bump the MemOpDiscriminators.
- ++MemOpDiscriminators[L];
- DI = DI->cloneWithDiscriminator(*EncodedDiscriminator);
- assert(DI && "DI should not be nullptr");
- updateDebugInfo(&MI, DI);
- Changed = true;
- std::pair<DenseSet<unsigned>::iterator, bool> MustInsert =
- Set.insert(DI->getBaseDiscriminator());
- (void)MustInsert; // Silence warning in release build.
- assert(MustInsert.second && "New discriminator shouldn't be present in set");
- }
-
- // Bump the reference DI to avoid cramming discriminators on line 0.
- // FIXME(mtrofin): pin ReferenceDI on blocks or first instruction with DI
- // in a block. It's more consistent than just relying on the last memop
- // instruction we happened to see.
- ReferenceDI = DI;
- }
- }
- return Changed;
-}
-
-FunctionPass *llvm::createX86DiscriminateMemOpsPass() {
- return new X86DiscriminateMemOps();
-}
diff --git a/llvm/lib/Target/X86/X86InsertPrefetch.cpp b/llvm/lib/Target/X86/X86InsertPrefetch.cpp
deleted file mode 100644
index 953b755a0ca4c..0000000000000
--- a/llvm/lib/Target/X86/X86InsertPrefetch.cpp
+++ /dev/null
@@ -1,259 +0,0 @@
-//===------- X86InsertPrefetch.cpp - Insert cache prefetch hints ----------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This pass applies cache prefetch instructions based on a profile. The pass
-// assumes DiscriminateMemOps ran immediately before, to ensure debug info
-// matches the one used at profile generation time. The profile is encoded in
-// afdo format (text or binary). It contains prefetch hints recommendations.
-// Each recommendation is made in terms of debug info locations, a type (i.e.
-// nta, t{0|1|2}) and a delta. The debug info identifies an instruction with a
-// memory operand (see X86DiscriminateMemOps). The prefetch will be made for
-// a location at that memory operand + the delta specified in the
-// recommendation.
-//
-//===----------------------------------------------------------------------===//
-
-#include "X86.h"
-#include "X86Subtarget.h"
-#include "llvm/CodeGen/MachineFunctionPass.h"
-#include "llvm/CodeGen/MachineModuleInfo.h"
-#include "llvm/IR/DebugInfoMetadata.h"
-#include "llvm/IR/Module.h"
-#include "llvm/ProfileData/SampleProf.h"
-#include "llvm/ProfileData/SampleProfReader.h"
-#include "llvm/Support/VirtualFileSystem.h"
-#include "llvm/Transforms/IPO/SampleProfile.h"
-using namespace llvm;
-using namespace sampleprof;
-
-static cl::opt<std::string>
- PrefetchHintsFile("prefetch-hints-file",
- cl::desc("Path to the prefetch hints profile. See also "
- "-x86-discriminate-memops"),
- cl::Hidden);
-namespace {
-
-class X86InsertPrefetch : public MachineFunctionPass {
- void getAnalysisUsage(AnalysisUsage &AU) const override;
- bool doInitialization(Module &) override;
-
- bool runOnMachineFunction(MachineFunction &MF) override;
- struct PrefetchInfo {
- unsigned InstructionID;
- int64_t Delta;
- };
- typedef SmallVectorImpl<PrefetchInfo> Prefetches;
- bool findPrefetchInfo(const FunctionSamples *Samples, const MachineInstr &MI,
- Prefetches &prefetches) const;
-
-public:
- static char ID;
- X86InsertPrefetch(const std::string &PrefetchHintsFilename);
- StringRef getPassName() const override {
- return "X86 Insert Cache Prefetches";
- }
-
-private:
- std::string Filename;
- std::unique_ptr<SampleProfileReader> Reader;
-};
-
-using PrefetchHints = SampleRecord::CallTargetMap;
-
-// Return any prefetching hints for the specified MachineInstruction. The hints
-// are returned as pairs (name, delta).
-ErrorOr<const PrefetchHints &>
-getPrefetchHints(const FunctionSamples *TopSamples, const MachineInstr &MI) {
- if (const auto &Loc = MI.getDebugLoc())
- if (const auto *Samples = TopSamples->findFunctionSamples(Loc))
- return Samples->findCallTargetMapAt(FunctionSamples::getOffset(Loc),
- Loc->getBaseDiscriminator());
- return std::error_code();
-}
-
-// The prefetch instruction can't take memory operands involving vector
-// registers.
-bool IsMemOpCompatibleWithPrefetch(const MachineInstr &MI, int Op) {
- Register BaseReg = MI.getOperand(Op + X86::AddrBaseReg).getReg();
- Register IndexReg = MI.getOperand(Op + X86::AddrIndexReg).getReg();
- return (BaseReg == 0 ||
- X86MCRegisterClasses[X86::GR64RegClassID].contains(BaseReg) ||
- X86MCRegisterClasses[X86::GR32RegClassID].contains(BaseReg)) &&
- (IndexReg == 0 ||
- X86MCRegisterClasses[X86::GR64RegClassID].contains(IndexReg) ||
- X86MCRegisterClasses[X86::GR32RegClassID].contains(IndexReg));
-}
-
-} // end anonymous namespace
-
-//===----------------------------------------------------------------------===//
-// Implementation
-//===----------------------------------------------------------------------===//
-
-char X86InsertPrefetch::ID = 0;
-
-X86InsertPrefetch::X86InsertPrefetch(const std::string &PrefetchHintsFilename)
- : MachineFunctionPass(ID), Filename(PrefetchHintsFilename) {}
-
-/// Return true if the provided MachineInstruction has cache prefetch hints. In
-/// that case, the prefetch hints are stored, in order, in the Prefetches
-/// vector.
-bool X86InsertPrefetch::findPrefetchInfo(const FunctionSamples *TopSamples,
- const MachineInstr &MI,
- Prefetches &Prefetches) const {
- assert(Prefetches.empty() &&
- "Expected caller passed empty PrefetchInfo vector.");
-
- // There is no point to match prefetch hints if the profile is using MD5.
- if (FunctionSamples::UseMD5)
- return false;
-
- static constexpr std::pair<StringLiteral, unsigned> HintTypes[] = {
- {"_nta_", X86::PREFETCHNTA},
- {"_t0_", X86::PREFETCHT0},
- {"_t1_", X86::PREFETCHT1},
- {"_t2_", X86::PREFETCHT2},
- };
- static const char *SerializedPrefetchPrefix = "__prefetch";
-
- auto T = getPrefetchHints(TopSamples, MI);
- if (!T)
- return false;
- int16_t max_index = -1;
- // Convert serialized prefetch hints into PrefetchInfo objects, and populate
- // the Prefetches vector.
- for (const auto &S_V : *T) {
- StringRef Name = S_V.first.stringRef();
- if (Name.consume_front(SerializedPrefetchPrefix)) {
- int64_t D = static_cast<int64_t>(S_V.second);
- unsigned IID = 0;
- for (const auto &HintType : HintTypes) {
- if (Name.consume_front(HintType.first)) {
- IID = HintType.second;
- break;
- }
- }
- if (IID == 0)
- return false;
- uint8_t index = 0;
- Name.consumeInteger(10, index);
-
- if (index >= Prefetches.size())
- Prefetches.resize(index + 1);
- Prefetches[index] = {IID, D};
- max_index = std::max(max_index, static_cast<int16_t>(index));
- }
- }
- assert(max_index + 1 >= 0 &&
- "Possible overflow: max_index + 1 should be positive.");
- assert(static_cast<size_t>(max_index + 1) == Prefetches.size() &&
- "The number of prefetch hints received should match the number of "
- "PrefetchInfo objects returned");
- return !Prefetches.empty();
-}
-
-bool X86InsertPrefetch::doInitialization(Module &M) {
- if (Filename.empty())
- return false;
-
- LLVMContext &Ctx = M.getContext();
- // TODO: Propagate virtual file system into LLVM targets.
- auto FS = vfs::getRealFileSystem();
- ErrorOr<std::unique_ptr<SampleProfileReader>> ReaderOrErr =
- SampleProfileReader::create(Filename, Ctx, *FS);
- if (std::error_code EC = ReaderOrErr.getError()) {
- std::string Msg = "Could not open profile: " + EC.message();
- Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg,
- DiagnosticSeverity::DS_Warning));
- return false;
- }
- Reader = std::move(ReaderOrErr.get());
- Reader->read();
- return true;
-}
-
-void X86InsertPrefetch::getAnalysisUsage(AnalysisUsage &AU) const {
- AU.setPreservesAll();
- MachineFunctionPass::getAnalysisUsage(AU);
-}
-
-bool X86InsertPrefetch::runOnMachineFunction(MachineFunction &MF) {
- if (!Reader)
- return false;
- const FunctionSamples *Samples = Reader->getSamplesFor(MF.getFunction());
- if (!Samples)
- return false;
-
- bool Changed = false;
-
- const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
- SmallVector<PrefetchInfo, 4> Prefetches;
- for (auto &MBB : MF) {
- for (auto MI = MBB.instr_begin(); MI != MBB.instr_end();) {
- auto Current = MI;
- ++MI;
-
- int Offset = X86II::getMemoryOperandNo(Current->getDesc().TSFlags);
- if (Offset < 0)
- continue;
- unsigned Bias = X86II::getOperandBias(Current->getDesc());
- int MemOpOffset = Offset + Bias;
- // FIXME(mtrofin): ORE message when the recommendation cannot be taken.
- if (!IsMemOpCompatibleWithPrefetch(*Current, MemOpOffset))
- continue;
- Prefetches.clear();
- if (!findPrefetchInfo(Samples, *Current, Prefetches))
- continue;
- assert(!Prefetches.empty() &&
- "The Prefetches vector should contain at least a value if "
- "findPrefetchInfo returned true.");
- for (auto &PrefInfo : Prefetches) {
- unsigned PFetchInstrID = PrefInfo.InstructionID;
- int64_t Delta = PrefInfo.Delta;
- const MCInstrDesc &Desc = TII->get(PFetchInstrID);
- MachineInstr *PFetch =
- MF.CreateMachineInstr(Desc, Current->getDebugLoc(), true);
- MachineInstrBuilder MIB(MF, PFetch);
-
- static_assert(X86::AddrBaseReg == 0 && X86::AddrScaleAmt == 1 &&
- X86::AddrIndexReg == 2 && X86::AddrDisp == 3 &&
- X86::AddrSegmentReg == 4,
- "Unexpected change in X86 operand offset order.");
-
- // This assumes X86::AddBaseReg = 0, {...}ScaleAmt = 1, etc.
- // FIXME(mtrofin): consider adding a:
- // MachineInstrBuilder::set(unsigned offset, op).
- MIB.addReg(Current->getOperand(MemOpOffset + X86::AddrBaseReg).getReg())
- .addImm(
- Current->getOperand(MemOpOffset + X86::AddrScaleAmt).getImm())
- .addReg(
- Current->getOperand(MemOpOffset + X86::AddrIndexReg).getReg())
- .addImm(Current->getOperand(MemOpOffset + X86::AddrDisp).getImm() +
- Delta)
- .addReg(Current->getOperand(MemOpOffset + X86::AddrSegmentReg)
- .getReg());
-
- if (!Current->memoperands_empty()) {
- MachineMemOperand *CurrentOp = *(Current->memoperands_begin());
- MIB.addMemOperand(MF.getMachineMemOperand(
- CurrentOp, CurrentOp->getOffset() + Delta, CurrentOp->getSize()));
- }
-
- // Insert before Current. This is because Current may clobber some of
- // the registers used to describe the input memory operand.
- MBB.insert(Current, PFetch);
- Changed = true;
- }
- }
- }
- return Changed;
-}
-
-FunctionPass *llvm::createX86InsertPrefetchPass() {
- return new X86InsertPrefetch(PrefetchHintsFile);
-}
diff --git a/llvm/lib/Target/X86/X86TargetMachine.cpp b/llvm/lib/Target/X86/X86TargetMachine.cpp
index d4ad98af9b30c..1bcc6f50b375a 100644
--- a/llvm/lib/Target/X86/X86TargetMachine.cpp
+++ b/llvm/lib/Target/X86/X86TargetMachine.cpp
@@ -563,8 +563,6 @@ void X86PassConfig::addPreEmitPass() {
addPass(createX86FixupVectorConstants());
}
addPass(createX86CompressEVEXPass());
- addPass(createX86DiscriminateMemOpsPass());
- addPass(createX86InsertPrefetchPass());
addPass(createX86InsertX87waitPass());
}
diff --git a/llvm/test/CodeGen/X86/O0-pipeline.ll b/llvm/test/CodeGen/X86/O0-pipeline.ll
index 0fbfb42d2a4dd..78a02b11b17bb 100644
--- a/llvm/test/CodeGen/...
[truncated]
|
mtrofin
approved these changes
Nov 17, 2025
Member
mtrofin
left a comment
mtrofin
left a comment
There was a problem hiding this comment.
LGTM, but please reference https://discourse.llvm.org/t/memory-prefetching-support-in-llvm/74078 and discourse.llvm.org/t/rfc-code-prefetch-insertion/88668. The former to support "turned down". The latter (and, btw, if there are other links relevant to the latter, please add) to support the statement about PLO optimizers (albeit the RFC I referenced is about code not data, but the idea holds)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As the PGPF effort has been turned down (see https://discourse.llvm.org/t/memory-prefetching-support-in-llvm/74078 for discussion), there is no current way to generate profiles that will be used by these passes. Current efforts are also focused around inserting prefetches in PLO optimizers (see https://discourse.llvm.org/t/rfc-code-prefetch-insertion/88668 for discussion. This is about instruction prefetching, not data prefetching, but the idea holds), which have a more accurate view of how the code looks.