Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[CSSPGO] Pseudo probes for function calls.
An indirect call site needs to be probed for its potential call targets. With CSSPGO a direct call also needs a probe so that a calling context can be represented by a stack of callsite probes. Unlike pseudo probes for basic blocks that are in form of standalone intrinsic call instructions, pseudo probes for callsites have to be attached to the call instruction, thus a separate instruction would not work. One possible way of attaching a probe to a call instruction is to use a special metadata that carries information about the probe. The special metadata will have to make its way through the optimization pipeline down to object emission. This requires additional efforts to maintain the metadata in various places. Given that the `!dbg` metadata is a first-class metadata and has all essential support in place , leveraging the `!dbg` metadata as a channel to encode pseudo probe information is probably the easiest solution. With the requirement of not inflating `!dbg` metadata that is allocated for almost every instruction, we found that the 32-bit DWARF discriminator field which mainly serves AutoFDO can be reused for pseudo probes. DWARF discriminators distinguish identical source locations between instructions and with pseudo probes such support is not required. In this change we are using the discriminator field to encode the ID and type of a callsite probe and the encoded value will be unpacked and consumed right before object emission. When a callsite is inlined, the callsite discriminator field will go with the inlined instructions. The `!dbg` metadata of an inlined instruction is in form of a scope stack. The top of the stack is the instruction's original `!dbg` metadata and the bottom of the stack is for the original callsite of the top-level inliner. Except for the top of the stack, all other elements of the stack actually refer to the nested inlined callsites whose discriminator field (which actually represents a calliste probe) can be used together to represent the inline context of an inlined PseudoProbeInst or CallInst. To avoid collision with the baseline AutoFDO in various places that handles dwarf discriminators where a check against the `-pseudo-probe-for-profiling` switch is not available, a special encoding scheme is used to tell apart a pseudo probe discriminator from a regular discriminator. For the regular discriminator, if all lowest 3 bits are non-zero, it means the discriminator is basically empty and all higher 29 bits can be reversed for pseudo probe use. Callsite pseudo probes are inserted in `SampleProfileProbePass` and a target-independent MIR pass `PseudoProbeInserter` is added to unpack the probe ID/type from `!dbg`. Note that with this work the switch -debug-info-for-profiling will not work with -pseudo-probe-for-profiling anymore. They cannot be used at the same time. Reviewed By: wmi Differential Revision: https://reviews.llvm.org/D91756
- Loading branch information
Showing
17 changed files
with
292 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
//===- PseudoProbe.h - Pseudo Probe IR Helpers ------------------*- C++ -*-===// | ||
// | ||
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. | ||
// See https://llvm.org/LICENSE.txt for license information. | ||
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception | ||
// | ||
//===----------------------------------------------------------------------===// | ||
// | ||
// Pseudo probe IR intrinsic and dwarf discriminator manipulation routines. | ||
// | ||
//===----------------------------------------------------------------------===// | ||
|
||
#ifndef LLVM_IR_PSEUDOPROBE_H | ||
#define LLVM_IR_PSEUDOPROBE_H | ||
|
||
#include <cassert> | ||
#include <cstdint> | ||
|
||
namespace llvm { | ||
|
||
enum class PseudoProbeType { Block = 0, IndirectCall, DirectCall }; | ||
|
||
struct PseudoProbeDwarfDiscriminator { | ||
// The following APIs encodes/decodes per-probe information to/from a | ||
// 32-bit integer which is organized as: | ||
// [2:0] - 0x7, this is reserved for regular discriminator, | ||
// see DWARF discriminator encoding rule | ||
// [18:3] - probe id | ||
// [25:19] - reserved | ||
// [28:26] - probe type, see PseudoProbeType | ||
// [31:29] - reserved for probe attributes | ||
static uint32_t packProbeData(uint32_t Index, uint32_t Type) { | ||
assert(Index <= 0xFFFF && "Probe index too big to encode, exceeding 2^16"); | ||
assert(Type <= 0x7 && "Probe type too big to encode, exceeding 7"); | ||
return (Index << 3) | (Type << 26) | 0x7; | ||
} | ||
|
||
static uint32_t extractProbeIndex(uint32_t Value) { | ||
return (Value >> 3) & 0xFFFF; | ||
} | ||
|
||
static uint32_t extractProbeType(uint32_t Value) { | ||
return (Value >> 26) & 0x7; | ||
} | ||
|
||
static uint32_t extractProbeAttributes(uint32_t Value) { | ||
return (Value >> 29) & 0x7; | ||
} | ||
}; | ||
} // end namespace llvm | ||
|
||
#endif // LLVM_IR_PSEUDOPROBE_H |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
//===- PseudoProbeInserter.cpp - Insert annotation for callsite profiling -===// | ||
// | ||
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. | ||
// See https://llvm.org/LICENSE.txt for license information. | ||
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception | ||
// | ||
//===----------------------------------------------------------------------===// | ||
// | ||
// This file implements PseudoProbeInserter pass, which inserts pseudo probe | ||
// annotations for call instructions with a pseudo-probe-specific dwarf | ||
// discriminator. such discriminator indicates that the call instruction comes | ||
// with a pseudo probe, and the discriminator value holds information to | ||
// identify the corresponding counter. | ||
//===----------------------------------------------------------------------===// | ||
|
||
#include "llvm/CodeGen/MachineBasicBlock.h" | ||
#include "llvm/CodeGen/MachineFunctionPass.h" | ||
#include "llvm/CodeGen/MachineInstr.h" | ||
#include "llvm/CodeGen/TargetInstrInfo.h" | ||
#include "llvm/IR/DebugInfoMetadata.h" | ||
#include "llvm/IR/PseudoProbe.h" | ||
#include "llvm/InitializePasses.h" | ||
#include "llvm/Target/TargetMachine.h" | ||
#include <unordered_map> | ||
|
||
#define DEBUG_TYPE "pseudo-probe-inserter" | ||
|
||
using namespace llvm; | ||
|
||
namespace { | ||
class PseudoProbeInserter : public MachineFunctionPass { | ||
public: | ||
static char ID; | ||
|
||
PseudoProbeInserter() : MachineFunctionPass(ID) { | ||
initializePseudoProbeInserterPass(*PassRegistry::getPassRegistry()); | ||
} | ||
|
||
StringRef getPassName() const override { return "Pseudo Probe Inserter"; } | ||
|
||
void getAnalysisUsage(AnalysisUsage &AU) const override { | ||
AU.setPreservesAll(); | ||
MachineFunctionPass::getAnalysisUsage(AU); | ||
} | ||
|
||
bool runOnMachineFunction(MachineFunction &MF) override { | ||
const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo(); | ||
bool Changed = false; | ||
for (MachineBasicBlock &MBB : MF) { | ||
for (MachineInstr &MI : MBB) { | ||
if (MI.isCall()) { | ||
if (DILocation *DL = MI.getDebugLoc()) { | ||
auto Value = DL->getDiscriminator(); | ||
if (DILocation::isPseudoProbeDiscriminator(Value)) { | ||
BuildMI(MBB, MI, DL, TII->get(TargetOpcode::PSEUDO_PROBE)) | ||
.addImm(getFuncGUID(MF.getFunction().getParent(), DL)) | ||
.addImm( | ||
PseudoProbeDwarfDiscriminator::extractProbeIndex(Value)) | ||
.addImm( | ||
PseudoProbeDwarfDiscriminator::extractProbeType(Value)) | ||
.addImm(PseudoProbeDwarfDiscriminator::extractProbeAttributes( | ||
Value)); | ||
Changed = true; | ||
} | ||
} | ||
} | ||
} | ||
} | ||
|
||
return Changed; | ||
} | ||
|
||
private: | ||
uint64_t getFuncGUID(Module *M, DILocation *DL) { | ||
auto *SP = DL->getScope()->getSubprogram(); | ||
auto Name = SP->getLinkageName(); | ||
if (Name.empty()) | ||
Name = SP->getName(); | ||
return Function::getGUID(Name); | ||
} | ||
}; | ||
} // namespace | ||
|
||
char PseudoProbeInserter::ID = 0; | ||
INITIALIZE_PASS_BEGIN(PseudoProbeInserter, DEBUG_TYPE, | ||
"Insert pseudo probe annotations for value profiling", | ||
false, false) | ||
INITIALIZE_PASS_DEPENDENCY(TargetPassConfig) | ||
INITIALIZE_PASS_END(PseudoProbeInserter, DEBUG_TYPE, | ||
"Insert pseudo probe annotations for value profiling", | ||
false, false) | ||
|
||
FunctionPass *llvm::createPseudoProbeInserter() { | ||
return new PseudoProbeInserter(); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.