Skip to content

Commit 7841e21

Browse files
rlavaeerlavaee
committed
Let -basic-block-sections=labels emit basicblock metadata in a new .bb_addr_map section, instead of emitting special unary-encoded symbols.
This patch introduces the new .bb_addr_map section feature which allows us to emit the bits needed for mapping binary profiles to basic blocks into a separate section. The format of the emitted data is represented as follows. It includes a header for every function: | Address of the function | -> 8 bytes (pointer size) | Number of basic blocks in this function (>0) | -> ULEB128 The header is followed by a BB record for every basic block. These records are ordered in the same order as MachineBasicBlocks are placed in the function. Each BB Info is structured as follows: | Offset of the basic block relative to function begin | -> ULEB128 | Binary size of the basic block | -> ULEB128 | BB metadata | -> ULEB128 [ MBB.isReturn() OR MBB.hasTailCall() << 1 OR MBB.isEHPad() << 2 ] The new feature will replace the existing "BB labels" functionality with -basic-block-sections=labels. The .bb_addr_map section scrubs the specially-encoded BB symbols from the binary and makes it friendly to profilers and debuggers. Furthermore, the new feature reduces the binary size overhead from 70% bloat to only 12%. For more information and results please refer to the RFC: https://lists.llvm.org/pipermail/llvm-dev/2020-July/143512.html Reviewed By: MaskRay, snehasish Differential Revision: https://reviews.llvm.org/D85408
1 parent ce6dd97 commit 7841e21

File tree

13 files changed

+184
-129
lines changed

13 files changed

+184
-129
lines changed

‎clang/docs/UsersManual.rst

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1700,9 +1700,12 @@ are listed below.
17001700
17011701
**-fbasic-block-sections=[labels, all, list=<arg>, none]**
17021702

1703-
Controls whether Clang emits a label for each basic block. Further, with
1704-
values "all" and "list=arg", each basic block or a subset of basic blocks
1705-
can be placed in its own unique section.
1703+
Controls how Clang emits text sections for basic blocks. With values ``all``
1704+
and ``list=<arg>``, each basic block or a subset of basic blocks can be placed
1705+
in its own unique section. With the "labels" value, normal text sections are
1706+
emitted, but a ``.bb_addr_map`` section is emitted which includes address
1707+
offsets for each basic block in the program, relative to the parent function
1708+
address.
17061709

17071710
With the ``list=<arg>`` option, a file containing the subset of basic blocks
17081711
that need to placed in unique sections can be specified. The format of the

‎clang/test/CodeGen/basic-block-sections.c

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,11 @@
11
// REQUIRES: x86-registered-target
22

3-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -o - < %s | FileCheck %s --check-prefix=PLAIN
4-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -fbasic-block-sections=all -fbasic-block-sections=none -o - < %s | FileCheck %s --check-prefix=PLAIN
3+
// RUN: %clang_cc1 -triple x86_64 -S -o - < %s | FileCheck %s --check-prefix=PLAIN
4+
// RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -fbasic-block-sections=none -o - < %s | FileCheck %s --check-prefix=PLAIN
55

6-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -fbasic-block-sections=labels -o - < %s | FileCheck %s --check-prefix=BB_LABELS
7-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -fbasic-block-sections=all -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_ALL
8-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -fbasic-block-sections=list=%S/Inputs/basic-block-sections.funcnames -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_LIST
9-
// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -S -fbasic-block-sections=all -funique-basic-block-section-names -o - < %s | FileCheck %s --check-prefix=UNIQUE
6+
// RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_ALL
7+
// RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=list=%S/Inputs/basic-block-sections.funcnames -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_LIST
8+
// RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -funique-basic-block-section-names -o - < %s | FileCheck %s --check-prefix=UNIQUE
109

1110
int world(int a) {
1211
if (a > 10)
@@ -26,12 +25,6 @@ int another(int a) {
2625
// PLAIN-NOT: section
2726
// PLAIN: world:
2827
//
29-
// BB_LABELS-NOT: section
30-
// BB_LABELS: world:
31-
// BB_LABELS: a.BB.world:
32-
// BB_LABELS: aa.BB.world:
33-
// BB_LABELS: a.BB.another:
34-
//
3528
// BB_WORLD: .section .text.world,"ax",@progbits{{$}}
3629
// BB_WORLD: world:
3730
// BB_WORLD: .section .text.world,"ax",@progbits,unique

‎llvm/include/llvm/CodeGen/AsmPrinter.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,6 +342,8 @@ class AsmPrinter : public MachineFunctionPass {
342342

343343
void emitStackSizeSection(const MachineFunction &MF);
344344

345+
void emitBBAddrMapSection(const MachineFunction &MF);
346+
345347
void emitRemarksSection(remarks::RemarkStreamer &RS);
346348

347349
enum CFIMoveType { CFI_M_None, CFI_M_EH, CFI_M_Debug };

‎llvm/include/llvm/CodeGen/MachineFunction.h

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -510,9 +510,6 @@ class MachineFunction {
510510

511511
void setBBSectionsType(BasicBlockSection V) { BBSectionsType = V; }
512512

513-
/// Creates basic block Labels for this function.
514-
void createBBLabels();
515-
516513
/// Assign IsBeginSection IsEndSection fields for basic blocks in this
517514
/// function.
518515
void assignBeginEndSections();

‎llvm/include/llvm/MC/MCObjectFileInfo.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -338,6 +338,8 @@ class MCObjectFileInfo {
338338

339339
MCSection *getStackSizesSection(const MCSection &TextSec) const;
340340

341+
MCSection *getBBAddrMapSection(const MCSection &TextSec) const;
342+
341343
// ELF specific sections.
342344
MCSection *getDataRelROSection() const { return DataRelROSection; }
343345
const MCSection *getMergeableConst4Section() const {

‎llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

Lines changed: 62 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1023,6 +1023,46 @@ void AsmPrinter::emitFrameAlloc(const MachineInstr &MI) {
10231023
MCConstantExpr::create(FrameOffset, OutContext));
10241024
}
10251025

1026+
/// Returns the BB metadata to be emitted in the bb_addr_map section for a given
1027+
/// basic block. This can be used to capture more precise profile information.
1028+
/// We use the last 3 bits (LSBs) to ecnode the following information:
1029+
/// * (1): set if return block (ret or tail call).
1030+
/// * (2): set if ends with a tail call.
1031+
/// * (3): set if exception handling (EH) landing pad.
1032+
/// The remaining bits are zero.
1033+
static unsigned getBBAddrMapMetadata(const MachineBasicBlock &MBB) {
1034+
const TargetInstrInfo *TII = MBB.getParent()->getSubtarget().getInstrInfo();
1035+
return ((unsigned)MBB.isReturnBlock()) |
1036+
((!MBB.empty() && TII->isTailCall(MBB.back())) << 1) |
1037+
(MBB.isEHPad() << 2);
1038+
}
1039+
1040+
void AsmPrinter::emitBBAddrMapSection(const MachineFunction &MF) {
1041+
MCSection *BBAddrMapSection =
1042+
getObjFileLowering().getBBAddrMapSection(*MF.getSection());
1043+
assert(BBAddrMapSection && ".bb_addr_map section is not initialized.");
1044+
1045+
const MCSymbol *FunctionSymbol = getFunctionBegin();
1046+
1047+
OutStreamer->PushSection();
1048+
OutStreamer->SwitchSection(BBAddrMapSection);
1049+
OutStreamer->emitSymbolValue(FunctionSymbol, getPointerSize());
1050+
// Emit the total number of basic blocks in this function.
1051+
OutStreamer->emitULEB128IntValue(MF.size());
1052+
// Emit BB Information for each basic block in the funciton.
1053+
for (const MachineBasicBlock &MBB : MF) {
1054+
const MCSymbol *MBBSymbol =
1055+
MBB.pred_empty() ? FunctionSymbol : MBB.getSymbol();
1056+
// Emit the basic block offset.
1057+
emitLabelDifferenceAsULEB128(MBBSymbol, FunctionSymbol);
1058+
// Emit the basic block size. When BBs have alignments, their size cannot
1059+
// always be computed from their offsets.
1060+
emitLabelDifferenceAsULEB128(MBB.getEndSymbol(), MBBSymbol);
1061+
OutStreamer->emitULEB128IntValue(getBBAddrMapMetadata(MBB));
1062+
}
1063+
OutStreamer->PopSection();
1064+
}
1065+
10261066
void AsmPrinter::emitStackSizeSection(const MachineFunction &MF) {
10271067
if (!MF.getTarget().Options.EmitStackSizeSection)
10281068
return;
@@ -1179,34 +1219,26 @@ void AsmPrinter::emitFunctionBody() {
11791219
}
11801220

11811221
// We must emit temporary symbol for the end of this basic block, if either
1182-
// we have BBLabels enabled and we want to emit size directive for the BBs,
1183-
// or if this basic blocks marks the end of a section (except the section
1184-
// containing the entry basic block as the end symbol for that section is
1185-
// CurrentFnEnd).
1186-
if ((MAI->hasDotTypeDotSizeDirective() && MF->hasBBLabels()) ||
1187-
(MBB.isEndSection() && !MBB.sameSection(&MF->front())))
1222+
// we have BBLabels enabled or if this basic blocks marks the end of a
1223+
// section (except the section containing the entry basic block as the end
1224+
// symbol for that section is CurrentFnEnd).
1225+
if (MF->hasBBLabels() ||
1226+
(MAI->hasDotTypeDotSizeDirective() && MBB.isEndSection() &&
1227+
!MBB.sameSection(&MF->front())))
11881228
OutStreamer->emitLabel(MBB.getEndSymbol());
11891229

1190-
// Helper for emitting the size directive associated with a basic block
1191-
// symbol.
1192-
auto emitELFSizeDirective = [&](MCSymbol *SymForSize) {
1193-
const MCExpr *SizeExp = MCBinaryExpr::createSub(
1194-
MCSymbolRefExpr::create(MBB.getEndSymbol(), OutContext),
1195-
MCSymbolRefExpr::create(SymForSize, OutContext), OutContext);
1196-
OutStreamer->emitELFSize(SymForSize, SizeExp);
1197-
};
1198-
1199-
// Emit size directive for the size of each basic block, if BBLabels is
1200-
// enabled.
1201-
if (MAI->hasDotTypeDotSizeDirective() && MF->hasBBLabels())
1202-
emitELFSizeDirective(MBB.getSymbol());
1203-
1204-
// Emit size directive for the size of each basic block section once we
1205-
// get to the end of that section.
12061230
if (MBB.isEndSection()) {
1231+
// The size directive for the section containing the entry block is
1232+
// handled separately by the function section.
12071233
if (!MBB.sameSection(&MF->front())) {
1208-
if (MAI->hasDotTypeDotSizeDirective())
1209-
emitELFSizeDirective(CurrentSectionBeginSym);
1234+
if (MAI->hasDotTypeDotSizeDirective()) {
1235+
// Emit the size directive for the basic block section.
1236+
const MCExpr *SizeExp = MCBinaryExpr::createSub(
1237+
MCSymbolRefExpr::create(MBB.getEndSymbol(), OutContext),
1238+
MCSymbolRefExpr::create(CurrentSectionBeginSym, OutContext),
1239+
OutContext);
1240+
OutStreamer->emitELFSize(CurrentSectionBeginSym, SizeExp);
1241+
}
12101242
MBBSectionRanges[MBB.getSectionIDNum()] =
12111243
MBBSectionRange{CurrentSectionBeginSym, MBB.getEndSymbol()};
12121244
}
@@ -1298,6 +1330,11 @@ void AsmPrinter::emitFunctionBody() {
12981330
HI.Handler->endFunction(MF);
12991331
}
13001332

1333+
// Emit section containing BB address offsets and their metadata, when
1334+
// BB labels are requested for this function.
1335+
if (MF->hasBBLabels())
1336+
emitBBAddrMapSection(*MF);
1337+
13011338
// Emit section containing stack size metadata.
13021339
emitStackSizeSection(*MF);
13031340

@@ -1807,7 +1844,7 @@ void AsmPrinter::SetupMachineFunction(MachineFunction &MF) {
18071844
F.hasFnAttribute("function-instrument") ||
18081845
F.hasFnAttribute("xray-instruction-threshold") ||
18091846
needFuncLabelsForEHOrDebugInfo(MF) || NeedsLocalForSize ||
1810-
MF.getTarget().Options.EmitStackSizeSection) {
1847+
MF.getTarget().Options.EmitStackSizeSection || MF.hasBBLabels()) {
18111848
CurrentFnBegin = createTempSymbol("func_begin");
18121849
if (NeedsLocalForSize)
18131850
CurrentFnSymForSize = CurrentFnBegin;

‎llvm/lib/CodeGen/BasicBlockSections.cpp

Lines changed: 5 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,11 @@
4848
// Basic Block Labels
4949
// ==================
5050
//
51-
// With -fbasic-block-sections=labels, or when a basic block is placed in a
52-
// unique section, it is labelled with a symbol. This allows easy mapping of
53-
// virtual addresses from PMU profiles back to the corresponding basic blocks.
54-
// Since the number of basic blocks is large, the labeling bloats the symbol
55-
// table sizes and the string table sizes significantly. While the binary size
56-
// does increase, it does not affect performance as the symbol table is not
57-
// loaded in memory during run-time. The string table size bloat is kept very
58-
// minimal using a unary naming scheme that uses string suffix compression. The
59-
// basic blocks for function foo are named "a.BB.foo", "aa.BB.foo", ... This
60-
// turns out to be very good for string table sizes and the bloat in the string
61-
// table size for a very large binary is ~8 %. The naming also allows using
62-
// the --symbol-ordering-file option in LLD to arbitrarily reorder the
63-
// sections.
51+
// With -fbasic-block-sections=labels, we emit the offsets of BB addresses of
52+
// every function into a .bb_addr_map section. Along with the function symbols,
53+
// this allows for mapping of virtual addresses in PMU profiles back to the
54+
// corresponding basic blocks. This logic is implemented in AsmPrinter. This
55+
// pass only assigns the BBSectionType of every function to ``labels``.
6456
//
6557
//===----------------------------------------------------------------------===//
6658

@@ -304,7 +296,6 @@ bool BasicBlockSections::runOnMachineFunction(MachineFunction &MF) {
304296

305297
if (BBSectionsType == BasicBlockSection::Labels) {
306298
MF.setBBSectionsType(BBSectionsType);
307-
MF.createBBLabels();
308299
return true;
309300
}
310301

@@ -314,7 +305,6 @@ bool BasicBlockSections::runOnMachineFunction(MachineFunction &MF) {
314305
FuncBBClusterInfo))
315306
return true;
316307
MF.setBBSectionsType(BBSectionsType);
317-
MF.createBBLabels();
318308
assignSections(MF, FuncBBClusterInfo);
319309

320310
// We make sure that the cluster including the entry basic block precedes all

‎llvm/lib/CodeGen/MIRParser/MIRParser.cpp

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -451,10 +451,8 @@ MIRParserImpl::initializeMachineFunction(const yaml::MachineFunction &YamlMF,
451451
}
452452
// Check Basic Block Section Flags.
453453
if (MF.getTarget().getBBSectionsType() == BasicBlockSection::Labels) {
454-
MF.createBBLabels();
455454
MF.setBBSectionsType(BasicBlockSection::Labels);
456455
} else if (MF.hasBBSections()) {
457-
MF.createBBLabels();
458456
MF.assignBeginEndSections();
459457
}
460458
PFS.SM = &SM;

‎llvm/lib/CodeGen/MachineBasicBlock.cpp

Lines changed: 5 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -60,28 +60,11 @@ MCSymbol *MachineBasicBlock::getSymbol() const {
6060
if (!CachedMCSymbol) {
6161
const MachineFunction *MF = getParent();
6262
MCContext &Ctx = MF->getContext();
63-
auto Prefix = Ctx.getAsmInfo()->getPrivateLabelPrefix();
6463

65-
assert(getNumber() >= 0 && "cannot get label for unreachable MBB");
66-
67-
// We emit a non-temporary symbol for every basic block if we have BBLabels
68-
// or -- with basic block sections -- when a basic block begins a section.
69-
// With basic block symbols, we use a unary encoding which can
70-
// compress the symbol names significantly. For basic block sections where
71-
// this block is the first in a cluster, we use a non-temp descriptive name.
72-
// Otherwise we fall back to use temp label.
73-
if (MF->hasBBLabels()) {
74-
auto Iter = MF->getBBSectionsSymbolPrefix().begin();
75-
if (getNumber() < 0 ||
76-
getNumber() >= (int)MF->getBBSectionsSymbolPrefix().size())
77-
report_fatal_error("Unreachable MBB: " + Twine(getNumber()));
78-
// The basic blocks for function foo are named a.BB.foo, aa.BB.foo, and
79-
// so on.
80-
std::string Prefix(Iter + 1, Iter + getNumber() + 1);
81-
std::reverse(Prefix.begin(), Prefix.end());
82-
CachedMCSymbol =
83-
Ctx.getOrCreateSymbol(Twine(Prefix) + ".BB." + Twine(MF->getName()));
84-
} else if (MF->hasBBSections() && isBeginSection()) {
64+
// We emit a non-temporary symbol -- with a descriptive name -- if it begins
65+
// a section (with basic block sections). Otherwise we fall back to use temp
66+
// label.
67+
if (MF->hasBBSections() && isBeginSection()) {
8568
SmallString<5> Suffix;
8669
if (SectionID == MBBSectionID::ColdSectionID) {
8770
Suffix += ".cold";
@@ -92,6 +75,7 @@ MCSymbol *MachineBasicBlock::getSymbol() const {
9275
}
9376
CachedMCSymbol = Ctx.getOrCreateSymbol(MF->getName() + Suffix);
9477
} else {
78+
const StringRef Prefix = Ctx.getAsmInfo()->getPrivateLabelPrefix();
9579
CachedMCSymbol = Ctx.getOrCreateSymbol(Twine(Prefix) + "BB" +
9680
Twine(MF->getFunctionNumber()) +
9781
"_" + Twine(getNumber()));

‎llvm/lib/CodeGen/MachineFunction.cpp

Lines changed: 0 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -341,33 +341,6 @@ void MachineFunction::RenumberBlocks(MachineBasicBlock *MBB) {
341341
MBBNumbering.resize(BlockNo);
342342
}
343343

344-
/// This is used with -fbasic-block-sections or -fbasicblock-labels option.
345-
/// A unary encoding of basic block labels is done to keep ".strtab" sizes
346-
/// small.
347-
void MachineFunction::createBBLabels() {
348-
const TargetInstrInfo *TII = getSubtarget().getInstrInfo();
349-
this->BBSectionsSymbolPrefix.resize(getNumBlockIDs(), 'a');
350-
for (auto MBBI = begin(), E = end(); MBBI != E; ++MBBI) {
351-
assert(
352-
(MBBI->getNumber() >= 0 && MBBI->getNumber() < (int)getNumBlockIDs()) &&
353-
"BasicBlock number was out of range!");
354-
// 'a' - Normal block.
355-
// 'r' - Return block.
356-
// 'l' - Landing Pad.
357-
// 'L' - Return and landing pad.
358-
bool isEHPad = MBBI->isEHPad();
359-
bool isRetBlock = MBBI->isReturnBlock() && !TII->isTailCall(MBBI->back());
360-
char type = 'a';
361-
if (isEHPad && isRetBlock)
362-
type = 'L';
363-
else if (isEHPad)
364-
type = 'l';
365-
else if (isRetBlock)
366-
type = 'r';
367-
BBSectionsSymbolPrefix[MBBI->getNumber()] = type;
368-
}
369-
}
370-
371344
/// This method iterates over the basic blocks and assigns their IsBeginSection
372345
/// and IsEndSection fields. This must be called after MBB layout is finalized
373346
/// and the SectionID's are assigned to MBBs.

0 commit comments

Comments
 (0)