[CGData] Outlined Hash Tree #89792
Merged
[CGData] Outlined Hash Tree #89792
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kyulee-com
force-pushed
the
outlinedhashtree
branch
3 times, most recently
from
e4d31d4
to
e405e02
Compare
kyulee-com
changed the title
|
@xuanzh-meta Thanks for the review! I incorporated the fixes. |
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
96dc768
to
2c26019
Compare
ellishg
reviewed
May 3, 2024
|
@ellishg Thanks for the review! I incorporated the fixes. |
|
@llvm/pr-subscribers-backend-arm @llvm/pr-subscribers-llvm-support Author: Kyungwoo Lee (kyulee-com) ChangesThis defines the OutlinedHashTree class. This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753. Patch is 89.97 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/89792.diff 30 Files Affected:
diff --git a/llvm/include/llvm/CodeGenData/OutlinedHashTree.h b/llvm/include/llvm/CodeGenData/OutlinedHashTree.h
new file mode 100644
index 00000000000000..c40038cd8c5174
--- /dev/null
+++ b/llvm/include/llvm/CodeGenData/OutlinedHashTree.h
@@ -0,0 +1,98 @@
+//===- OutlinedHashTree.h --------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===---------------------------------------------------------------------===//
+//
+// This defines the OutlinedHashTree class. It contains sequences of stable
+// hash values of instructions that have been outlined. This OutlinedHashTree
+// can be used to track the outlined instruction sequences across modules.
+//
+//===---------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGENDATA_OUTLINEDHASHTREE_H
+#define LLVM_CODEGENDATA_OUTLINEDHASHTREE_H
+
+#include "llvm/ADT/StableHashing.h"
+#include "llvm/ObjectYAML/YAML.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <unordered_map>
+#include <vector>
+
+namespace llvm {
+
+/// A HashNode is an entry in an OutlinedHashTree, holding a hash value
+/// and a collection of Successors (other HashNodes). If a HashNode has
+/// a positive terminal value (Terminals > 0), it signifies the end of
+/// a hash sequence with that occurrence count.
+struct HashNode {
+ /// The hash value of the node.
+ stable_hash Hash = 0;
+ /// The number of terminals in the sequence ending at this node.
+ std::optional<unsigned> Terminals;
+ /// The successors of this node.
+ /// We don't use DenseMap as a stable_hash value can be tombstone.
+ std::unordered_map<stable_hash, std::unique_ptr<HashNode>> Successors;
+};
+
+class OutlinedHashTree {
+
+ using EdgeCallbackFn =
+ std::function<void(const HashNode *, const HashNode *)>;
+ using NodeCallbackFn = std::function<void(const HashNode *)>;
+
+ using HashSequence = SmallVector<stable_hash>;
+ using HashSequencePair = std::pair<HashSequence, unsigned>;
+
+public:
+ /// Walks every edge and node in the OutlinedHashTree and calls CallbackEdge
+ /// for the edges and CallbackNode for the nodes with the stable_hash for
+ /// the source and the stable_hash of the sink for an edge. These generic
+ /// callbacks can be used to traverse a OutlinedHashTree for the purpose of
+ /// print debugging or serializing it.
+ void walkGraph(NodeCallbackFn CallbackNode,
+ EdgeCallbackFn CallbackEdge = nullptr,
+ bool SortedWalk = false) const;
+
+ /// Release all hash nodes except the root hash node.
+ void clear() {
+ assert(getRoot()->Hash == 0 && !getRoot()->Terminals);
+ getRoot()->Successors.clear();
+ }
+
+ /// \returns true if the hash tree has only the root node.
+ bool empty() { return size() == 1; }
+
+ /// \returns the size of a OutlinedHashTree by traversing it. If
+ /// \p GetTerminalCountOnly is true, it only counts the terminal nodes
+ /// (meaning it returns the the number of hash sequences in the
+ /// OutlinedHashTree).
+ size_t size(bool GetTerminalCountOnly = false) const;
+
+ /// \returns the depth of a OutlinedHashTree by traversing it.
+ size_t depth() const;
+
+ /// \returns the root hash node of a OutlinedHashTree.
+ const HashNode *getRoot() const { return &Root; }
+ HashNode *getRoot() { return &Root; }
+
+ /// Inserts a \p Sequence into the this tree. The last node in the sequence
+ /// will increase Terminals.
+ void insert(const HashSequencePair &SequencePair);
+
+ /// Merge a \p OtherTree into this Tree.
+ void merge(const OutlinedHashTree *OtherTree);
+
+ /// \returns the matching count if \p Sequence exists in the OutlinedHashTree.
+ std::optional<unsigned> find(const HashSequence &Sequence) const;
+
+private:
+ HashNode Root;
+};
+
+} // namespace llvm
+
+#endif
diff --git a/llvm/include/llvm/CodeGenData/OutlinedHashTreeRecord.h b/llvm/include/llvm/CodeGenData/OutlinedHashTreeRecord.h
new file mode 100644
index 00000000000000..2960e319604489
--- /dev/null
+++ b/llvm/include/llvm/CodeGenData/OutlinedHashTreeRecord.h
@@ -0,0 +1,76 @@
+//===- OutlinedHashTreeRecord.h --------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===---------------------------------------------------------------------===//
+//
+// This defines the OutlinedHashTreeRecord class. This class holds the outlined
+// hash tree for both serialization and deserialization processes. It utilizes
+// two data formats for serialization: raw binary data and YAML.
+// These two formats can be used interchangeably.
+//
+//===---------------------------------------------------------------------===//
+
+#ifndef LLVM_CODEGENDATA_OUTLINEDHASHTREERECORD_H
+#define LLVM_CODEGENDATA_OUTLINEDHASHTREERECORD_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/CodeGenData/OutlinedHashTree.h"
+
+namespace llvm {
+
+/// HashNodeStable is the serialized, stable, and compact representation
+/// of a HashNode.
+struct HashNodeStable {
+ llvm::yaml::Hex64 Hash;
+ unsigned Terminals;
+ std::vector<unsigned> SuccessorIds;
+};
+
+using IdHashNodeStableMapTy = std::map<unsigned, HashNodeStable>;
+using IdHashNodeMapTy = DenseMap<unsigned, HashNode *>;
+using HashNodeIdMapTy = DenseMap<const HashNode *, unsigned>;
+
+struct OutlinedHashTreeRecord {
+ std::unique_ptr<OutlinedHashTree> HashTree;
+
+ OutlinedHashTreeRecord() { HashTree = std::make_unique<OutlinedHashTree>(); }
+ OutlinedHashTreeRecord(std::unique_ptr<OutlinedHashTree> HashTree)
+ : HashTree(std::move(HashTree)){};
+
+ /// Serialize the outlined hash tree to a raw_ostream.
+ void serialize(raw_ostream &OS) const;
+ /// Deserialize the outlined hash tree from a raw_ostream.
+ void deserialize(const unsigned char *&Ptr);
+ /// Serialize the outlined hash tree to a YAML stream.
+ void serializeYAML(yaml::Output &YOS) const;
+ /// Deserialize the outlined hash tree from a YAML stream.
+ void deserializeYAML(yaml::Input &YIS);
+
+ /// Merge the other outlined hash tree into this one.
+ void merge(const OutlinedHashTreeRecord &Other) {
+ HashTree->merge(Other.HashTree.get());
+ }
+
+ /// \returns true if the outlined hash tree is empty.
+ bool empty() const { return HashTree->empty(); }
+
+ /// Print the outlined hash tree in a YAML format.
+ void print(raw_ostream &OS = llvm::errs()) const {
+ yaml::Output YOS(OS);
+ serializeYAML(YOS);
+ }
+
+private:
+ /// Convert the outlined hash tree to stable data.
+ void convertToStableData(IdHashNodeStableMapTy &IdNodeStableMap) const;
+
+ /// Convert the stable data back to the outlined hash tree.
+ void convertFromStableData(const IdHashNodeStableMapTy &IdNodeStableMap);
+};
+
+} // end namespace llvm
+
+#endif // LLVM_CODEGENDATA_OUTLINEDHASHTREERECORD_H
diff --git a/llvm/include/llvm/Support/SuffixTree.h b/llvm/include/llvm/Support/SuffixTree.h
index 4940fbbf308d8b..37b73666404300 100644
--- a/llvm/include/llvm/Support/SuffixTree.h
+++ b/llvm/include/llvm/Support/SuffixTree.h
@@ -42,6 +42,9 @@ class SuffixTree {
/// Each element is an integer representing an instruction in the module.
ArrayRef<unsigned> Str;
+ /// Whether to consider leaf descendants or only leaf children.
+ bool OutlinerLeafDescendants;
+
/// A repeated substring in the tree.
struct RepeatedSubstring {
/// The length of the string.
@@ -130,11 +133,27 @@ class SuffixTree {
/// this step.
unsigned extend(unsigned EndIdx, unsigned SuffixesToAdd);
+ /// This vector contains all leaf nodes of this suffix tree. These leaf nodes
+ /// are identified using post-order depth-first traversal, so that the order
+ /// of these leaf nodes in the vector matches the order of the leaves in the
+ /// tree from left to right if one were to draw the tree on paper.
+ std::vector<SuffixTreeLeafNode *> LeafNodes;
+
+ /// Perform a post-order depth-first traversal of the tree and perform two
+ /// tasks during the traversal. The first is to populate LeafNodes, adding
+ /// nodes in order of the traversal. The second is to keep track of the leaf
+ /// descendants of every internal node by assigning values to LeftLeafIndex
+ /// and RightLefIndex fields of SuffixTreeNode for all internal nodes.
+ void setLeafNodes();
+
public:
/// Construct a suffix tree from a sequence of unsigned integers.
///
/// \param Str The string to construct the suffix tree for.
- SuffixTree(const ArrayRef<unsigned> &Str);
+ /// \param OutlinerLeafDescendants Whether to consider leaf descendants or
+ /// only leaf children (used by Machine Outliner).
+ SuffixTree(const ArrayRef<unsigned> &Str,
+ bool OutlinerLeafDescendants = false);
/// Iterator for finding all repeated substrings in the suffix tree.
struct RepeatedSubstringIterator {
@@ -154,6 +173,12 @@ class SuffixTree {
/// instruction lengths.
const unsigned MinLength = 2;
+ /// Vector of leaf nodes of the suffix tree.
+ const std::vector<SuffixTreeLeafNode *> &LeafNodes;
+
+ /// Whether to consider leaf descendants or only leaf children.
+ bool OutlinerLeafDescendants = !LeafNodes.empty();
+
/// Move the iterator to the next repeated substring.
void advance();
@@ -179,7 +204,10 @@ class SuffixTree {
return !(*this == Other);
}
- RepeatedSubstringIterator(SuffixTreeInternalNode *N) : N(N) {
+ RepeatedSubstringIterator(
+ SuffixTreeInternalNode *N,
+ const std::vector<SuffixTreeLeafNode *> &LeafNodes = {})
+ : N(N), LeafNodes(LeafNodes) {
// Do we have a non-null node?
if (!N)
return;
@@ -191,7 +219,7 @@ class SuffixTree {
};
typedef RepeatedSubstringIterator iterator;
- iterator begin() { return iterator(Root); }
+ iterator begin() { return iterator(Root, LeafNodes); }
iterator end() { return iterator(nullptr); }
};
diff --git a/llvm/include/llvm/Support/SuffixTreeNode.h b/llvm/include/llvm/Support/SuffixTreeNode.h
index 7d0d1cf0c58b95..84b590f2deb0cd 100644
--- a/llvm/include/llvm/Support/SuffixTreeNode.h
+++ b/llvm/include/llvm/Support/SuffixTreeNode.h
@@ -46,6 +46,17 @@ struct SuffixTreeNode {
/// the root to this node.
unsigned ConcatLen = 0;
+ /// These two indices give a range of indices for its leaf descendants.
+ /// Imagine drawing a tree on paper and assigning a unique index to each leaf
+ /// node in monotonically increasing order from left to right. This way of
+ /// numbering the leaf nodes allows us to associate a continuous range of
+ /// indices with each internal node. For example, if a node has leaf
+ /// descendants with indices i, i+1, ..., j, then its LeftLeafIdx is i and
+ /// its RightLeafIdx is j. These indices are for LeafNodes in the SuffixTree
+ /// class, which is constructed using post-order depth-first traversal.
+ unsigned LeftLeafIdx = EmptyIdx;
+ unsigned RightLeafIdx = EmptyIdx;
+
public:
// LLVM RTTI boilerplate.
NodeKind getKind() const { return Kind; }
@@ -56,6 +67,18 @@ struct SuffixTreeNode {
/// \returns the end index of this node.
virtual unsigned getEndIdx() const = 0;
+ /// \return the index of this node's left most leaf node.
+ unsigned getLeftLeafIdx() const;
+
+ /// \return the index of this node's right most leaf node.
+ unsigned getRightLeafIdx() const;
+
+ /// Set the index of the left most leaf node of this node to \p Idx.
+ void setLeftLeafIdx(unsigned Idx);
+
+ /// Set the index of the right most leaf node of this node to \p Idx.
+ void setRightLeafIdx(unsigned Idx);
+
/// Advance this node's StartIdx by \p Inc.
void incrementStartIdx(unsigned Inc);
@@ -168,4 +191,4 @@ struct SuffixTreeLeafNode : SuffixTreeNode {
virtual ~SuffixTreeLeafNode() = default;
};
} // namespace llvm
-#endif // LLVM_SUPPORT_SUFFIXTREE_NODE_H
\ No newline at end of file
+#endif // LLVM_SUPPORT_SUFFIXTREE_NODE_H
diff --git a/llvm/lib/CMakeLists.txt b/llvm/lib/CMakeLists.txt
index 74e2d03c07953d..2ac0b0dc026e16 100644
--- a/llvm/lib/CMakeLists.txt
+++ b/llvm/lib/CMakeLists.txt
@@ -10,6 +10,7 @@ add_subdirectory(InterfaceStub)
add_subdirectory(IRPrinter)
add_subdirectory(IRReader)
add_subdirectory(CodeGen)
+add_subdirectory(CodeGenData)
add_subdirectory(CodeGenTypes)
add_subdirectory(BinaryFormat)
add_subdirectory(Bitcode)
diff --git a/llvm/lib/CodeGen/MachineOutliner.cpp b/llvm/lib/CodeGen/MachineOutliner.cpp
index dc2f5ef15206e8..01a9966adb097a 100644
--- a/llvm/lib/CodeGen/MachineOutliner.cpp
+++ b/llvm/lib/CodeGen/MachineOutliner.cpp
@@ -121,6 +121,12 @@ static cl::opt<unsigned> OutlinerBenefitThreshold(
cl::desc(
"The minimum size in bytes before an outlining candidate is accepted"));
+static cl::opt<bool> OutlinerLeafDescendants(
+ "outliner-leaf-descendants", cl::init(true), cl::Hidden,
+ cl::desc("Consider all leaf descendants of internal nodes of the suffix "
+ "tree as candidates for outlining (if false, only leaf children "
+ "are considered)"));
+
namespace {
/// Maps \p MachineInstrs to unsigned integers and stores the mappings.
@@ -576,7 +582,7 @@ void MachineOutliner::emitOutlinedFunctionRemark(OutlinedFunction &OF) {
void MachineOutliner::findCandidates(
InstructionMapper &Mapper, std::vector<OutlinedFunction> &FunctionList) {
FunctionList.clear();
- SuffixTree ST(Mapper.UnsignedVec);
+ SuffixTree ST(Mapper.UnsignedVec, OutlinerLeafDescendants);
// First, find all of the repeated substrings in the tree of minimum length
// 2.
@@ -593,7 +599,11 @@ void MachineOutliner::findCandidates(
unsigned NumDiscarded = 0;
unsigned NumKept = 0;
#endif
- for (const unsigned &StartIdx : RS.StartIndices) {
+ // Sort the start indices so that we can efficiently check if candidates
+ // overlap with each other in MachineOutliner::findCandidates().
+ SmallVector<unsigned> SortedStartIndices(RS.StartIndices);
+ llvm::sort(SortedStartIndices);
+ for (const unsigned &StartIdx : SortedStartIndices) {
// Trick: Discard some candidates that would be incompatible with the
// ones we've already found for this sequence. This will save us some
// work in candidate selection.
@@ -616,17 +626,15 @@ void MachineOutliner::findCandidates(
// * End before the other starts
// * Start after the other ends
unsigned EndIdx = StartIdx + StringLen - 1;
- auto FirstOverlap = find_if(
- CandidatesForRepeatedSeq, [StartIdx, EndIdx](const Candidate &C) {
- return EndIdx >= C.getStartIdx() && StartIdx <= C.getEndIdx();
- });
- if (FirstOverlap != CandidatesForRepeatedSeq.end()) {
+ if (CandidatesForRepeatedSeq.size() > 0 &&
+ StartIdx <= CandidatesForRepeatedSeq.back().getEndIdx()) {
#ifndef NDEBUG
++NumDiscarded;
- LLVM_DEBUG(dbgs() << " .. DISCARD candidate @ [" << StartIdx
- << ", " << EndIdx << "]; overlaps with candidate @ ["
- << FirstOverlap->getStartIdx() << ", "
- << FirstOverlap->getEndIdx() << "]\n");
+ LLVM_DEBUG(dbgs() << " .. DISCARD candidate @ [" << StartIdx << ", "
+ << EndIdx << "]; overlaps with candidate @ ["
+ << CandidatesForRepeatedSeq.back().getStartIdx()
+ << ", " << CandidatesForRepeatedSeq.back().getEndIdx()
+ << "]\n");
#endif
continue;
}
@@ -828,10 +836,12 @@ bool MachineOutliner::outline(Module &M,
<< "\n");
bool OutlinedSomething = false;
- // Sort by benefit. The most beneficial functions should be outlined first.
+ // Sort by priority where priority := getNotOutlinedCost / getOutliningCost.
+ // The function with highest priority should be outlined first.
stable_sort(FunctionList,
[](const OutlinedFunction &LHS, const OutlinedFunction &RHS) {
- return LHS.getBenefit() > RHS.getBenefit();
+ return LHS.getNotOutlinedCost() * RHS.getOutliningCost() >
+ RHS.getNotOutlinedCost() * LHS.getOutliningCost();
});
// Walk over each function, outlining them as we go along. Functions are
diff --git a/llvm/lib/CodeGenData/CMakeLists.txt b/llvm/lib/CodeGenData/CMakeLists.txt
new file mode 100644
index 00000000000000..3ba90f96cc86d4
--- /dev/null
+++ b/llvm/lib/CodeGenData/CMakeLists.txt
@@ -0,0 +1,14 @@
+add_llvm_component_library(LLVMCodeGenData
+ OutlinedHashTree.cpp
+ OutlinedHashTreeRecord.cpp
+
+ ADDITIONAL_HEADER_DIRS
+ ${LLVM_MAIN_INCLUDE_DIR}/llvm/CodeGenData
+
+ DEPENDS
+ intrinsics_gen
+
+ LINK_COMPONENTS
+ Core
+ Support
+ )
diff --git a/llvm/lib/CodeGenData/OutlinedHashTree.cpp b/llvm/lib/CodeGenData/OutlinedHashTree.cpp
new file mode 100644
index 00000000000000..3111f7e910dc30
--- /dev/null
+++ b/llvm/lib/CodeGenData/OutlinedHashTree.cpp
@@ -0,0 +1,132 @@
+//===-- OutlinedHashTree.cpp ----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// An OutlinedHashTree is a Trie that contains sequences of stable hash values
+// of instructions that have been outlined. This OutlinedHashTree can be used
+// to understand the outlined instruction sequences collected across modules.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGenData/OutlinedHashTree.h"
+
+#include <stack>
+#include <tuple>
+
+#define DEBUG_TYPE "outlined-hash-tree"
+
+using namespace llvm;
+
+void OutlinedHashTree::walkGraph(NodeCallbackFn CallbackNode,
+ EdgeCallbackFn CallbackEdge,
+ bool SortedWalk) const {
+ SmallVector<const HashNode *> Stack;
+ Stack.emplace_back(getRoot());
+
+ while (!Stack.empty()) {
+ const auto *Current = Stack.pop_back_val();
+ if (CallbackNode)
+ CallbackNode(Current);
+
+ auto HandleNext = [&](const HashNode *Next) {
+ if (CallbackEdge)
+ CallbackEdge(Current, Next);
+ Stack.emplace_back(Next);
+ };
+ if (SortedWalk) {
+ std::map<stable_hash, const HashNode *> SortedSuccessors;
+ for (const auto &P : Current->Successors)
+ SortedSuccessors[P.first] = P.second.get();
+ for (const auto &P : SortedSuccessors)
+ HandleNext(P.second);
+ } else {
+ for (const auto &P : Current->Successors)
+ HandleNext(P.second.get());
+ }
+ }
+}
+
+size_t OutlinedHashTree::size(bool GetTerminalCountOnly) const {
+ size_t Size = 0;
+ walkGraph([&Size, GetTerminalCountOnly](const HashNode *N) {
+ Size += (N && (!GetTerminalCountOnly || N->Terminals));
+ });
+ return Size;
+}
+
+size_t OutlinedHashTree::depth() const {
+ size_t Size = 0;
+ std::unordered_map<const HashNode *, size_t> DepthMap;
+ walkGraph([&Size, &DepthMap](
+ const HashNode *N) { Size = std::max(Size, DepthMap[N]); },
+ [&DepthMap](const HashNode *Src, const HashNode *Dst) {
+ size_t Depth = DepthMap[Src];
+ DepthMap[Dst] = Depth + 1;
+ });
+ return Size;
+}
+
+void OutlinedHashTree::insert(const HashSequencePair &SequencePair) {
+ const auto &Sequence = SequencePair.first;
+ unsigned Count = SequencePair.second;
+ HashNode *Current = getRoot();
+
+ for (stable_hash StableHash : Sequence) {
+ auto I = Current->Successors.find(StableHash);
+ if (I == Current->Successors.end()) {
+ std::unique_ptr<HashNode> Next = std::make_unique<HashNode>();
+ HashNode *NextPtr = Next.get();
+ NextPtr->Hash = StableHash;
+ Current->Successors.emplace(StableHash, std::move(Next));
+ Current = NextPtr;
+ } else
+ Current = I->second.get();
+ }
+ if (Count)
+ Current->Terminals = (Current->Terminals ? *Current->Terminals : 0) + Count;
+}
+
+voi...
[truncated]
|
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
338a5d3
to
f2eff82
Compare
|
To eliminate any confusion for reviewing this PR, I now removed the stacked commits related to Part 1: https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-1-fulllto-part-2-thinlto-nolto-to-come/78732 |
kyulee-com
force-pushed
the
outlinedhashtree
branch
3 times, most recently
from
5205a52
to
c05bdf6
Compare
Merged
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
c05bdf6
to
b4b520f
Compare
ellishg
reviewed
Jun 18, 2024
thevinster
approved these changes
Jun 25, 2024
| @@ -0,0 +1,14 @@ | |||
| add_llvm_component_library(LLVMCodeGenData | |||
There was a problem hiding this comment.
Where are you using this library? It seems to only be defined and only used in the test.
There was a problem hiding this comment.
yeah. It is currently used only in the test, but the subsequent PRs will use it.
This defines the OutlinedHashTree class. It contains sequences of stable hash values of instructions that have been outlined. This OutlinedHashTree can be used to track the outlined instruction sequences across modules. A trie structure is used in its implementation, allowing for a compact sharing of common prefixes.
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
b2f1d9f
to
c6fa20b
Compare
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
c6fa20b
to
37d2944
Compare
kyulee-com
force-pushed
the
outlinedhashtree
branch
from
37d2944
to
9e23ff5
Compare
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/168/builds/776 Here is the relevant piece of the build log for the reference: |
kyulee-com
added a commit
that referenced
this pull request
Jul 7, 2024
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/145/builds/536 Here is the relevant piece of the build log for the reference: |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/64/builds/228 Here is the relevant piece of the build log for the reference: |
| [&NodeIdMap](const HashNode *Current) { | ||
| size_t Index = NodeIdMap.size(); | ||
| NodeIdMap[Current] = Index; | ||
| assert(Index = NodeIdMap.size() + 1 && |
There was a problem hiding this comment.
@kyulee-com This line looks suspicious and generates a warning:
llvm/lib/CodeGenData/OutlinedHashTreeRecord.cpp:122:22: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
122 | assert(Index = NodeIdMap.size() + 1 &&
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
123 | "Expected size of NodeMap to increment by 1");
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Do you want I create an issue for that or you'll fix that in your WIP?
There was a problem hiding this comment.
Thanks for catching this. I will fix it soon!
kyulee-com
added a commit
to kyulee-com/llvm-project
that referenced
this pull request
Jul 10, 2024
kyulee-com
added a commit
that referenced
this pull request
Jul 11, 2024
aaryanshukla
pushed a commit
to aaryanshukla/llvm-project
that referenced
this pull request
Jul 14, 2024
kyulee-com
added a commit
that referenced
this pull request
Jul 23, 2024
The llvm-cgdata tool has been introduced to handle reading and writing of codegen data. This data includes an optimistic codegen summary that can be utilized to enhance subsequent codegen. Currently, the tool supports saving and restoring the outlined hash tree, facilitating machine function outlining across modules. Additional codegen summaries can be incorporated into separate sections as required. This patch primarily establishes basic support for the reader and writer, similar to llvm-profdata. The high-level operations of llvm-cgdata are as follows: 1. It reads local raw codegen data from a custom section (for example, __llvm_outline) embedded in native binary files 2. It merges local raw codegen data into an indexed codegen data, complete with a suitable header. 3. It handles reading and writing of the indexed codegen data into a standalone file. This depends on #89792. This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753. --------- Co-authored-by: Kyungwoo Lee <kyulee@fb.com>
sgundapa
pushed a commit
to sgundapa/upstream_effort
that referenced
this pull request
Jul 23, 2024
The llvm-cgdata tool has been introduced to handle reading and writing of codegen data. This data includes an optimistic codegen summary that can be utilized to enhance subsequent codegen. Currently, the tool supports saving and restoring the outlined hash tree, facilitating machine function outlining across modules. Additional codegen summaries can be incorporated into separate sections as required. This patch primarily establishes basic support for the reader and writer, similar to llvm-profdata. The high-level operations of llvm-cgdata are as follows: 1. It reads local raw codegen data from a custom section (for example, __llvm_outline) embedded in native binary files 2. It merges local raw codegen data into an indexed codegen data, complete with a suitable header. 3. It handles reading and writing of the indexed codegen data into a standalone file. This depends on llvm#89792. This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753. --------- Co-authored-by: Kyungwoo Lee <kyulee@fb.com>
yuxuanchen1997
pushed a commit
that referenced
this pull request
Jul 25, 2024
Summary: The llvm-cgdata tool has been introduced to handle reading and writing of codegen data. This data includes an optimistic codegen summary that can be utilized to enhance subsequent codegen. Currently, the tool supports saving and restoring the outlined hash tree, facilitating machine function outlining across modules. Additional codegen summaries can be incorporated into separate sections as required. This patch primarily establishes basic support for the reader and writer, similar to llvm-profdata. The high-level operations of llvm-cgdata are as follows: 1. It reads local raw codegen data from a custom section (for example, __llvm_outline) embedded in native binary files 2. It merges local raw codegen data into an indexed codegen data, complete with a suitable header. 3. It handles reading and writing of the indexed codegen data into a standalone file. This depends on #89792. This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753. --------- Co-authored-by: Kyungwoo Lee <kyulee@fb.com> Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: https://phabricator.intern.facebook.com/D60251410
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This defines the OutlinedHashTree class.
It contains sequences of stable hash values of instructions that have been outlined. This OutlinedHashTree can be used to track the outlined instruction sequences across modules. A trie structure is used in its implementation, allowing for a compact sharing of common prefixes.
This is a patch for https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.