[libc] Remove automemcpy folder #118781
Merged
[libc] Remove automemcpy folder #118781
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The build is currently broken and we don't have the resources to keep it up to date :-/
|
@llvm/pr-subscribers-libc Author: Guillaume Chatelet (gchatelet) ChangesThe build is currently broken and we don't have the resources to keep it up to date :-/ Patch is 91.29 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/118781.diff 17 Files Affected:
diff --git a/libc/benchmarks/CMakeLists.txt b/libc/benchmarks/CMakeLists.txt
index 52e3f942d16ea3..5cd612a14b5402 100644
--- a/libc/benchmarks/CMakeLists.txt
+++ b/libc/benchmarks/CMakeLists.txt
@@ -212,5 +212,3 @@ target_link_libraries(libc.benchmarks.memory_functions.opt_host
benchmark_main
)
llvm_update_compile_flags(libc.benchmarks.memory_functions.opt_host)
-
-add_subdirectory(automemcpy)
diff --git a/libc/benchmarks/automemcpy/CMakeLists.txt b/libc/benchmarks/automemcpy/CMakeLists.txt
deleted file mode 100644
index ef9b4218c8d617..00000000000000
--- a/libc/benchmarks/automemcpy/CMakeLists.txt
+++ /dev/null
@@ -1,12 +0,0 @@
-if(NOT LIBC_BUILD_AUTOMEMCPY)
- return ()
-endif()
-
-if(NOT LLVM_WITH_Z3)
- MESSAGE(FATAL_ERROR "Building llvm-libc automemcpy requires Z3")
-endif()
-
-set(LIBC_AUTOMEMCPY_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/include)
-
-add_subdirectory(lib)
-add_subdirectory(unittests)
diff --git a/libc/benchmarks/automemcpy/README.md b/libc/benchmarks/automemcpy/README.md
deleted file mode 100644
index 8583368993ef06..00000000000000
--- a/libc/benchmarks/automemcpy/README.md
+++ /dev/null
@@ -1,111 +0,0 @@
-This folder contains an implementation of [automemcpy: A framework for automatic generation of fundamental memory operations](https://research.google/pubs/pub50338/).
-
-It uses the [Z3 theorem prover](https://github.com/Z3Prover/z3) to enumerate a subset of valid memory function implementations. These implementations are then materialized as C++ code and can be [benchmarked](../) against various [size distributions](../distributions). This process helps the design of efficient implementations for a particular environnement (size distribution, processor or custom compilation options).
-
-This is not enabled by default, as it is mostly useful when working on tuning the library implementation. To build it, use `LIBC_BUILD_AUTOMEMCPY=ON` (see below).
-
-## Prerequisites
-
-You may need to install `Z3` from source if it's not available on your system.
-Here we show instructions to install it into `<Z3_INSTALL_DIR>`.
-You may need to `sudo` to `make install`.
-
-```shell
-mkdir -p ~/git
-cd ~/git
-git clone https://github.com/Z3Prover/z3.git
-python scripts/mk_make.py --prefix=<Z3_INSTALL_DIR>
-cd build
-make -j
-make install
-```
-
-## Configuration
-
-```shell
-mkdir -p <BUILD_DIR>
-cd <LLVM_PROJECT_DIR>/llvm
-cmake -DCMAKE_C_COMPILER=/usr/bin/clang \
- -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
- -DLLVM_ENABLE_PROJECTS="libc" \
- -DLLVM_ENABLE_Z3_SOLVER=ON \
- -DLLVM_Z3_INSTALL_DIR=<Z3_INSTALL_DIR> \
- -DLIBC_BUILD_AUTOMEMCPY=ON \
- -DCMAKE_BUILD_TYPE=Release \
- -B<BUILD_DIR>
-```
-
-## Targets and compilation
-
-There are three main CMake targets
- 1. `automemcpy_implementations`
- - runs `Z3` and materializes valid memory functions as C++ code, a message will display its ondisk location.
- - the source code is then compiled using the native host optimizations (i.e. `-march=native` or `-mcpu=native` depending on the architecture).
- 2. `automemcpy`
- - the binary that benchmarks the autogenerated implementations.
- 3. `automemcpy_result_analyzer`
- - the binary that analyses the benchmark results.
-
-You may only compile the binaries as they both pull the autogenerated code as a dependency.
-
-```shell
-make -C <BUILD_DIR> -j automemcpy automemcpy_result_analyzer
-```
-
-## Running the benchmarks
-
-Make sure to save the results of the benchmark as a json file.
-
-```shell
-<BUILD_DIR>/bin/automemcpy --benchmark_out_format=json --benchmark_out=<RESULTS_DIR>/results.json
-```
-
-### Additional useful options
-
-
- - `--benchmark_min_time=.2`
-
- By default, each function is benchmarked for at least one second, here we lower it to 200ms.
-
- - `--benchmark_filter="BM_Memset|BM_Bzero"`
-
- By default, all functions are benchmarked, here we restrict them to `memset` and `bzero`.
-
-Other options might be useful, use `--help` for more information.
-
-## Analyzing the benchmarks
-
-Analysis is performed by running `automemcpy_result_analyzer` on one or more json result files.
-
-```shell
-<BUILD_DIR>/bin/automemcpy_result_analyzer <RESULTS_DIR>/results.json
-```
-
-What it does:
- 1. Gathers all throughput values for each function / distribution pair and picks the median one.\
- This allows picking a representative value over many runs of the benchmark. Please make sure all the runs happen under similar circumstances.
-
- 2. For each distribution, look at the span of throughputs for functions of the same type (e.g. For distribution `A`, memcpy throughput spans from 2GiB/s to 5GiB/s).
-
- 3. For each distribution, give a normalized score to each function (e.g. For distribution `A`, function `M` scores 0.65).\
- This score is then turned into a grade `EXCELLENT`, `VERY_GOOD`, `GOOD`, `PASSABLE`, `INADEQUATE`, `MEDIOCRE`, `BAD` - so that each distribution categorizes how function perform according to them.
-
- 4. A [Majority Judgement](https://en.wikipedia.org/wiki/Majority_judgment) process is then used to categorize each function. This enables finer analysis of how distributions agree on which function is better. In the following example, `Function_1` and `Function_2` are rated `EXCELLENT` but looking at the grade's distribution might help decide which is best.
-
-| | EXCELLENT | VERY_GOOD | GOOD | PASSABLE | INADEQUATE | MEDIOCRE | BAD |
-|------------|:---------:|:---------:|:----:|:--------:|:----------:|:--------:|:---:|
-| Function_1 | 7 | 1 | 2 | | | | |
-| Function_2 | 6 | 4 | | | | | |
-
-The tool outputs the histogram of grades for each function. In case of tie, other dimensions might help decide (e.g. code size, performance on other microarchitectures).
-
-```
-EXCELLENT |█▁▂ | Function_0
-EXCELLENT |█▅ | Function_1
-VERY_GOOD |▂█▁ ▁ | Function_2
-GOOD | ▁█▄ | Function_3
-PASSABLE | ▂▆▄█ | Function_4
-INADEQUATE | ▃▃█▁ | Function_5
-MEDIOCRE | █▆▁| Function_6
-BAD | ▁▁█| Function_7
-```
diff --git a/libc/benchmarks/automemcpy/include/automemcpy/CodeGen.h b/libc/benchmarks/automemcpy/include/automemcpy/CodeGen.h
deleted file mode 100644
index 389e8249f93993..00000000000000
--- a/libc/benchmarks/automemcpy/include/automemcpy/CodeGen.h
+++ /dev/null
@@ -1,26 +0,0 @@
-//===-- C++ code generation from NamedFunctionDescriptors -------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LIBC_BENCHMARKS_AUTOMEMCPY_CODEGEN_H
-#define LIBC_BENCHMARKS_AUTOMEMCPY_CODEGEN_H
-
-#include "automemcpy/FunctionDescriptor.h"
-#include <llvm/ADT/ArrayRef.h>
-#include <llvm/Support/raw_ostream.h>
-#include <vector>
-
-namespace llvm {
-namespace automemcpy {
-
-// This function serializes the array of FunctionDescriptors as a C++ file.
-void Serialize(raw_ostream &Stream, ArrayRef<FunctionDescriptor> FD);
-
-} // namespace automemcpy
-} // namespace llvm
-
-#endif // LIBC_BENCHMARKS_AUTOMEMCPY_CODEGEN_H
diff --git a/libc/benchmarks/automemcpy/include/automemcpy/FunctionDescriptor.h b/libc/benchmarks/automemcpy/include/automemcpy/FunctionDescriptor.h
deleted file mode 100644
index 65477d9d72a0ee..00000000000000
--- a/libc/benchmarks/automemcpy/include/automemcpy/FunctionDescriptor.h
+++ /dev/null
@@ -1,159 +0,0 @@
-//===-- Pod structs to describe a memory function----------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_COMMON_H
-#define LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_COMMON_H
-
-#include <climits>
-#include <cstddef>
-#include <llvm/ADT/ArrayRef.h>
-#include <llvm/ADT/Hashing.h>
-#include <llvm/ADT/StringRef.h>
-#include <optional>
-#include <tuple>
-
-namespace llvm {
-namespace automemcpy {
-
-// Boilerplate code to be able to sort and hash types.
-#define COMPARABLE_AND_HASHABLE(T, ...) \
- inline auto asTuple() const { return std::tie(__VA_ARGS__); } \
- bool operator==(const T &O) const { return asTuple() == O.asTuple(); } \
- bool operator<(const T &O) const { return asTuple() < O.asTuple(); } \
- struct Hasher { \
- std::size_t operator()(const T &K) const { \
- return llvm::hash_value(K.asTuple()); \
- } \
- };
-
-// Represents the maximum value for the size parameter of a memory function.
-// This is an `int` so we can use it as an expression in Z3.
-// It also allows for a more readable and compact representation when storing
-// the SizeSpan in the autogenerated C++ file.
-static constexpr int kMaxSize = INT_MAX;
-
-// This mimics the `Arg` type in libc/src/string/memory_utils/elements.h without
-// having to depend on it.
-enum class AlignArg { _1, _2, ARRAY_SIZE };
-
-// Describes a range of sizes.
-// We use the begin/end representation instead of first/last to allow for empty
-// range (i.e. Begin == End)
-struct SizeSpan {
- size_t Begin = 0;
- size_t End = 0;
-
- COMPARABLE_AND_HASHABLE(SizeSpan, Begin, End)
-};
-
-// Describes a contiguous region.
-// In such a region all sizes are handled individually.
-// e.g. with Span = {0, 2};
-// if(size == 0) return Handle<0>();
-// if(size == 1) return Handle<1>();
-struct Contiguous {
- SizeSpan Span;
-
- COMPARABLE_AND_HASHABLE(Contiguous, Span)
-};
-
-// This struct represents a range of sizes over which to use an overlapping
-// strategy. An overlapping strategy of size N handles all sizes from N to 2xN.
-// The span may represent several contiguous overlaps.
-// e.g. with Span = {16, 128};
-// if(size >= 16 and size < 32) return Handle<Overlap<16>>();
-// if(size >= 32 and size < 64) return Handle<Overlap<32>>();
-// if(size >= 64 and size < 128) return Handle<Overlap<64>>();
-struct Overlap {
- SizeSpan Span;
-
- COMPARABLE_AND_HASHABLE(Overlap, Span)
-};
-
-// Describes a region using a loop handling BlockSize bytes at a time. The
-// remaining bytes of the loop are handled with an overlapping operation.
-struct Loop {
- SizeSpan Span;
- size_t BlockSize = 0;
-
- COMPARABLE_AND_HASHABLE(Loop, Span, BlockSize)
-};
-
-// Same as `Loop` but starts by aligning a buffer on `Alignment` bytes.
-// A first operation handling 'Alignment` bytes is performed followed by a
-// sequence of Loop.BlockSize bytes operation. The Loop starts processing from
-// the next aligned byte in the chosen buffer. The remaining bytes of the loop
-// are handled with an overlapping operation.
-struct AlignedLoop {
- Loop Loop;
- size_t Alignment = 0; // Size of the alignment.
- AlignArg AlignTo = AlignArg::_1; // Which buffer to align.
-
- COMPARABLE_AND_HASHABLE(AlignedLoop, Loop, Alignment, AlignTo)
-};
-
-// Some processors offer special instruction to handle the memory function
-// completely, we refer to such instructions as accelerators.
-struct Accelerator {
- SizeSpan Span;
-
- COMPARABLE_AND_HASHABLE(Accelerator, Span)
-};
-
-// The memory functions are assembled out of primitives that can be implemented
-// with regular scalar operations (SCALAR), with the help of vector or bitcount
-// instructions (NATIVE) or by deferring it to the compiler (BUILTIN).
-enum class ElementTypeClass {
- SCALAR,
- NATIVE,
- BUILTIN,
-};
-
-// A simple enum to categorize which function is being implemented.
-enum class FunctionType {
- MEMCPY,
- MEMCMP,
- BCMP,
- MEMSET,
- BZERO,
-};
-
-// This struct describes the skeleton of the implementation, it does not go into
-// every detail but is enough to uniquely identify the implementation.
-struct FunctionDescriptor {
- FunctionType Type;
- std::optional<Contiguous> Contiguous;
- std::optional<Overlap> Overlap;
- std::optional<Loop> Loop;
- std::optional<AlignedLoop> AlignedLoop;
- std::optional<Accelerator> Accelerator;
- ElementTypeClass ElementClass;
-
- COMPARABLE_AND_HASHABLE(FunctionDescriptor, Type, Contiguous, Overlap, Loop,
- AlignedLoop, Accelerator, ElementClass)
-
- inline size_t id() const { return llvm::hash_value(asTuple()); }
-};
-
-// Same as above but with the function name.
-struct NamedFunctionDescriptor {
- StringRef Name;
- FunctionDescriptor Desc;
-};
-
-template <typename T> llvm::hash_code hash_value(const ArrayRef<T> &V) {
- return llvm::hash_combine_range(V.begin(), V.end());
-}
-template <typename T> llvm::hash_code hash_value(const T &O) {
- return llvm::hash_value(O.asTuple());
-}
-
-} // namespace automemcpy
-} // namespace llvm
-
-#endif /* LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_COMMON_H */
diff --git a/libc/benchmarks/automemcpy/include/automemcpy/RandomFunctionGenerator.h b/libc/benchmarks/automemcpy/include/automemcpy/RandomFunctionGenerator.h
deleted file mode 100644
index 28756e8f86c0ee..00000000000000
--- a/libc/benchmarks/automemcpy/include/automemcpy/RandomFunctionGenerator.h
+++ /dev/null
@@ -1,62 +0,0 @@
-//===-- Generate random but valid function descriptors ---------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_RANDOM_FUNCTION_GENERATOR_H
-#define LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_RANDOM_FUNCTION_GENERATOR_H
-
-#include "automemcpy/FunctionDescriptor.h"
-#include <cstddef>
-#include <cstdint>
-#include <llvm/ADT/ArrayRef.h>
-#include <llvm/ADT/StringRef.h>
-#include <optional>
-#include <vector>
-#include <z3++.h>
-
-namespace llvm {
-namespace automemcpy {
-
-// Holds the state for the constraint solver.
-// It implements a single method that returns the next valid description.
-struct RandomFunctionGenerator {
- RandomFunctionGenerator();
-
- // Get the next valid FunctionDescriptor or std::nullopt.
- std::optional<FunctionDescriptor> next();
-
-private:
- // Returns an expression where `Variable` is forced to be one of the `Values`.
- z3::expr inSetConstraint(z3::expr &Variable, ArrayRef<int> Values) const;
- // Add constaints to `Begin` and `End` so that they are:
- // - between 0 and kMaxSize (inclusive)
- // - ordered (begin<=End)
- // - amongst a set of predefined values.
- void addBoundsAndAnchors(z3::expr &Begin, z3::expr &End);
- // Add constraints to make sure that the loop block size is amongst a set of
- // predefined values. Also makes sure that the loop that the loop is iterated
- // at least `LoopMinIter` times.
- void addLoopConstraints(const z3::expr &LoopBegin, const z3::expr &LoopEnd,
- z3::expr &LoopBlockSize, int LoopMinIter);
-
- z3::context Context;
- z3::solver Solver;
-
- z3::expr Type;
- z3::expr ContiguousBegin, ContiguousEnd;
- z3::expr OverlapBegin, OverlapEnd;
- z3::expr LoopBegin, LoopEnd, LoopBlockSize;
- z3::expr AlignedLoopBegin, AlignedLoopEnd, AlignedLoopBlockSize,
- AlignedAlignment, AlignedArg;
- z3::expr AcceleratorBegin, AcceleratorEnd;
- z3::expr ElementClass;
-};
-
-} // namespace automemcpy
-} // namespace llvm
-
-#endif /* LLVM_LIBC_BENCHMARKS_AUTOMEMCPY_RANDOM_FUNCTION_GENERATOR_H */
diff --git a/libc/benchmarks/automemcpy/include/automemcpy/ResultAnalyzer.h b/libc/benchmarks/automemcpy/include/automemcpy/ResultAnalyzer.h
deleted file mode 100644
index d4bf2725827671..00000000000000
--- a/libc/benchmarks/automemcpy/include/automemcpy/ResultAnalyzer.h
+++ /dev/null
@@ -1,109 +0,0 @@
-//===-- Analyze benchmark JSON files ----------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LIBC_BENCHMARKS_AUTOMEMCPY_RESULTANALYZER_H
-#define LIBC_BENCHMARKS_AUTOMEMCPY_RESULTANALYZER_H
-
-#include "automemcpy/FunctionDescriptor.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/StringMap.h"
-#include <array>
-#include <vector>
-
-namespace llvm {
-namespace automemcpy {
-
-// A Grade as in the Majority Judgment voting system.
-struct Grade {
- enum GradeEnum {
- EXCELLENT,
- VERY_GOOD,
- GOOD,
- PASSABLE,
- INADEQUATE,
- MEDIOCRE,
- BAD,
- ARRAY_SIZE,
- };
-
- // Returns a human readable string of the enum.
- static StringRef getString(const GradeEnum &GE);
-
- // Turns 'Score' into a GradeEnum.
- static GradeEnum judge(double Score);
-};
-
-// A 'GradeEnum' indexed array with counts for each grade.
-using GradeHistogram = std::array<size_t, Grade::ARRAY_SIZE>;
-
-// Identifies a Function by its name and type. Used as a key in a map.
-struct FunctionId {
- StringRef Name;
- FunctionType Type;
- COMPARABLE_AND_HASHABLE(FunctionId, Type, Name)
-};
-
-struct PerDistributionData {
- std::vector<double> BytesPerSecondSamples;
- double BytesPerSecondMedian; // Median of samples for this distribution.
- double BytesPerSecondMean; // Mean of samples for this distribution.
- double BytesPerSecondVariance; // Variance of samples for this distribution.
- double Score; // Normalized score for this distribution.
- Grade::GradeEnum Grade; // Grade for this distribution.
-};
-
-struct FunctionData {
- FunctionId Id;
- StringMap<PerDistributionData> PerDistributionData;
- double ScoresGeoMean; // Geomean of scores for each distribution.
- GradeHistogram GradeHisto = {}; // GradeEnum indexed array
- Grade::GradeEnum FinalGrade = Grade::BAD; // Overall grade for this function
-};
-
-// Identifies a Distribution by its name. Used as a key in a map.
-struct DistributionId {
- StringRef Name;
- COMPARABLE_AND_HASHABLE(DistributionId, Name)
-};
-
-// Identifies a Sample by its distribution and function. Used as a key in a map.
-struct SampleId {
- FunctionId Function;
- DistributionId Distribution;
- COMPARABLE_AND_HASHABLE(SampleId, Function.Type, Function.Name,
- Distribution.Name)
-};
-
-// The type of Samples as reported by the Google Benchmark's JSON result file.
-// We are only interested in the "iteration" samples, the "aggregate" ones
-// represent derived metrics such as 'mean' or 'median'.
-enum class SampleType { UNKNOWN, ITERATION, AGGREGATE };
-
-// A SampleId with an associated measured throughput.
-struct Sample {
- SampleId Id;
- SampleType Type = SampleType::UNKNOWN;
- double BytesPerSecond = 0;
-};
-
-// This function collects Samples that belong to the same distribution and
-// function and retains the median one. It then stores each of them into a
-// 'FunctionData' and returns them as a vector.
-std::vector<FunctionData> getThroughputs(ArrayRef<Sample> Samples);
-
-// Normalize the function's throughput per distribution.
-void fillScores(MutableArrayRef<FunctionData> Functions);
-
-// Convert scores into Grades, stores an histogram of Grade for each functions
-// and cast a median grade for the function.
-void castVotes(MutableArrayRef<FunctionData> Functions);
-
-} // namespace automemcpy
-} // namespace llvm
-
-#endif // LIBC_BENCHMARKS_AUTOMEMCPY_RESULTANALYZER_H
diff --git a/libc/benchmarks/automemcpy/lib/CMakeLists.txt b/libc/benchmarks/automemcpy/lib/CMakeLists.txt
deleted file mode 100644
index e66b9045b6074a..00000000000000
--- a/libc/benchmarks/automemcpy/lib/CMakeLists.txt
+++ /dev/null
@@ -1,37 +0,0 @@
-add_library(automemcpy_codegen CodeGen.cpp)
-target_link_libraries(automemcpy_codegen PUBLIC LLVMSupport)
-target_include_directories(automemcpy_codegen PUBLIC ${LIBC_AUTOMEMCPY_INCLUDE_DIR})
-llvm_update_compile_flags(automemcpy_codegen)
-
-add_executable(automemcpy...
[truncated]
|
michaelrj-google
approved these changes
Dec 5, 2024
vonosmas
approved these changes
Dec 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The build is currently broken and we don't have the resources to keep it up to date :-/