Compile Faster with the Program Repository and ccache
This page shows the Program Repository (Pepo) and ccache together achieve much faster build.
- Introduce the Repo Project
- Repo Build Results for the Upstream LLVM and Clang Project
- Introduce the Repo and ccache Project
- Repo and ccache Build Results for the Upstream LLVM and Clang Project
- Conclusion
llvm-project-prepo (Repo) is an LLVM/Clang compiler with program repository support. It aims to improve turnaround times and eliminate duplication of effort by centralising program data in a repository. It reduces compilation time by reusing previously optimised functions and global variable fragments, including both sharing them across multiple translation units and reusing them even when other portions of the relevant source files have changed. The Repo implementation in LLVM is illustrated in Fig.1.
The process of building LLVM with the Repo compiler is documented here.
For now, we use an additional tool, repo2obj, to create ELF files from the program repository which enables us to link on Linux. We compared three full builds: 1. ELF as a baseline, 2. First run with Repo (no database). This records objects into the repository and measures the overhead for a build, and 3. Second run with Repo after ninja clean (with existing database). This build uses objects stored in the repository.
This comparison was run with the Repo release compiler (llvm-project-prepo commit: 96e2af12 and pstore commit: 2022f6e4).
Fig.2 shows the overall CPU time take to compile LLVM+Clang. The compilation time is measured by adding -ftime-trace to all files and analysed with ClangBuildAnalyzer. Enabling the program repository adds a small overhead for hashing objects and storing them to the database. We could see that the Repo can shorten backend compilation time by 77% for the second time LLVM build compared to the ELF.
| Title | ELF | prepo 1st run | prepo 2nd build |
|---|---|---|---|
| Frontend (s) | 16279.8 | 16169.3 | 15691.4 |
| Backend (s) | 8856.9 | 10427.4 | 2065.9 |
| Total (s) | 25136.7 | 26596.7 | 17757.3 |
As an example of the benefit, Fig.3 shows -ftime-trace output compiling SemaDeclAttr.cpp for the first time Repo build: one of the longest files to compile in LLVM backend. We could see most of the remaining backend time is spent in "CallGraph Pass Manager" and "RepoMetadataGeneration". Fig. 4 shows the same file for the second time Repo build. The program repo compiler hashes each object in the pass "RepoMetadataGeneration". A following pass "RepoPruning" marks objects as “available externally” linkage type if they have already been compiled by the program repository. Another following pass "Dead Global Elimination", removes functions and global variables if they are not referenced by other global objects.
ccache is a compiler caching tool which uses textual hashing of the source files. It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again. When used to build a large project, the ccache cache can quickly become invalid due to the frequency of header file changes. The ccache implementation is illustrated in Fig.5.
ccache reduces the build time for unchanged files, whereas Repo reduces the build time even for changed files. Using Repo+ccache together achieves much faster builds. The implementation in LLVM is illustrated in Fig.6.
The different cmake flags are shown in the Table.1. We ran 4 different test runs: 1. ELF as a baseline, 2. Build with Repo only, 3. Build with ccache only, and 4. Build with Repo+ccache. The compilation time is measured by adding -ftime-trace to all files and analysed with ClangBuildAnalyzer.
| Title | Triple | Extra CMake Flags |
|---|---|---|
| ELF | x86_64-pc-linux-gnu | -DCMAKE_CXX_FLAGS=-ftime-trace |
| Repo | x86_64-pc-linux-gnu-repo | -DCMAKE_CXX_FLAGS=-ftime-trace |
| ccache | x86_64-pc-linux-gnu | -DCMAKE_CXX_FLAGS=-ftime-trace -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache |
| ccache+repo | x86_64-pc-linux-gnu-repo | -DCMAKE_CXX_FLAGS=-ftime-trace -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache |
The benefits of building the LLVM+Clang project with release configuration are shown below. While Fig.7 shows the build time, Fig.8 shows the rebuild time taken to compile LLVM+Clang at points through its commit history. For six consecutive days, the first commit of each day is selected to test the compilation time. Each of those commits is built in turn, from left to right, keeping the cache contents and Repo database between them. For example, the cache contents and Repo database were kept between the 4th, 8th, 12th column of Fig 7. and so on. So there were four separate sequences of experiments done for that chart.
Now Fig.9 shows the percentage of the compilation time for multiple builds compared to ELF when using Repo+ccache whereas Fig.10 shows the rebuild time comparatively. Except the first build, Repo+ccache together achieves much faster builds than using either as individuals for both build and rebuild. For the build, the total compilation time has up to 62% reduction whilst the rebuild has up to 69% reduction.
-
The program repository can shorten compiler back-end time by 77% for the second time LLVM build compared to the ELF.
-
For both build and rebuild, Repo+ccache together achieves much faster builds than using either of them individually. For the build, the total compilation time has up to 62% reduction whilst the rebuild has up to 69% reduction.