Skip to content

Commit

Permalink
Use ccache in Github Actions (#3218)
Browse files Browse the repository at this point in the history
This works very well for CUDA and HIP builds. However, the benefit is
relatively small for GCC, Clang and Mac builds because most of their
time are spent on clang-tidy and running tests.

There are two remaining issues. Ccache works with Intel classic compiler
icpc, but it does not work with the Intel SYCL compiler, icpx. On
Windows, MSVC works, but I cannot get ClangCl to work. We can revisit
these two issues in the future.

In earlier versions of this PR, we compiled ccache from source because
the package version on Ubuntu 20.04 is too old. Because of that, we had
to move `env: {CXXFLAGS: ...}` down to AMReX's `Build & Install` section
to avoid the flags being picked up when compiling ccache. In the current
version, we download the binary version of ccache 4.8. However, we did
not move the env setup back. It seems to make sense to keep it in the
`Build & Install` section.
 
To use github cache, we need to provide a primary key and optionally
restore keys. If the primary key is found, it will be used, but the
change to the cache during the CI will NOT be saved at the end of the
job. That's how github cache works. If it is not found, partially
matched restore keys will be considered and the latest one will be used.
At the end, the cache (whether it's updated or not) will be saved under
the name of the primary key. Below is an example of our setup.
```
    with:
        path: ~/.cache
        key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
        restore-keys: |
             ccache-clang.yml-${{ env.cache-name }}-git-
```
Here, we use the workflow string and job string as part of the names for
the keys. We also include the git commit hash `${{github.sha}}` in the
name of the primary key. In this setup, the CI will always fail to find
the primary key. This is what we want because it guarantees the latest
ccache data will be saved. Furthermore, the most recently saved cache
from the last CI job will be used to restore the cache during
initialization.
  • Loading branch information
WeiqunZhang committed Mar 29, 2023
1 parent 831d148 commit b41c441
Show file tree
Hide file tree
Showing 11 changed files with 664 additions and 102 deletions.
64 changes: 56 additions & 8 deletions .github/workflows/clang.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# -Wno-c++17-extensions: Clang complains about nodiscard if the standard is not set to c++17.

name: Linux Clang
name: LinuxClang

on: [push, pull_request]

Expand All @@ -14,15 +14,28 @@ jobs:
library_clang:
name: Clang@7.0 C++17 SP NOMPI Debug [lib]
runs-on: ubuntu-20.04
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -Wnon-virtual-dtor"}
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: |
.github/workflows/dependencies/dependencies_clang.sh 7
.github/workflows/dependencies/dependencies_clang-tidy.sh 12
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -Wnon-virtual-dtor"}
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=125M
ccache -z
mkdir build
cd build
cmake .. \
Expand All @@ -40,7 +53,8 @@ jobs:
-DCMAKE_CXX_COMPILER=$(which clang++-7) \
-DCMAKE_Fortran_COMPILER=$(which gfortran) \
-DAMReX_CLANG_TIDY=ON \
-DAMReX_CLANG_TIDY_WERROR=ON
-DAMReX_CLANG_TIDY_WERROR=ON \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
make -j 2
make install
make test_install
Expand All @@ -50,19 +64,34 @@ jobs:
ctest --output-on-failure
ccache -s
tests_clang:
name: Clang@14.0 C++17 SP Particles DP Mesh Debug [tests]
runs-on: ubuntu-22.04
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -O1 -Wnon-virtual-dtor"}
# It's too slow with -O0
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: |
.github/workflows/dependencies/dependencies_clang.sh 14
.github/workflows/dependencies/dependencies_clang-tidy.sh 14
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -O1 -Wnon-virtual-dtor"}
# It's too slow with -O0
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=125M
ccache -z
mkdir build
cd build
cmake .. \
Expand All @@ -79,24 +108,43 @@ jobs:
-DCMAKE_CXX_COMPILER=$(which clang++-14) \
-DCMAKE_Fortran_COMPILER=$(which gfortran) \
-DAMReX_CLANG_TIDY=ON \
-DAMReX_CLANG_TIDY_WERROR=ON
-DAMReX_CLANG_TIDY_WERROR=ON \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
make -j 2
ctest --output-on-failure -E GhostsAndVirtuals
ccache -s
# Build 2D libamrex with configure
configure-2d:
name: Clang@7.0 NOMPI Release [configure 2D]
name: Clang NOMPI Release [configure 2D]
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: |
.github/workflows/dependencies/dependencies_clang.sh 14
.github/workflows/dependencies/dependencies_clang-tidy.sh 14
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=125M
ccache -z
./configure --dim 2 --with-fortran no --comp llvm --with-mpi no
make -j2 WARN_ALL=TRUE WARN_ERROR=TRUE XTRA_CXXFLAGS="-fno-operator-names" \
USE_CLANG_TIDY=TRUE CLANG_TIDY=clang-tidy-14 CLANG_TIDY_WARN_ERROR=TRUE
USE_CLANG_TIDY=TRUE CLANG_TIDY=clang-tidy-14 CLANG_TIDY_WARN_ERROR=TRUE \
CCACHE=ccache
make install
ccache -s
68 changes: 60 additions & 8 deletions .github/workflows/cuda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,27 @@ jobs:
tests-cuda11:
name: CUDA@11.2 GNU@9.3.0 C++17 Release [tests]
runs-on: ubuntu-20.04
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches"}
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: .github/workflows/dependencies/dependencies_nvcc11.sh
run: |
.github/workflows/dependencies/dependencies_nvcc11.sh
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches"}
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=500M
ccache -z
export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}
which nvcc || echo "nvcc not in PATH!"
Expand All @@ -35,21 +49,39 @@ jobs:
-DCMAKE_Fortran_COMPILER=$(which gfortran) \
-DAMReX_CUDA_ARCH=7.0 \
-DAMReX_CUDA_ERROR_CROSS_EXECUTION_SPACE_CALL=ON \
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON \
-DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
cmake --build build -j 2
ccache -s
# Build libamrex and all tests with NVHPC (recent supported)
tests-nvhpc23-1-nvcc:
name: NVHPC@23.1 NVCC/NVC++ C++17 Release [tests]
runs-on: ubuntu-20.04
env: {CXXFLAGS: "-Werror -Wall -Wextra -Wpedantic -Wshadow --diag_suppress=code_is_unreachable"}
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: .github/workflows/dependencies/dependencies_nvhpc23-1.sh
run: |
.github/workflows/dependencies/dependencies_nvhpc23-1.sh
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
env: {CXXFLAGS: "-Werror -Wall -Wextra -Wpedantic -Wshadow --diag_suppress=code_is_unreachable"}
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=500M
ccache -z
source /etc/profile.d/modules.sh
module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/23.1
Expand All @@ -75,25 +107,45 @@ jobs:
-DCMAKE_Fortran_COMPILER=$(which nvfortran) \
-DAMReX_CUDA_ARCH=8.0 \
-DAMReX_CUDA_ERROR_CROSS_EXECUTION_SPACE_CALL=ON \
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON \
-DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
cmake --build build -j 2
ccache -s
# Build 3D libamrex cuda build with configure
configure-3d-cuda:
name: CUDA@11.2 GNU@9.3.0 [configure 3D]
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- name: Dependencies
run: .github/workflows/dependencies/dependencies_nvcc11.sh
run: |
.github/workflows/dependencies/dependencies_nvcc11.sh
.github/workflows/dependencies/dependencies_ccache.sh
- name: Set Up Cache
uses: actions/cache@v3
with:
path: ~/.cache
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
restore-keys: |
ccache-${{ github.workflow }}-${{ github.job }}-git-
- name: Build & Install
run: |
export CCACHE_COMPRESS=1
export CCACHE_COMPRESSLEVEL=10
export CCACHE_MAXSIZE=500M
ccache -z
export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
./configure --dim 3 --with-cuda yes --enable-eb yes --enable-xsdk-defaults yes --with-fortran no
#
# /home/runner/work/amrex/amrex/Src/Base/AMReX_GpuLaunchGlobal.H:16:41: error: unused parameter ‘f0’ [-Werror=unused-parameter]
# 16 | AMREX_GPU_GLOBAL void launch_global (L f0) { f0(); }
#
make -j2 WARN_ALL=TRUE WARN_ERROR=TRUE XTRA_CXXFLAGS="-fno-operator-names -Wno-unused-parameter"
make -j2 WARN_ALL=TRUE WARN_ERROR=TRUE XTRA_CXXFLAGS="-fno-operator-names -Wno-unused-parameter" CCACHE=ccache
make install
ccache -s
11 changes: 11 additions & 0 deletions .github/workflows/dependencies/dependencies_ccache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/env bash

if [[ $# -eq 2 ]]; then
CVER=$1
else
CVER=4.8
fi

wget https://github.com/ccache/ccache/releases/download/v${CVER}/ccache-${CVER}-linux-x86_64.tar.xz
tar xvf ccache-${CVER}-linux-x86_64.tar.xz
sudo cp -f ccache-${CVER}-linux-x86_64/ccache /usr/local/bin/
Loading

0 comments on commit b41c441

Please sign in to comment.