Skip to content

ci: make ccache effective and observable, shallow checkout, add cache… - #7756

Open
MrLi000001 wants to merge 1 commit into
deepmodeling:developfrom
MrLi000001:ci/ccache-observability
Open

ci: make ccache effective and observable, shallow checkout, add cache…#7756
MrLi000001 wants to merge 1 commit into
deepmodeling:developfrom
MrLi000001:ci/ccache-observability

Conversation

@MrLi000001

Copy link
Copy Markdown

… warmer

Measured on the self-hosted runners, the Build step of the same workflow ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a full-history checkout took ~12 min on cache-cold runners.

  • Set ccache --max-size=30G (default ~10G is tight for -O3 -g builds) and print ccache -s before/after the build so hit rates are visible in every run log (if: always()).
  • fetch-depth: 0 -> 1; CMake only runs 'git log -1' for commit info.
  • Add cache_warmer.yml: on every push to develop, build (without tests) on both the X64 and gpu pools so /tmp/ccache stays primed with the latest develop. This prevents the first PR runs after a big refactor merge from all paying a full rebuild.

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Fix #

Unit Tests and/or Case Tests for my changes

  • Commands run:
  • Result summary:
  • Checks not run, with reason:

What's changed?

  • Example: brief summary of the user-visible or developer-facing change.

Governance Notes

  • INPUT/docs changes:
  • Core module impact:
  • Exceptions requested:

… warmer

Measured on the self-hosted runners, the Build step of the same workflow
ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a
full-history checkout took ~12 min on cache-cold runners.

- Set ccache --max-size=30G (default ~10G is tight for -O3 -g builds)
  and print ccache -s before/after the build so hit rates are visible
  in every run log (if: always()).
- fetch-depth: 0 -> 1; CMake only runs 'git log -1' for commit info.
- Add cache_warmer.yml: on every push to develop, build (without tests)
  on both the X64 and gpu pools so /tmp/ccache stays primed with the
  latest develop. This prevents the first PR runs after a big refactor
  merge from all paying a full rebuild.
@mohanchen mohanchen added GPU & DCU & HPC GPU and DCU and HPC related any issues Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS Refactor Refactor ABACUS codes labels Aug 2, 2026
@Stardust0831

Stardust0831 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

The shallow checkout and ccache statistics are useful, but I suggest splitting out the cache warmer. It is not yet clear whether /tmp/ccache persists across jobs and nodes in the dynamic runner pool, so one warmer may only warm a single node while consuming an additional GPU. It also builds the default seven CUDA architectures, while #7753 uses sm_70, so those CUDA cache entries will not be effectively reused.

@Critsium-xy

Copy link
Copy Markdown
Collaborator

A few concerns on the cache_warmer.yml half (the test.yml changes look fine to me — I checked that fetch-depth: 1 is safe, since the only git-history consumer is git log -1 in CMakeLists.txt:163-174):

1. The configure flags are copy-pasted and already out of sync. The header comment says to keep them aligned with test.yml / cuda.yml, but as of this commit they already differ: -Werror=dev is missing, and the parallelism is -j8 / -j4 / -j $(nproc) across the three files. A comment can't hold three copies of a build recipe together. Suggest extracting "install toolchain + configure + build" into a workflow_call reusable workflow (or a composite action) that all three call with parameters, so there is one source of truth.

2. /tmp/ccache persistence across the runner pool is unverified. /tmp/ccache is a host path, so if the X64 / gpu pools are multi-node or ephemeral, the warmer only primes whichever node it happens to land on, and every other node still pays the cold build. This is the premise the whole warmer rests on — worth confirming first (e.g. print hostname alongside ccache -s over a few runs and check whether the cache size accumulates on the same host).

3. Neither warmer job sets timeout-minutes. The default is 6 hours. A hung build would hold a self-hosted runner — including a scarce GPU one — for that long and starve PR CI. Please add something like timeout-minutes: 60.

Given 2 and 3, I'd suggest splitting as @Stardust0831 proposed: land the test.yml part now, and bring the warmer back as its own PR once the cache-persistence question is answered.

MrLi000001 added a commit to MrLi000001/abacus-develop that referenced this pull request Aug 2, 2026
The previous commit (167904f) tried to make the workflow adapt to
whatever GPU the runner has by auto-detecting compute capability at
CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly
flagged two independent defects and a deeper design issue.

Defects in the previous commit:
  1. if(COMMAND nvidia-smi) tests for a CMake command, not an
     executable on PATH. It was always false, so execute_process never
     ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi).
  2. The detection block and the historical default list were both
     inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer
     guard. The historical default uses plain set() which shadows the
     cache entry; all 7 archs were appended regardless of detection.
  3. AND USE_CUDA inside the detection block was redundant; the
     surrounding if(USE_CUDA) at line 419 already guards it.

Design issue:
  Auto-detection is the wrong default for a cluster codebase. ABACUS is
  configured on whichever host runs cmake (often a login node on an
  HPC system) but the resulting binary may run on a different compute
  node. Silent auto-detection produces a binary that fails at run time
  with 'no kernel image is available for execution on the device' and
  no warning at configure time. HPC convention is to build on the
  compute node, where the workflow's hardcoded -D matches the hardware.

Furthermore, heterogeneous auto-detection fragments the ccache key per
node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin
across CI is exactly what cache_warmer relies on.

This commit:
  - Reverts the CMakeLists.txt auto-detect block.
  - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment
    noting that the value assumes a homogeneous V100 pool and should be
    updated if the pool changes.
  - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism).

Note on the measured 33 min -> 18 min result: the reviewer is correct
that the savings probably come almost entirely from -j4 -> -j $(nproc),
not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs
dominating build time, doubling the build parallelism is enough to
halve the wall time; the arch reduction's contribution, if any, is
small. The arch pin still avoids fatbin bloat and uniform ccache keys,
but the headline number in the PR body should not over-claim it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS GPU & DCU & HPC GPU and DCU and HPC related any issues Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants