Skip to content

ENH: Optimize GitHub Actions ccache strategy for faster C++ builds#5954

Merged
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:improve-ccache-for-with-larger-quota
Mar 16, 2026
Merged

ENH: Optimize GitHub Actions ccache strategy for faster C++ builds#5954
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:improve-ccache-for-with-larger-quota

Conversation

@hjmjohnson
Copy link
Copy Markdown
Member

@hjmjohnson hjmjohnson commented Mar 16, 2026

Summary

Use best practices now that we have a quota of 50GB for cache storage (Big enough so that we do not constantly evict PR's based on running over the 10GB limit).

Key knowlege: Caches are not mutable. Once written, they can not be updated. That is why the git sha is recommended with fallback patterns to gain the latest commit with the same prefix (to keep the cache warm).

  • Enable direct mode: Remove CCACHE_NODIRECT=1 so ccache skips the preprocessor for cache lookups (significantly faster on hits)
  • Add CCACHE_SLOPPINESS=pch_defines,time_macros: Avoid cache misses from __DATE__/__TIME__ macros that change every build (standard CI-safe setting)
  • Increase CCACHE_MAXSIZE from 2.4G to 5G: ITK has ~4000 translation units; a larger cache retains more objects across incremental builds
  • SHA-based cache key with restore-keys fallback: Each commit produces a unique cache entry (ccache-v4-<os>-<config>-<sha>), while restore-keys prefix match always restores the most recent cache for the same OS and configuration. Replaces the old static key that was immutable once written
  • Save on !cancelled() instead of main-only: PR builds now persist their cache (success or failure), but cancelled runs skip saving potentially incomplete caches
  • Use runner.os instead of matrix.os in cache keys for consistency across runner image updates

GitHub Actions Cache Key Notes for ccache

What does if: ${{ !cancelled() }} do?

Controls when a step runs based on the workflow's status:

Condition Success Failure Cancelled
(default) runs skips skips
if: always() runs runs runs
if: ${{ !cancelled() }} runs runs skips

For the ccache save step, !cancelled() saves the cache on both success and failure
(compilation progress is still valuable), but skips saving on cancellation to avoid
persisting an inconsistent mid-write cache.

What is ${{ github.sha }} and does it work with merged branches?

${{ github.sha }} is the full 40-character Git commit SHA that triggered the workflow.
Appending it to the cache key (e.g., ccache-v4-Linux-name-<sha>) makes each commit
produce a unique, immutable cache entry. The restore-keys prefix fallback ensures
each build restores the most recent cache for the same OS and configuration.

Branch scope restriction

GitHub Actions caches are scoped by branch:

Build runs on Can restore caches from
main main only
PR branch PR branch + main (base branch)

After a PR is merged, the main branch build cannot access caches created on the
PR branch. It falls back to the most recent cache saved by a prior main build. This
means main does not benefit from incremental cache improvements made during PR CI,
but in practice, this is acceptable because the main builds frequently enough to keep its
own cache warm.

🤖 Generated with Claude Code

Key changes from upstream:

1. Remove CCACHE_NODIRECT=1: Enables direct mode so ccache skips the
   preprocessor for cache lookups, significantly faster on hits.

2. Add CCACHE_SLOPPINESS=pch_defines,time_macros: Avoids cache misses
   caused by __DATE__/__TIME__ macros that change every build. Standard
   CI-safe setting.

3. CCACHE_MAXSIZE 2.4G -> 5G: ITK has ~4000 translation units; a
   larger cache retains more objects across incremental builds.

4. SHA-based cache key with restore-keys fallback:
   - key: ccache-v4-<os>-<config>-<sha>  (unique per commit)
   - restore-keys: ccache-v4-<os>-<config>-  (most recent match)
   Each build creates a new cache entry; restore-keys always finds
   the most recent cache for the same OS and configuration.
   Replaces the old static key that could never be updated once written.

5. Save on !cancelled() instead of main-only: PR builds now persist
   their cache for subsequent pushes, but cancelled runs do not save
   potentially incomplete caches.

6. Use runner.os (Linux/macOS/Windows) instead of matrix.os
   (ubuntu-24.04-arm/macos-15) for cache key consistency across
   runner image updates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Enhancement Improvement of existing methods or implementation labels Mar 16, 2026
@dzenanz dzenanz requested a review from blowekamp March 16, 2026 19:54
@hjmjohnson hjmjohnson self-assigned this Mar 16, 2026
@hjmjohnson
Copy link
Copy Markdown
Member Author

/azp run ITK.Windows

@hjmjohnson hjmjohnson merged commit b2b6097 into InsightSoftwareConsortium:main Mar 16, 2026
18 checks passed
@hjmjohnson
Copy link
Copy Markdown
Member Author

Post-merge CI timing analysis — 45-day measurement

@dzenanz @thewtex — sharing a quantitative analysis of the build time impact from this PR, measured over 700+ successful CI runs across the full 45-day window (2026-02-17 → 2026-04-03).

Three-period breakdown

Period ARM64 mean ARM64 median Pixi mean Pixi median
A — No ccache (before 2026-03-09) 369 min 244 min 255 min 123 min
B — Initial ccache, broken (2026-03-09 → 2026-03-16) 417 min ↑ 365 min ↑ 238 min 159 min
C — This PR (2026-03-16 → present) 221 min 183 min 127 min 117 min

Period B was counterproductiveCCACHE_NODIRECT=1 forced the preprocessor on every lookup and the static ccache-v1-* key could never be updated, so cache writes were wasted. This PR fixed both root causes.

Hypothesis: CONFIRMED

  • ARM64: −47% mean, −50% median wall-clock time (B→C)
  • Pixi: −47% mean, −26% median wall-clock time (B→C)
  • Variance collapsed 65–67% — the standard deviation dropped from ~255–412 min to 53–90 min; the catastrophic outliers (previously up to 52 h on ARM64, 71 h on Pixi) are completely gone in Period C
  • Representative per-job comparison (Feb 27 cold vs Apr 1 warm):
    • ARMBUILD-Ubuntu-24.04-arm: 60 min → 6 min (−90%)
    • ARMBUILD-x86_64-rosetta: 48 min → 20 min (−58%)
    • ARMBUILD-Python (Python wrapping, not ccache-able): no change

Open question for @dzenanz and @thewtex

The ARMBUILD-x86_64-rosetta per-job timings show a +20% mean increase in the small-sample (n=8-9) per-job analysis, though the single-pair comparison above shows -58%. There is enough variance in the full population that this is likely noise, but it would be worth monitoring whether Rosetta builds are consistently hitting the cache or experiencing key mismatches under the new SHA-based key strategy.

Full analysis with raw data tables: Documentation/CodingEfforts/CI_CacheAnalysis.md (local).

@thewtex
Copy link
Copy Markdown
Member

thewtex commented Apr 6, 2026

@hjmjohnson awesome!! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:Enhancement Improvement of existing methods or implementation type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants