CI ccache Optimizations

CI ccache Optimizations

Executive Summary

To prevent slow compilation times during development, ccache is utilized in the GitHub Actions workflow (build-rockchip.yml). However, the previous tag-based caching strategy led to severe bottlenecks, cache misses, and cache eviction cycles due to GitHub's 10 GB storage limits per repository.

This document describes how the caching strategy was redesigned to achieve optimal cache hits across concurrent builds.

---

Challenges in Cross-Compilation Caching

1. Absolute Path Mismatches (CCACHE_BASEDIR)

By default, ccache stores absolute file paths of source code. In virtualized GitHub Actions environments, the absolute workspace path (/home/runner/work/llama.cpp-rockchip/...) may change or vary. This causes cache misses even when the underlying code is identical.

2. VM Compiler Mtime Changes (CCACHE_COMPILERCHECK)

ccache normally checks the compiler's modification time (mtime) to verify if the compiler has changed. Since the cross-compiler toolchain is re-installed on each virtual machine run, its mtime changes, systematically invalidating the entire cache.

3. Eviction Conflicts on Tag-based Cache Keys

Previously, cache keys were partitioned by Git tag/branch. With concurrent builds executing across different tags, each runner would attempt to write its own massive cache file. This filled the 10 GB repository cache budget, causing continuous eviction of warmed baselines and resulting in slow, cold compiles on subsequent runs.

---

The Solution

We overhauled the cache environment and key architecture in .github/workflows/build-rockchip.yml to implement a unified rolling cache mechanism.

1. Cache Environment Tuning

We added key parameters to the build environment to normalize paths and compiler signatures:

• CCACHE_BASEDIR: ${{ github.workspace }}: Rewrites absolute paths to relative paths based on the workspace root, enabling cross-compile hits across differing VM directory trees.
• CCACHE_COMPILERCHECK: content: Forces ccache to check the compiler's binary hash (content) rather than its file modification time.
• CCACHE_MAXSIZE: 1.5G: Sets a reasonable boundary per cache file to prevent runaway bloat.

2. Unified Rolling Key Architecture

The cache configuration was modified to write under a run-specific key while falling back to a shared rolling prefix:

- name: ccache
  uses: actions/cache@v4
  with:
    path: ~/.cache/ccache
    key: ${{ runner.os }}-ccache-${{ github.run_id }}-${{ github.run_attempt }}
    restore-keys: |
      ${{ runner.os }}-ccache-

How it works:

• Reads: Concurrent runners search for a prefix match of ${{ runner.os }}-ccache-. This pulls the most recently saved global cache, ensuring they build on top of a warmed baseline.
• Writes: Since the output key contains the unique run_id and run_attempt, each concurrent job writes its updated cache atomically without locking or colliding.
• Eviction Balance: Older caches are naturally evicted by GitHub's LRU policy, but since all runs pull from and contribute to the same rolling lineage, the subsequent run always inherits a warm baseline.