Bazel Cache for GitHub Actions

[AlexanderLanin]

What a Bazel Cache Solution for GitHub Actions Must Achieve

Before discussing implementation details, we should agree on the problems the solution needs to solve.

A successful solution should:

1. Cover all relevant Bazel caches

Caching only part of Bazel's data is insufficient. The solution must support both the disk cache and the repository cache.

2. Work with parallel CI execution

Our CI executes multiple jobs in parallel. The solution must ensure that cache data produced by different jobs can be reused effectively and that repository cache contents are not lost simply because they were overwritten by another job.

3. Stay within GitHub cache limits

GitHub cache storage is a limited resource. The solution must operate within the current 50 GB limit and should ideally target roughly 10 GB per repository.

4. Avoid wasting CI time

Caching should reduce CI execution time. Uploading and downloading cache data should not cost more time than the cache saves. Unnecessary cache re-uploads should be avoided.

5. Prioritize important branches

Long-lived branches such as main and release/* branches should benefit from stable caches. Short-lived branches and pull requests must not reduce cache availability for important branches. It is acceptable not to cache pull requests at all.

Note: only relevant if we ever develop multiple versions at the same time.

6. Require no manual maintenance

Once configured, the solution should run fully automatically. Repository maintainers should not need to manually clean caches, rotate cache keys, or perform other routine maintenance tasks.

7. Be simple to adopt

The solution should be easy to understand and easy to use. Repository maintainers should not need deep Bazel or GitHub Actions cache expertise to benefit from it.

[AlexanderLanin]

Option comparison

Criterion	setup-bazel	setup-bazel optimized	daily cleanup	fetch all
Supports disk cache and repository cache	✅	✅	✅	✅
preserve the repository-cache required by all relevant CI workloads	❌	❌	❌	✅
disk-cache is up to date	❌	✅	✅	⚠️
Stays within 50 GB GitHub cache limit	✅	❌	✅	⚠️
Optimizes for ~10 GB usage	❌	✅	✅	⚠️
Avoids unnecessary cache uploads	❌	❌	❌	✅
Easy for repository maintainers to use	✅	✅	🟡	✅

Legend: ✅ supported · ⚠️ partially supported / unclear · 🟡 neutral · ❌ not supported

---

fetch all:

bazel fetch //... --config=linux
bazel fetch //... --config=qnx

[lurtz]

Meeting with @lurtz @AlexanderLanin @nradakovic

Problem:

• builds are slow
• caches big and / or outdated

Discussion:

• how does bazel in CI use caches
  • needs at starting phase link code with downloaded caches, which can take minutes
  • build-without-bytes might work, but might only be available for remote caches (not available to us)
• Repository cache tends in inc_someip_gateway to get quite big (3GB), disc caches are relatively small (200MB)
• Disk caches in communication vary from 800 MB to > 5GB
  • They might contain stale data because setup-bazel-cache should only extend caches without any garbage collection at the moment
• idea is to rebuild the repository cache with a dedicated job
  • in best case it is only triggered when approx. this changes

${{ hashFiles(
  'MODULE.bazel',
  'MODULE.bazel.lock',
  '.bazelrc',
  '.bazelversion',
  'extensions/**/*.bzl',
  'tools/bazel/**/*.bzl'
) }}

• configure setup-bazel-cache not to reupload a repository cache
• nightly will only be used to delete the disk cache and rebuild it

[AlexanderLanin]

flowchart TD

    %% Dark-mode friendly styling
    classDef trigger fill:#102a43,stroke:#63b3ed,color:#e6f6ff,stroke-width:2px
    classDef action fill:#3a2a00,stroke:#f6ad55,color:#fff7e6,stroke-width:2px
    classDef cache fill:#123524,stroke:#68d391,color:#e6fffa,stroke-width:2px
    classDef ci fill:#271a45,stroke:#b794f4,color:#f5f0ff,stroke-width:2px
    classDef event fill:#1f2933,stroke:#cbd5e1,color:#f8fafc,stroke-dasharray: 5 5

    subgraph Repository Cache
        A["&lt;trigger><br>Push to main<br/>on file *.bazel*, .*bazelrc*, .bazelversion, **/*.bzl etc"]

        A --> B["Rebuild repository cache<br/><br/>bazel fetch //... --config=linux<br/>bazel fetch //... --config=qnx<br/>..."]
        B --> C["Upload and replace<br/>repository cache"]
        C --> D[("GitHub Actions Cache<br/>Repository Cache")]
    end

    subgraph Disk Cache
        M["&lt;trigger><br>Scheduled maintenance<br/>nightly"]

        M --> N["Delete disk cache<br/><br/>reduce disk cache size"]
        N --> O["Rebuild disk cache"]
        O --> OO["Upload and replace<br/>disk cache"]
        OO --> DC[("GitHub Actions Cache<br/>Disk Cache")]
    end

    D --> P{{"Pull Request CI"}}
    DC --> P

    P --> UD["Build / test<br/>upload growing disk cache"]
    UD --> DC
    UD --> MM(["Merge to main"])

    MM -.-> A
    MM -.-> M

    class A,M trigger
    class B,C,N,O,OO,UD action
    class D,DC cache
    class P ci
    class MM event

Loading

[lurtz]

This requires hash based cache identification for the repository cache and timestamp based identification for the disk cache.

[lurtz]

I start to wonder about https://github.com/bazel-contrib/setup-bazel#repository-cache

The description sounds like it behaves as we want it to if we give the the right files to calculate the hash sum if the default is not sufficient. Now I wonder why the repository cache grew so big in the past.

[lurtz]

One problem I see is using multiple --config= parameters. I guess the first job kind of wins and sets the cache until bazel files change again.

[lurtz]

I branched off @castler setup-bazel code and added a few changes: https://github.com/castler/setup-bazel/compare/cache-optimized...lurtz:setup-bazel:cache-optimized?expand=1

The behavior of setup-bazel is now as follows:

• input files for the repository cache hash are extended to MODULE.bazel.lock, .bazelversion, .bazelrc and .bzl files
• the repository cache will not be used and stored on a timestamp basis, but the hash is used instead
• there is an option to skip restoring the cache for writing clean caches

I now have a working workflow: https://github.com/lurtz/inc_someip_gateway/actions/workflows/cache_maintenance.yml

This combined with the modified setup-bazel and is triggered on pushes to main and it is the only workflow to be triggered on pushes to main. It then checks if input files for the repository cache have been changed. If yes it will rebuild first the repository cache and then delete all existing caches. It will always run the other build jobs to update the disk cache on a timestamp basis.

As such there are still some rough edges:

• if bazel files are not touched for a long time, the disk cache could become quite huge. I hope this is not a big issue, if people bump dependencies from time to time