Enable always writing cache to support hermetic build systems

[wchargin]

I’d like to use actions/cache to cache my Bazel build state, which
includes dependencies that have been fetched, binaries and generated
code that have been built, and results for tests that have run. Bazel is
a hermetic build system, so the standard Bazel pattern is to always use
a single cache. Bazel will take care of invalidation at a fine-grained
level: if you only change one source file, it will only re-build and
re-test targets that depend on that source file.

Thus, the pattern that makes sense to me for Bazel projects is to always
fetch the cache and always store the cache. We can always fetch the
cache by using a constant cache key, but then the cache will never be
stored. Bazel doesn’t have a single package-lock.json-style file that
can be used as a cache key; it’s the combination of all build and source
files in the whole repository. We could key use the Git tree (or commit)
hash as a cache key, but this would lead to storing a mountain of
caches, too, which seems wasteful.

Ideally, the fetched cache would be taken from origin/master, but
really taking it from any recent commit should be fine, even if that
commit was in a broken or failing state.

On my repository, it takes 33 seconds to save the Bazel cache after a
successful job, but on a clean cache it takes 2 minutes to fetch remote
dependencies and 26 minutes to build all targets. I would be more than
happy to pay those 33 seconds every time if it would save half an hour
in the rest of the build!

For comparison, on Travis we achieve this by simply pointing to the
Bazel cache directory:
https://github.com/tensorflow/tensorboard/blob/1d1bd9a237fe23a3f2c31282ab44e7dfbcac717c/.travis.yml#L30-L32

[chrispat]

@wchargin this is an interesting topic thanks for bringing it up.

In this example if we had a way to skip storing the cache unless the run was on master you could use the git commit as part of your key and get the desired behavior without writing a new cache for each run of a pull request.

Do you think that would work for you?

[wchargin]

@chrispat: Yeah, that sounds reasonable! At a glance, I don’t see a way
to save the cache only if it’s running on master… but perhaps I could
hack something together that restores the cache to its initial state at
the end of the job for builds that aren’t running on master—just as
a proof of concept to see how this strategy works.

If I understand correctly, we’d still be proliferating caches with each
commit to master, right? I understand that cache eviction kicks in, but
it still seems unfortunate, especially if I have to worry about other
caches (e.g., node_modules) being evicted prematurely.

[hvr]

For the record, cabal's Nix-style store/cache also falls into this category; see my comment at #38 (comment)

[chrispat]

@wchargin given the version of the sources is part of the bazel caching algorithm what key do you think should be used to prevent a huge number of updates? My assumption is travis is uploading new caches essentially every build if they are just looking at changes to the cache directory.

[wchargin]

Yes, Travis uploads new caches every build. And you’re right that this
is a performance problem: Travis re-uploads the entire cache directory
from scratch every build, which can take minutes. (Also, the build
doesn’t report success until this upload has completed, and this
upload can cause an otherwise successful build to time out and fail,
which is super frustrating…)

We do want to update the cache on every build, but it should be cheap to
perform a partial update of only files that changed, rsync-style. The
action cache will be updated on basically every commit, but is small
(~500K). The fetch caches will be very rarely updated, and can be large
(hundreds of MB). And the build cache for any given target will be
updated whenever that target changes, but not only if unrelated targets
change, and can be of varying sizes (typically fairly small, but there
are lots of them).

I see that actions/cache currently tars and gzips everything into a
single bundle, but it would be much more effective for caches in the
style of Bazel/Nix/Cabal to support incremental updates, perhaps by
using use a content-addressable store like that of Git itself. What do
you think?

[chrispat]

For something like bazel I wonder if having a truly remote cache is actually a better option https://github.com/buchgr/bazel-remote. This is not something we are going to get around to implementing anytime soon but it is something we can consider for the future.

The model we have for caching enables to user to control the key and also requires that all caches are immutable by key. While that is not ideal for all scenarios it does work generally well for a large number of different technology stacks and scenarios. This immutable nature make incremental update untenable and likely not possible. Even if we could incrementally update the cache the download on next run is going to have to be the entire cache as we have to provision a fresh VM for each job.

[mborgerson]

I believe I have a similar use case to the issue described here, and ideally would like to see an update-cache option added to the action, but I've worked around the issue by leveraging the restore-keys option.

A project of mine consists largely of C files, and naturally a significant portion of my CI cycle time is spent in compilation. To speed things up, I've employed ccache, which will opportunistically recycle previously built object files when it detects that the compilation would be the same for the current build. This has a dramatic performance improvement on CI times. In order to do this though, I need some persistence of storage between workflow runs in order to save and restore ccache's cache directory. Of course, as the code base evolves, the cache of object files will change too.

I was pleased to discover actions/cache, as it fits my use case very nicely; but, I was surprised to find that when a cache hit occurs, actions/cache will not attempt to update the cache at all, and there's not an option to request such update.

To work around this, I do the following:

    - name: Initialize Compiler Cache
      id: cache
      uses: actions/cache@v1
      with:
        path: /tmp/xqemu-ccache
        key: cache-${{ runner.os }}-${{ matrix.configuration }}-${{ github.sha }}
        restore-keys: cache-${{ runner.os }}-${{ matrix.configuration }}-

It works like this: when the cache is loaded for a workflow, there will be an initial cache miss because the cache key contains the current commit sha. actions/cache will fall back to the most recently added cache via restore-keys prefix matching policy, then after the build has completed, create a new cache entry to satisfy the initial cache miss.

This solution seems to work very well for me, and hopefully this will be useful to others with a similar use case. Ideally though, I think actions/cache should just support updating the cache, to a new immutable revision perhaps--as I have done above.

[wchargin]

Having the caches be immutable makes a lot of sense. Immutable caches
seem perfectly compatible with incremental updates—in fact, this is a
strong point of Git. If your repository has 100 top-level directories
each with 100 files, then you have 101 trees and 10000 blobs; if you
change just one of those files, then you have 103 trees and 10001 blobs,
not 202 trees and 20000 blobs. Does this make sense, or am I missing
something?

A truly remote cache is an appealing option, but comes with a lot more
operational overhead for the user. Storing files is much easier than
running a server.

Downloading the full latest cache on each run may not be perfect, but
it’s still an improvement over rebuilding all the artifacts, faster by
about 20 minutes in my case.

[mborgerson]

@wchargin I agree that immutability is acceptable on the condition that we can restore and create a new cache as I have described above (though it can be quite wasteful as you mentioned). My guess is that this particular use case will be desirable by many projects. Perhaps the documentation could simply be updated to demonstrate this type of use case? To me, it wasn't immediately obvious. My suggestion would be to mention using ${{ github.sha }} in the key.

[wchargin]

Right; immutability is space-wasteful if the caches are stored
independently (which will happen if you use ${{ github.sha }}) but not
if they’re stored as part of one content-addressed store (which would
require changes to the actions/cache implementation).

[chrispat]

A truly remote cache is an appealing option, but comes with a lot more
operational overhead for the user. Storing files is much easier than
running a server.

I was thinking we would run that server on behalf of the user so the operational overhead should be essentially the same is it would be for the existing cache action. I am not 100% sure that is the best option but it seems like it might be a really good one for build systems that support it.

[wchargin]

A truly remote cache is an appealing option, but comes with a lot more
operational overhead for the user. Storing files is much easier than
running a server.

I was thinking we would run that server on behalf of the user so the
operational overhead should be essentially the same is it would be for
the existing cache action.

Oh, that would be fantastic! —being able to just point Bazel to a remote
cache provided by a GitHub-managed action would be a huge value-add
for us compared to other CI services.

[dsilva]

Similar use case: ~/.cache/sccache for sccache works like the bazel cache. For now it's probably easier to point sccache at S3 or GCS to avoid the issues described above, and it would be nice if GitHub ran an sccache store as well.

[jacquesbh]

Hi!

I have the same issue I think with composer cache, in PHP.

Composer saves every download it makes in cache and most of the time this cache is globally used.
As example on a developer machine the cache is growing all the time with new releases.

I've been using the github.sha in my cache keys since it allows me to re-save the cache and avoid the case where it hits the cache but new versions of dependencies exist and it's always downloading them since they're not in cache.

[jvolkman]

Specifically for Bazel: the cache protocol is pretty simple. I wonder if it would be feasible to write a service that simply proxies to Github's own artifactcache service and stand it up locally? Not sure how many cache keys Github allows.

[pramodka-revefi]

Since this task is still open, What's the current best practice for bazel caching + github actions? Does someone have a snippet of their github workflow they can share?

Update: Just sharing the CI pipeline yaml with caching that we went with, hopefully it'll help the next person who lands on this task. (Slightly more permissive approach to @nanddalal above)

[github-actions]

This issue is stale because it has been open for 200 days with no activity. Leave a comment to avoid closing this issue in 5 days.

[mihaimaruseac]

I still think this is useful to have and not close

[yongtang]

I think this will also help reduce the overall cost of compute resources on GitHub actions, as many open source projects can minimize the GitHub actions minutes they use for every run.

[jsoref]

So... If you're willing to use an a/b system, you could probably do something like:

- uses: actions/cache/restore@v3
  with:
    key: preferred
    restore-key: fallback
- run: do-work
- if: no-cache
  uses: actions/cache/save@v3
  with:
    key: fallback
- if: no-cache
  uses: actions/cache/save@v3
  with:
    key: preferred
- if: used-preferred-cache
  uses: ./delete-cache
  with:
    key: fallback
- if: used-preferred-cache
  uses: actions/cache/save@v3
  with:
    key: fallback
- if: used-fallback-cache
  uses: actions/cache/save@v3
  with:
    key: preferred
- if: used-preferred-cache
  uses: ./delete-cache
  with:
    key: preferred
- if: used-preferred-cache
  uses: actions/cache/save@v3
  with:
    key: preferred

Notes:

• used-fallback-cache, used-preferred-cache, and no-cache aren't technical things, but actions/cache has outputs that you can use to construct the concepts
• You might be able to condense the various stages, but you definitely want to ensure that at least one of the caches is available, and given how simple the with:'s will be, it might be simpler just to have lots of steps than to try to be incredibly fancy about it.
• in addition to wrapping delete-cache into an action, you could wrap the entire delete+save pattern into an action

./delete-cache can be implemented using the APIs that were made available in circa June 27, 2022:
https://github.blog/changelog/2022-06-27-list-and-delete-caches-in-your-actions-workflows/

[github-actions]

This issue is stale because it has been open for 200 days with no activity. Leave a comment to avoid closing this issue in 5 days.

[Frenzie]

Bots suck.

[IanButterworth]

I think this can be closed as it's now released in v4

[jsoref]

I don't see how v4 changes anything. Either it was already possible (and I think my suggestions and others show that there are ways to do something) or it might still not be possible.

If it's now possible as of v4, it'd be nice if someone put together an actual example of how to do it.

[IanButterworth]

My bad. v4 has a save-always option. But this would be more like a save-overwrite option?

[jsoref]

I mean, I'd probably just use an epoch time value with a fallback of none:

        key: cache-${{ steps.time.outputs.epoch }}
        restore-keys: cache-

That'd result in it always writing one. Older caches will get wiped out as they become least recently used. Sure, you pay a bit to store a duplicate of the cache (or you could use actions/cache/restore and actions/cache/save and only conditionally call actions/cache/save if you made any changes...), but, so what?

[ephemient]

but, so what?

That excess space usage causes other caches to get dropped too.

[jsoref]

Then use restore & saveseparately and use an if: to only use save when you have changes.

If you're being really aggressive, you might be able to portion the cache into lots of pieces and have steps to calculate and retrieve/save them.

There will be a trade-off between how many steps you need to run and how big your cache pieces are.

[github-actions]

This issue is stale because it has been open for 200 days with no activity. Leave a comment to avoid closing this issue in 5 days.

[Frenzie]

https://github.com/c-hive/fresh-bot

[mihaimaruseac]

This is still an issue worth resolving

[TheGiraffe3]

Why wasn't the stale label removed?