Use with actions/cache

[deemp]

Hi @blaggacao, @stephank, @rickynils! Thanks a lot for your work!

actions/cache

Because your action makes /nix/store owned by an unpriviliged user, actions/cache can restore and save /nix/store without intermediate stores.

As a PoC, I made the deemp/cache-nix-too action to close DeterminateSystems/magic-nix-cache-action#16.

cache-nix-too is actions/cache, but also caches several Nix directories as suggested by @blaggacao in DeterminateSystems/magic-nix-cache-action#11 (comment).

README

I suggest you write in the README that nix-quick-install-action can be used together with actions/cache so that people can benefit from your action even more.

nix-cache-too open problems

Directoris

I'm not sure I cache the right directories in deemp/cache-nix-too.

• It was suggested to exclude some of them in Cache /nix/store between jobs cachix/install-nix-action#56 (comment), but I don't know why
• I don't know about the implications of restoring the whole /nix/

Cache eviction

I can't find a good policy for cache eviction. Say, I build a docker image in each job run. Next, I cache the whole store. In the next run, I restore the whole store with that old image. If I don't remove that image, it'll be cached and restored indefinitely.

Possible solutions

• I tried caching only paths accessed during a run by consulting their atime. I used install-nix-action. However, I observed unstable cache sizes on macOS (issue) runners.
• It may be enough to run an installable, nix profile install installable, do other steps, collect garbage, rm -rf /nix/var/nix/profiles, save cache, run a job again.
• Start direnv in background.
• Use nix-heuristic-gc
• Somehow keep inputs (Offline flake development NixOS/nix#4250, Flakes in the nix store do not retain references to their inputs NixOS/nix#6895)

New action

I believe you can make a new nix-quick-cache-action action that caches the right directories and has a reasonable paths eviction policy.

[blaggacao]

I don't know about the implications of restoring the whole /nix/

It may painstaikingly slow down the system on unrelated store entries that need to be compressed, transported and decompressed. Either due to cache growth or due to multiple packages / builds.

It was suggested to exclude some of them in

Looks mostly like runtime nix data.

it'll be cached and restored indefinitely

GH evicts about every 7 days or after ~10GB. But that's no solution to cache growth in a sprint.

Imo, the best design would be a background upload by magic cache and the cache unit would be best individual store paths.

This makes the system as efficient as it can get and using (slow) bandwidth on only the bits and pieces that are needed.

In any case, you work is taking an interesting approach and I'll have a look with a little more time at hand.

We've had some.experiences gathered with v0.23.0 of divnix/std-action on the nix cache restore approach.

[deemp]

Imo, the best design would be a background upload by magic cache and the cache unit would be best individual store paths.

Well, I ran into rate limit with magic-nix-cache in the second job run (DeterminateSystems/magic-nix-cache-action#16).

With nix-quick-install-action and cache-nix-too, I restored a ~1.5 GB /nix in under a minute on Linux (see workflow) and in ~4 minutes on macOS. I was impressed and decided to share with you.

[blaggacao]

I ran into rate limit

I wonder if that's something that the NixOS Foundation could touch upon in their sync-up meeting with GH engineers. 🤷

My bet is that this may eventually happen due to the vested interests of DetSys and their representation across the nixos governance structure.

Btw, we've been using something similar in the past.

But indeed the ~1 - 2 min penalty for cache creation & restore was critical cycle time for us.

It's not 100% related, but here we've taken an approach to evaluate in a persistent "Discovery" host which brought down discovery of buildable targets from 20mins to 40s on IFD-heavy projects.
The building is then offloaded onto nixbuild from the GH worker build matrix, or alternatively the magic cache action.