Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use with actions/cache #33

Open
deemp opened this issue Jul 8, 2023 · 3 comments
Open

Use with actions/cache #33

deemp opened this issue Jul 8, 2023 · 3 comments

Comments

@deemp
Copy link
Contributor

deemp commented Jul 8, 2023

Hi @blaggacao, @stephank, @rickynils! Thanks a lot for your work!

actions/cache

Because your action makes /nix/store owned by an unpriviliged user, actions/cache can restore and save /nix/store without intermediate stores.

As a PoC, I made the deemp/cache-nix-too action to close DeterminateSystems/magic-nix-cache-action#16.

cache-nix-too is actions/cache, but also caches several Nix directories as suggested by @blaggacao in DeterminateSystems/magic-nix-cache-action#11 (comment).

README

I suggest you write in the README that nix-quick-install-action can be used together with actions/cache so that people can benefit from your action even more.

nix-cache-too open problems

Directoris

I'm not sure I cache the right directories in deemp/cache-nix-too.

Cache eviction

I can't find a good policy for cache eviction. Say, I build a docker image in each job run. Next, I cache the whole store. In the next run, I restore the whole store with that old image. If I don't remove that image, it'll be cached and restored indefinitely.

Possible solutions

New action

I believe you can make a new nix-quick-cache-action action that caches the right directories and has a reasonable paths eviction policy.

@blaggacao
Copy link
Contributor

blaggacao commented Jul 8, 2023

I don't know about the implications of restoring the whole /nix/

It may painstaikingly slow down the system on unrelated store entries that need to be compressed, transported and decompressed. Either due to cache growth or due to multiple packages / builds.

It was suggested to exclude some of them in

Looks mostly like runtime nix data.

it'll be cached and restored indefinitely

GH evicts about every 7 days or after ~10GB. But that's no solution to cache growth in a sprint.

Imo, the best design would be a background upload by magic cache and the cache unit would be best individual store paths.

This makes the system as efficient as it can get and using (slow) bandwidth on only the bits and pieces that are needed.

In any case, you work is taking an interesting approach and I'll have a look with a little more time at hand.

We've had some.experiences gathered with v0.23.0 of divnix/std-action on the nix cache restore approach.

@deemp
Copy link
Contributor Author

deemp commented Jul 8, 2023

Imo, the best design would be a background upload by magic cache and the cache unit would be best individual store paths.

Well, I ran into rate limit with magic-nix-cache in the second job run (DeterminateSystems/magic-nix-cache-action#16).

With nix-quick-install-action and cache-nix-too, I restored a ~1.5 GB /nix in under a minute on Linux (see workflow) and in ~4 minutes on macOS. I was impressed and decided to share with you.

@blaggacao
Copy link
Contributor

I ran into rate limit

I wonder if that's something that the NixOS Foundation could touch upon in their sync-up meeting with GH engineers. 🤷

My bet is that this may eventually happen due to the vested interests of DetSys and their representation across the nixos governance structure.

Btw, we've been using something similar in the past.

But indeed the ~1 - 2 min penalty for cache creation & restore was critical cycle time for us.

It's not 100% related, but here we've taken an approach to evaluate in a persistent "Discovery" host which brought down discovery of buildable targets from 20mins to 40s on IFD-heavy projects.
The building is then offloaded onto nixbuild from the GH worker build matrix, or alternatively the magic cache action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants