Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetching ca derivation outputs is very slow #4746

Closed
thufschmitt opened this issue Apr 27, 2021 · 0 comments · Fixed by #4781
Closed

Fetching ca derivation outputs is very slow #4746

thufschmitt opened this issue Apr 27, 2021 · 0 comments · Fixed by #4781
Assignees
@thufschmitt
Labels
bug ca-derivations Derivations with content addressed outputs regression Something doesn't work anymore

Comments

@thufschmitt
Copy link
Member

Is your feature request related to a problem? Please describe.

With the ca-derivation experimental feature, a rather big part of the build time seems to be spent fetching the .doi files from the remote caches.

Describe the solution you'd like

Make it faster!

I think there's two possible ways to improve this

  1. Make sure that the calls are asynchronous and properly parallelized (because each one has a rather high latency)
  2. Add these information to the remote-cache caches to make them faster
@thufschmitt thufschmitt added bug regression Something doesn't work anymore ca-derivations Derivations with content addressed outputs labels Apr 27, 2021
@thufschmitt thufschmitt self-assigned this Apr 27, 2021
thufschmitt added a commit that referenced this issue May 6, 2021
@thufschmitt
Add a realisations disk cache
b662341 
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
@thufschmitt thufschmitt mentioned this issue May 6, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thufschmitt thufschmitt

Labels
bug ca-derivations Derivations with content addressed outputs regression Something doesn't work anymore
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add a realisations disk cache NixOS/nix
1 participant
@thufschmitt