Don't download the same .nar file twice if we already realized it elsewhere in the store

[flokli]

Is your feature request related to a problem? Please describe.
The .narinfo files for /nix/store/f1jbb8m9q9rylqjv8ckihy4ykzd1a7dl-nerdfonts-2.1.0 and /nix/store/f4w8zyr11q3pcqbrf2rhb9dfihidsgbz-nerdfonts-2.1.0 point to the same .nar file, with the same NarHash.

Nix downloads the same .nar file twice, even though it wouldn't need to

Describe the solution you'd like

We probably already store the narhash of every nix store output. In case we encounter a .narinfo file, we should probably check if we already have another store path with that narhash, and instead of downloading, copying it from there.

[thufschmitt]

shameless plug: CA derivations probably would make this problem essentially non-existent except in very convoluted situations

We probably already store the narhash of every nix store output. In case we encounter a .narinfo file, we should probably check if we already have another store path with that narhash, and instead of downloading, copying it from there.

We do store it in the sqlite db indeed (although there’s no index so we can’t efficiently query it). So this should be possible to do reasonably easily. I probably wouldn’t do it myself, but happy to help if you want to give it a try.

(Note that this is an instance of a more general problem: Given two substituters sub1 and sub2, we should ideally be able to fetch a narinfo from sub1 and the nar from sub2, but this isn’t possible right now)

[roberth]

A similar optimization is to avoid downloading the nar altogether when the narinfo reports a nar hash that matches the contents of any store path in the local store. (edit: on re-reading, I think I just rephrased the same optimization)

However, the lack of this optimization is exploited here, to run fetchers whenever their implementation changes, for testing purposes. We'll need a special derivation attribute to turn off that optimization when it is implemented.

[flokli]

This relies on the assumption that Nix isn’t clever enough to reuse its database of local store contents to optimize fetching.

Yah, IMHO it's fine if Nix gets more clever at some time, and instead of exploiting bugs we add a CLI flag (or derivation attribute) to trigger these redownloads.

[stale]

I marked this as stale due to inactivity. → More info

[flokli]

not stale.