With nix options changes, first substituter is always cache.nixos.org

[rapenne-s]

Describe the bug

It seems with the move from nix.binaryCaches to nix.settings.substituters, the first substitutes server is always https://cache.nixos.org which defeats the use of a local caching server.

/etc/nix/nix.conf with nix.settings.substituters = [ "http://10.42.42.150/" "https://cache.nixos.org" ];

substituters = https://cache.nixos.org/ http://10.42.42.150/ https://cache.nixos.org

If I use mkForce to force a value with nix.settings.substituters = pkgs.lib.mkForce [ "http://10.42.42.150/" "https://cache.nixos.org" ]; I get the following /etc/nix/nix.conf

substituters = http://10.42.42.150/ https://cache.nixos.org

Steps To Reproduce

Define a custom substituter nix.settings.substituters = [ "http://10.42.42.150/" ]; , check grep ^substituters /etc/nix/nix.conf, you will have https://cache.nixos.org first in the list.

Expected behavior

It seems https://cache.nixos.org is always merged first in the list, it should be merged after the user defined value.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.12-xanmod1-tt, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.6.0`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[roberth]

The order of cache queries is actually determined by a metadata file served by the caches.
That's what the priority number is for.

$ curl https://cache.nixos.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40

I suppose this will only affect you if the caches have an equal priority number.

It seems https://cache.nixos.org is always merged first in the list, it should be merged after the user defined value.

This can be achieved by putting a lib.mkOrder on the cache.nixos.org definition, or mkBefore on your own definitions.
Or use the priority number.

cc @domenkozar fyi

[rapenne-s]

After a git bisect, I found that the behavior changed with commit 4a9d992

Prior to this and the renaming, nix.binaryCaches = [ "http://10.42.42.150/" ]; used to work as expected but after, it adds https://cache.nixos.org/ first in the list.

[roberth]

cc @polykernel

[polykernel]

This seems to be caused by non-determinism in the merging order of definitions with the same priority. In the nix.settings submodule, the relative order of the substituters definitions are merged is reversed compared to before the PR(I don't know if this is specific to submodules).

I also noticed that mkRenamedOption does not preserve order priority, which means it can not fully map legacy options to the structural settings. I think the easiest solution for now is to just make the default option definition have lower order priority rather than make mkRenamedOption aware of order priority(this still needs to be done at some point), as it doesn't require changes to the module system.

[rapenne-s]

I confirm this has been fixed since I made the issue.

[stuser81]

Everyone, if you have time you're welcome to chime in at #8101

I quote "It seems people want this automatic appending [of "cache.nixos.org"], which feels strange to me. What about people who also want to offload the cache downloading [and not just building] to the build machine?"

[huandzh]

In my particular situation, the default cache "https://cache.nixos.org/" was completely unreachable, yet it continued to be appended to the substituters, causing the nix-* commands to attempt to download and check its priority in vain.

The issue was resolved by using mkForce to prevent automatic appending.

# configuration.nix
{ config, pkgs, lib, ... }:
{
 # ...
 nix.settings.substituters = lib.mkForce [ "https://mirrors.tuna.tsinghua.edu.cn/nix-channels/store" ];
 # ...
}

I hope the code above can be helpful for someone who has encountered the same problem and reached here through searching.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-channel-update-on-isolated-system/48588/1