Unexpected remote builder behavior; builders-use-substitutes = true not working?
#8101
Comments
If you change the nix.conf file you need to restart the daemon.
The 10 there is for maxJobs. I would recommend to use https://search.nixos.org/options?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=nix.buildMachine to avoid confusion.
No, it is not. The builders are read by the daemon which only read /etc/nix/nix.conf.
There is likely something else happening, too. Is there any log indicating that the substitution failed? What are the substituters on the build machine? Is the command working as expected if you run it directly on the build machine? |
@SuperSandro2000 I have not changed the default substituters on the build machine (it's still I dig a bit more digging:
Here is a wild guess (take everything below with a grain of salt because it's just a guess):
Does this guess have any merit? |
I recently misconfigured my substituters setting and through that build everything on remote builders which in fact downloaded the derivations from cache.nixos.org
Not really. How are you installing Nix? Are you using the installer? Can you try it with a NixOS machine? |
@SuperSandro2000 I just tried. I noticed that on NixOS, So I suspect you were hitting So I believe we've been comparing apples and oranges. Is there any way to get NixOS to not do this strange automatic appending? Related: NixOS/nixpkgs#158356 It seems people want this automatic appending, which feels strange to me. What about people who also want to offload the cache downloading to the build machine? |
|
I finally got it to work. These are the main changes I made since last time:
Things started working after that. (I can't guarantee I did nothing else important but I doubt it.) I don't think the root vs. regular user thing was the real issue. The real issue was maybe that I didn't have any TODO room for improvements: Replace Worth noting: When the client machine shows you tons of lines like it will not display if the build server is actually building it or if it's downloading from a substituter (e.g. The inability of NixOS clients to leave |
|
Yep, just started doing remote builds again recently and I'm seeing this again too. edit: I don't mean to be too whiny, but after years, it's disappointing how little confidence I have in many scenarios around remote building. |
|
And now I can't tell (after adding a regular "trusted-user") if it's working or not because it's still copying sources up. It would be great if there was a way to do a remote build with EDIT: this is probably the result of me copying derivations to the remote, and the fact that some of my config uses IFD ? |
|
I'm still not able to get remote builder to pull packages from cache directly, it always goes through my local machine. Is there any exact steps if anybody got it right? EDIT: Turns out I didn't had substituters set up on the remote build machine, adding a nix-channel and updating fixed it. |
|
I see this as well. This makes remote builds longer than building locally :s I've checked my local and remote configs and everything looks sensible to me, I have no idea why I have to push stuff from my machine :s It doesn't help that the nix-daemon is very quit. Is there any flag to make it more verbose (--help is not that helpful either) ? |
I have two machines on my LAN:
My goals:
cache.nixos.orgwhen possible.This is what happens on the laptop without remote building setup (i.e. without any
~/.config/nix/nix.confand without any modifications to/etc/nix/nix.conf). I have truncated the output, but in fact everything is already available oncache.nixos.orgso it gets downloaded from there.Let's now set up remote building:
ssh root@192.168.1.80without password (using the normal SSH key stuff)./etc/nix/nix.conf(only the first line was there by default):builders-use-substitutes = trueasks the desktop to grab fromcache.nixos.orgif possible,max-jobs = 0makes sure no building takes place on the laptop, andbuilders = ssh://192.168.1.80 - - 10makes sure 10 cores are used on the desktop.Note: On my laptop I ran
sudo systemctl stop nix-daemon.serviceandsudo systemctl start nix-daemon.servicebut I'm not sure if that was needed. However, I noticed I indeed had to modify/etc/nix/nix.confon my laptop. Modifying~/.config/nix/nix.confon my laptop wasn't causing the remote builder to get registered at all. (This is possibly a separate bug altogether.)The problem:
Check out what happens now. As you can see, it's building things on the desktop machine! This is unexpected because as we already saw,
cache.nixos.orgshould already have everything needed. (I have not modified the default substituters on the desktop NixOS machine.) It seems thatbuilders-use-substitutes = trueis not working properly.The text was updated successfully, but these errors were encountered: