distributed builds require a trusted remote user

[arcnmx]

Attempting to initiate a remote build results in the following: error: you are not privileged to build derivations (nix (Nix) 2.2.2)

Adding the remote user to the remote machine's nix.conf trusted-users allows the build to continue as normal. This seems to be intended behaviour, but I had a few problems with it:

1. The nix manual and wiki page do not seem to mention this requirement at all, so this was surprising. This should probably be mentioned in instructions and documented with a warning.
   • My assumption was that since untrusted/allowed local users can initiate builds through the daemon, doing so over ssh with the distributed build functionality would work in the same way without requiring a change to a less secure system configuration. Considering that it's recommended to store the ssh key for a remote build account without a passphrase, and that "trusted-users is essentially equivalent to giving that user root access", it seems reasonable to expect a warning here before setting up such a machine.
   • There is a small troubleshooting footnote on the wiki page that mentions nix.trustedUsers, but it wasn't clear whether it was referring to the local (like when needed for use with --builders) or the remote machine.
2. The above error message was also unclear to me, since the user does indeed have permission to build derivations under normal circumstances. Very few google results exist for the message, so I assumed I was dealing with some sort of misconfiguration or environment issue. Something more explicit like remote builds require a trusted user might be more obvious?
   • nix copy --to ssh://builder /nix/store/whatever.drv && ssh builder nix build /nix/store/whatever.drv && nix copy --no-check-sigs --from ssh://builder /nix/store/whatever works fine, so I was confused why remote builds with --builder and nix.conf builders weren't able to do the same.

I understood there would be drawbacks to setting it up this way (not trusting the remote user will prevent them from supplying binary substitutes for example, and with multiple builders could even result in duplicated work if the build scheduler isn't aware), but did not expect or find mention that it would be disallowed and prevented entirely. Is there a reason why this check exists? Attempting to send an unsigned build input to the remote machine would've already failed out the build before hitting this wopBuildDerivation error, so it's not immediately clear to me why this particular operation is gated to trusted users to begin with? Sorry if I'm misunderstanding what a "build derivation" operation actually is/does.

[orivej]

I have stumbled upon this in a different situation. My remote non-NixOS machine has already had my user in trusted-users, but it uses nsswith for authentication, and nix-daemon was rejecting remote builds until I added my user into /etc/passwd.

[vaibhavsagar]

This requirement means that sharing a build cluster across multiple users makes them all susceptible to a cache poisoning attack (e.g. if one user's machine is compromised then anything can be copied into the build server's Nix store). I asked @grahamc about a more secure method and he mentioned an possible approach where

1. A .drv is created
2. The .drv's closure is copied over to the build machine
3. The .drv is built on the remote machine
4. The resulting closure is copied back to the user

which avoids the possibility of cache poisoning because everything in a .drv is content-addressed. This is discussed in the original comment as well. Can this be supported as an alternative remote build mechanism?

[edolstra]

That's how remote builds used to work, but it's inefficient to copy the drv closure. That's why buildDerivation was added.

[vaibhavsagar]

Is there a reason we can't have both?

[Ericson2314]

Yes please let's have both.

One gotcha, A derivation can directly refer to a store path direclty (rather than derivation that built it) for sake of store paths with no associated derivation. But say the store path is associated with a derivation? We should outright either ban that, or ensure the derivation is fixed output so the remote builder's drv->build trust mapping isn't grown.

[cdepillabout]

Is there a reason why adding your user as a trusted-user is required?

It seems strange that trusted-user is required, even though in most(?) cases you would be able to directly ssh to the remote machine and run nix-build, which works without being a trusted-user.

---

I think @arcnmx asked this question in their initial post, but it doesn't appear to be answered in this thread. (Unless it actually has been answered and I just don't understand the answer...)

[7c6f434c]

It seems strange that `trusted-user` is required, even though in most(?) cases you would be able to directly `ssh` to the remote machine and run `nix-build`, which works without being a `trusted-user`.

The way distributed builds work with Nix, you are supposed not only to be able to trigger a build on a remote machine, but also to export some locally-available build results to the remote machine and make it accept it. If you have two remote builders, and you change stdenv, and the first one has built glibc, you want just to copy this glibc to the second builder instead of building it again. And importing a non-fixed-output store path (bypassing the build) requires some way of establishing trust.

[edolstra]

@cdepillabout See above. Remote builds don't use the same mechanism as local builds. They use a special API called buildDerivation() that allows the client to only send the contents of the top-level .drv file that it wants to build, rather than all the dependencies of the .drv. However, this prevents Nix from checking that the .drv is legit (i.e. that its output paths are a hash of the derivation graph) so it has to trust the client. Hence buildDerivation() is restricted to trusted users.

[cdepillabout]

@edolstra @7c6f434c That makes sense. Thanks for the explanation.

[roconnor]

However, this prevents Nix from checking that the .drv is legit (i.e. that its output paths are a hash of the derivation graph)

@edolstra Can you explain this comment and integrity issues that come from allowing untrusted users from uploading arbitrary .drv files into the /nix/store? I thought that all correctness properties of the .drv file can be checked using just the .drv file itself (e.g. the hash prefix of the .drv files matches the content of the file).

Also, surely the remote machine needs more that just the top-level .drv file. In order to build all the dependencies it will need the the .drv of all the the dependencies down to the the ones that the remote machine has already built, but this is exactly the set of .drv files that nix-copy-closure needs to send.

(I do get that if the local machine is sending the remote machine build outputs, then, of course the local machine needs to be trusted.)

[roconnor]

Okay, after reviewing page 108 of edolstra's thesis I now see that hashDrv for derivation files isn't just a hash of the contents of the derivation file, but it replaces the inputDrvs contents with a hash of the contents of those inputDrvs files. ... I'll need to think about why things are done this way, but it does mean that it is impossible to stick a .drv file into the nix store without having the contents of the input derivation files on hand.

Edit: hashDrv is used to create the output field of a valid .drv file, not the store hash of the .drv file (which is based on the contents of the .drv file). This is what edolstra means by not being able to validate the output fields of drv files without the contents of the input drv files. Though I still don't quite understand why this needs to be done.

[Ericson2314]

I've started to fix this. I'll link this in the PRs which relate to it.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/35

[stale]

I marked this as stale due to inactivity. → More info

[ldesgoui]

I am still interested in this being resolved

[stale]

I marked this as stale due to inactivity. → More info

[vaibhavsagar]

Not stale

[Ericson2314]

#3921 this is the PR, for the record.

[stale]

I marked this as stale due to inactivity. → More info

[typetetris]

Not stale, still an issue

[fricklerhandwerk]

Discussed in the Nix team meeting:

• @edolstra: constraints: as little configuration as possible, also efficiency
  • @thufschmitt: we may fail somewhat gracefully when input-addressed paths cannot be pushed
  • @roberth: the failure should be opt-in if your situation is susceptible to that case
• agreement: general idea approved, but implementation approach needs more deliberation
  • assigned to @Ericson2314

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-23-nix-team-meeting-minutes-43/26758/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-27-nix-team-meeting-minutes-44/26759/1

[lorenzleutgeb]

@Ericson2314 I understand a fix has landed in Nix 2.16. I'd like to allow trustless remote builds for some users on my machine. Could you please clarify whether

1. the builder machine needs Nix 2.16, or
2. the users that invoke nix build --builders='ssh://machine' ... need Nix 2.16, or
3. both (1.) and (2.), i.e., both ends need Nix 2.16

for trustless remote builds to work? Thank you!

[SuperSandro2000]

• the users that invoke nix build --builders='ssh://machine' ... need Nix 2.16, or

I am pretty sure at least this. If that doesn't work, try updating both sides.

[MathiasSven]

I don't understand how the PR addresses the issue. When a local trusted user runs:
nix build .#foobar -j0 --builders 'ssh://nixremote@remoteip' it will still fail with you are not privileged to build input-addressed derivations on an untrusted remote user nixremote.

Local: nix (Nix) 2.18.1
Remote: nix (Nix) 2.17.0

[the-sun-will-rise-tomorrow]

@MathiasSven I saw the same thing, but then switched from ssh:// to ssh-ng:// and that worked.

[roconnor-blockstream]

For those using NixOS, I had to add

protocol = "ssh-ng";

to my nix.buildMachines entry to remove the error that MathiasSven noted above.

[con-f-use]

I just walked right into that issue and found the error-message thoroughly unhelpful:

error: builder for '/nix/store/1qjb35rc609b23cqwvcv758ndsklbapg-linux-6.1.78-modules-shrunk.drv' failed with exit code 1;
       last 1 log lines:
       > error: you are not privileged to build input-addressed derivations
       For full logs, run 'nix log /nix/store/1qjb35rc609b23cqwvcv758ndsklbapg-linux-6.1.78-modules-shrunk.drv'.
error: builder for '/nix/store/rmgfgnzlf05cyb35n6flnhx3six81khj-update_vault_certs.drv' failed with exit code 1;
       last 2 log lines:
       > copying path '/nix/store/7892njsw5x5wrn2f45bgqzavlgj40gzn-ShellCheck-0.9.0' from 'https://cache.nixos.org'...
       > error: you are not privileged to build input-addressed derivations
       For full logs, run 'nix log /nix/store/rmgfgnzlf05cyb35n6flnhx3six81khj-update_vault_certs.drv'.

While putting builders = ssh-ng:// in /etc/nix/nix.confdid help, it should somehow also be made clear that adding whatever build-user to (extra-)trusted-users, which still pops up as first solution is not a good option. It is not a good option, because for many reasons it is effectively equivalent to making the build-user root on the build-system, which for security-reasons is not always what the user wants.

Another issue is, that if you use singing, the ssh-ng way results in errors of this kind: error: cannot add path '/nix/store/q1bp9795z3cmzhgf9g0m56b46qlsi6w0-link-units' because it lacks a signature by a trusted key, while the trusted-user option does not. I might have misconfigured something, but shouldn't the build-system take care of signing in both cases?

[fricklerhandwerk]

adding whatever build-user to (extra-)trusted-users, which still pops up as first solution is not a good option

By "pops up" you mean appears in web search results? We can only do so much about that. @Ericson2314 could we cheaply improve the error message here? It seems like inferring from the circumstances what the correct steps would be is not exactly trivial.

if you use singing, the ssh-ng way results in errors

I'm not exactly sure, but cannot add path seems to indicate that you're moving unsigned paths from the client to the build machine or vice versa. In both cases you need to set up signing on the machine that creates the paths, and add the public key on the machine that receives the paths.

[con-f-use]

By "pops up" you mean appears in web search results?

Mostly yes, but I also heard this answer from people in the community. A nice error message when that occurs on a remote build would probably, eventually cure both.

In both cases you need to set up signing on the machine that creates the paths, and add the public key on the machine that receives the paths.

Yeah, I probably have something configured badly, but I find it strange that everything build when the build-user is trusted. Should it not require-sigs in that case, too?

---

As a side-note, if you happen to read this and can invite me to the nixos-org so I can keep on maintaining packages effectively, I'd appreciate it. I missed my automatic invite and didn't get a response here.