-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remote build proxying #1914
Comments
Doesn't this already work? I haven't tried it, but in principle there is nothing stopping a remote builder from forwarding builds itself. |
There's at least one complication with this today: a proxy build server would have to have all of the supported features of anything it forwards to, even though that might not be the case. It must also be of the same system type. An ideal user story for this would be something like a very lightweight proxy server which technically has a nix build environment and is setup to forward builds, but won't ever be used for any builds itself. I don't really see a way for a machine to forward a build it received without it also having some chance of being the thing that builds it. For build clusters with heterogeneous supported / mandatory features and systems, this won't work. |
I marked this as stale due to inactivity. → More info |
@edolstra I just tried to achieve sort-of the same thing, and I'm running into an issue. Building for different architectures on the proxy server works just fine when executing $ nix build -L --impure --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname -mo > $out")'
error: build of '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' on 'ssh-ng://nixbld@proxyhost' failed: error: unable to start any build; remote machines may not have all required system features.
https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html
error: builder for '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' failed with exit code 1 Local machine configMy local
Where
Proxy machine config
And it's
I already tried setting the The {
services.openssh.extraConfig = let
command = pkgs.writeShellApplication {
name = "command";
runtimeInputs = [ pkgs.nix pkgs.coreutils ];
text = builtins.readFile ./nixbld-force-command.sh;
};
in ''
Match User nixbld
AllowAgentForwarding no
AllowTcpForwarding no
PermitTTY no
PermitTunnel no
X11Forwarding no
ForceCommand ${command}/bin/command
Match All
'';
} The #!/usr/bin/env bash
if [[ "${SSH_ORIGINAL_COMMAND:=''}" != 'nix-daemon --stdio' ]]; then
echo "Hi ${USER}! You've successfully authenticated!";
echo "However, we do not provide shell access :( Sorry!";
exit 0;
fi
nix-daemon --stdio; I believe I'm running into the following piece of code. nix/src/libstore/build/worker.cc Lines 307 to 325 in d00fe5f
|
I think the user story for many people sharing a cluster of remote nix builders is currently limited by the fact that remote building only works one level deep. i.e., if you have a cluster of build machines that you want to share with many Nix users, each user has to maintain their own copy of the list of each machine, and each user has to have an account on each of the machines (or otherwise share an account and SSH key).
This could be improved with allowing remote building to forward builds to machines that are themselves configured as remote builders on the currently selected remote builder. This would allow you to create clusters of build machines where in machine A has B, C, and D configured as remote builders, and anyone who wants to use the large cluster for remote building only has to know about and have access to machine A.
The text was updated successfully, but these errors were encountered: