NIX_SSHOPTS not passed properly to nix-build in nixos-rebuild.sh

[wamserma]

Describe the bug

When running

export NIX_SSHOPTS="-i /root/.ssh/id_nix_store"
nixos-rebuild switch --upgrade --option extra-binary-caches ssh://nix-ssh@server

no public key authentication happens (as shown in log of sshd on server)
stracing showed me that this seems to be the failing call

execve("/tmp/nixos-rebuild.Rvox9Q/nix/bin/nix-build", ["nix-build", "<nixpkgs/nixos>", "--no-out-link", "-A", "system", "--option", "extra-binary-caches", "ssh://nix-ssh@server"], 0x1ca2008 /* 56 vars */) = 0

I first suspected this to be a nixissue, hence my report at NixOS/nix#3314

To Reproduce
Steps to reproduce the behavior:
please see NixOS/nix#3314

Expected behavior
nixos-rebuiöd should fetch all missing binaries from the ssh-served store, then rebuild the system.

Additional context
Add any other context about the problem here.

Metadata
Client

- system: `"x86_64-linux"`
 - host os: `Linux 4.19.101, NixOS, 19.09.2070.b9cb3b2fb2f (Loris)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.2`
 - channels(root): `"nixos-19.09.2070.b9cb3b2fb2f, unstable-20.03pre212208.8130f3c1c2b"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Server

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.102, NixOS, 19.09.2070.b9cb3b2fb2f (Loris)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.2`
 - channels(root): `"nixos-19.09.2070.b9cb3b2fb2f, unstable-20.03pre212208.8130f3c1c2b"`
 - channels(markus): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: system
# a list of nixos modules affected by the problem
module:

[danbst]

reading through NixOS/nix#1994 it seems NIX_SSHOPTS isn't supported for ssh:// paths.
The best thing you could do would to create custom ssh_config and pass as a parameter, described in NixOS/nix#1994 (comment)

[wamserma]

reading through NixOS/nix#1994 it seems NIX_SSHOPTS isn't supported for ssh:// paths.
The best thing you could do would to create custom ssh_config and pass as a parameter, described in NixOS/nix#1994 (comment)

No, NIX_SSHOPTS works fine when nix-build/ nix build is invoked directly. NixOS/nix#1994 is about passing the port number in the ssh://-URL instead of NIX_SSHOPTS. This is issue is about NIX_SSHOPTS somehow not getting properly passed to nix-build/ nix build when called through nixos-rebuild. I tried some debbuging with local nixpkgs (because nixos-rebuild re-launches itself after updating) and tracing (ob bash and strace level) on and so forth, but couldn't pinpoint, less fix the issue on my own.

[danbst]

ok. I've debugged the issue and figure out that NIX_SSHOPTS passes fine up to inner nix-build. There may be issues when you use remote nixos operations, but for stock switch it passes fine.

Can you try

nix-build '<nixpkgs/nixos>' -A system \
  --option extra-binary-caches ssh://nix-ssh@server \
  --option builders ssh://nix-ssh@server \
  -j 0

with your NIX_SSHOPTS hook?

For me, NIX_SSHOPTS doesn't work at all. What works, is setting ssh host in /etc/ssh/ssh_config:

Host remote_host
  IdentityFile /root/.ssh/private.key
  User your_user

[wamserma]

Can you try

nix-build '<nixpkgs/nixos>' -A system \
  --option extra-binary-caches ssh://nix-ssh@server \
  --option builders ssh://nix-ssh@server \
  -j 0

with your NIX_SSHOPTS hook?

This works fine, just as expected. In fact my current workaround is a snippet of bash doing

export NIX_SSHOPTS="-i /root/.ssh/id_nix_store -o ServerAliveInterval=15 -o ServerAliveCountMax=3000 -o IPQoS=throughput"

nix-channel --update
nix build  --no-link '(with import <nixpkgs/nixos> { }; system)' --option extra-binary-caches ssh://nix-ssh@server
nixos-rebuild switch

For me, NIX_SSHOPTS doesn't work at all. What works, is setting ssh host in /etc/ssh/ssh_config:

did you mean /root/.ssh/ssh_config?

Host remote_host
  IdentityFile /root/.ssh/private.key
  User your_user

Well that didn't make a difference. On the server I still see:

Feb 18 --:--:-- server sshd[2767]: Connection from 192.168.XXX.XXX port 36508 on 192.168.YYY.YYY port 22
Feb 18 --:--:-- server sshd[2767]: error: PAM: Authentication failure for nix-ssh from 192.168.XXX.XXX
Feb 18 --:--:-- server sshd[2767]: Failed keyboard-interactive/pam for nix-ssh from 192.168.XXX.XXX port 36508 ssh2
Feb 18 --:--:-- server sshd[2767]: Connection closed by authenticating user nix-ssh 192.168.XXX.XXX port 36508 [preauth]

but when calling nix-build directly I can observe

Feb 18 --:--:-- server sshd[2842]: Connection from 192.168.XXX.XXX port 36534 on 192.168.YYY.YYY port 22
Feb 18 --:--:-- server sshd[2842]: Accepted key ED25519 SHA256:xxxxxxxx found at /etc/ssh/authorized_keys.d/nix-ssh:2
Feb 18 --:--:-- server sshd[2842]: Postponed publickey for nix-ssh from 192.168.XXX.XXX port 36534 ssh2 [preauth]
Feb 18 --:--:-- server sshd[2842]: Accepted key ED25519 SHA256:xxxxxxxx found at /etc/ssh/authorized_keys.d/nix-ssh:2
Feb 18 --:--:-- server sshd[2842]: Accepted publickey for nix-ssh from 192.168.XXX.XXX port 36534 ssh2: ED25519 SHA256:xxxxxxxx
Feb 18 --:--:-- server sshd[2842]: pam_unix(sshd:session): session opened for user nix-ssh by (uid=0)
Feb 18 --:--:-- server sshd[2842]: User child is on pid 2850

Both machines on 19.09-stable and up-to-date (well, for testing this is let the client fall behind a few days).

[danbst]

okay, I give up. NIX_SSHOPTS doesn't work at all for me, I can't pass any SSH arguments even to nix-build. /root/.ssh/config and /etc/ssh/ssh_config both work for me though.

[wamserma]

That, in turn, seems strange to me as I have a pretty standard out-of-the box setup.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[dzmitry-lahoda]

so no way to use non default ssh key with nixos-rebuild switch?

[dzmitry-lahoda]

I tested.

NIX_SSHOPTS="-i ./terraform/testnet/id_rsa.pem"         
export NIX_SSHOPTS
nixos-rebuild switch --fast --flake .#node-test-a  --target-host root@ec2-3-250-164-158.eu-west-1.compute.amazonaws.com

It worked for me. Nix 22.11. Issue in theory may be closed, so may be where one could add docs about it?

[imincik]

In case it helps to others, I use following command for deployment using no-standard SSH port:

NIX_SSHOPTS="-p <SSH-PORT>" nixos-rebuild switch --use-remote-sudo --fast --flake .#<HOST-NAME> --target-host <USER>@<HOST>

<USER> must exist in nix.settings.trusted-users = [ <USER> <@GROUP> ]; on target host.

Using NixOS 22.11