nixos/test-driver: add backdoor based on systemd-ssh-proxy & AF_VSOCK #392030
Conversation
github-actions
bot
added
6.topic: nixos
wegank
added
the
2.status: merge conflict
Ma27
force-pushed
the
testdriver-ssh-backdoor-interactive
branch
from
057f211 to
0200cad
Compare
github-actions
bot
removed
the
6.topic: systemd
ofborg
bot
removed
the
2.status: merge conflict
github-actions
bot
added
10.rebuild-darwin: 0
With this it's possible to trivially SSH into running machines from the test-driver. This is especially useful when running VM tests interactively on a remote system. This is based on `systemd-ssh-proxy(1)`, so there's no need to configure any additional networking on the host-side. Suggested-by: Ryan Lahfa <masterancpp@gmail.com>
Ma27
force-pushed
the
testdriver-ssh-backdoor-interactive
branch
from
0200cad to
1bd8073
Compare
|
@tfc this would be the next one in line: it essentially adds an optional SSH backdoor that can be used on interactive runs, based on AF_VSOCK (so you can skip key management and don't need to modify the networking for the test which is probably useful on networking-related testcases). I don't really know Darwin, so I can't say how easy/hard it'd be to get it to run there. |
…door Co-authored-by: Ryan Lahfa <masterancpp@gmail.com>
|
@Ma27 when activated, it would be great if users don't need to find out which VM to reach via which magic vsock number. can you add a print message please that is something like |
|
@tfc nice idea! |
I'm a little annoyed at myself that I only realized this _after_ NixOS#392030 got merged. But I realized that if something else is using AF_VSOCK or you simply have another interactive test running (e.g. by another user on a larger builder), starting up VMs in the driver fails with qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=3: vhost-vsock: unable to set guest cid: Address already in use Multi-user setups are broken anyways because you usually don't have permissions to remove the VM state from another user and thus starting the driver fails with PermissionError: [Errno 13] Permission denied: PosixPath('/tmp/vm-state-machine') but this is something you can work around at least. I was considering to generate random offsets, but that's not feasible given we need to know the numbers at eval time to inject them into the QEMU args. Also, while we could do this via the test-driver, we should also probe if the vsock numbers are unused making the code even more complex for a use-case I consider rather uncommon. Hence the solution is to do sshBackdoor.vsockOffset = 23542; when encountering conflicts.
I'm a little annoyed at myself that I only realized this _after_ NixOS#392030 got merged. But I realized that if something else is using AF_VSOCK or you simply have another interactive test running (e.g. by another user on a larger builder), starting up VMs in the driver fails with qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=3: vhost-vsock: unable to set guest cid: Address already in use Multi-user setups are broken anyways because you usually don't have permissions to remove the VM state from another user and thus starting the driver fails with PermissionError: [Errno 13] Permission denied: PosixPath('/tmp/vm-state-machine') but this is something you can work around at least. I was considering to generate random offsets, but that's not feasible given we need to know the numbers at eval time to inject them into the QEMU args. Also, while we could do this via the test-driver, we should also probe if the vsock numbers are unused making the code even more complex for a use-case I consider rather uncommon. Hence the solution is to do sshBackdoor.vsockOffset = 23542; when encountering conflicts.
|
Is it possible to set |
That's exactly the point of this, i.e. easier ssh for interactive debugging. There's also #392117 which allows the same in the sandbox (and pdb), but I expect this to be a little more controversial. |
|
Right, but it's not enabled by default when using |
|
Oh you mean by default! No, that's not the case yet. |
|
I'm getting the following error on non-interactive tests (which have the new option enabled), after having used the vsock stuff interactively: It seems that something got messed up on my system after the interactive test run exited. |
|
I'll take a look tomorrow. |
|
So, I was a little too tired to spot it last evening, but: the vsock stuff only works when passing This is a debugging tool only. Considering that the Anyways, I agree though that the error-message is bad and that this is not sufficiently documented. I guess I'll file a follow-up for this soon :) |
|
Ah ok, yeah I think that it's OK then, but indeed it's confusing if you're not aware of this, since the error message is not very obvious. Maybe there's a way to detect that a test has this backdoor enabled and is being run non-interactively, so that we can print a clearer error message to explain what's going on. Or we could ignore this option for non-interactive builds, so that it just gets ignored in that case. |
Note: Only the last commit is relevant right now.
Depends on #390996 & #372979.
With this it's possible to trivially SSH into running machines from the
test-driver. This is especially useful when running VM tests
interactively on a remote system.
This is based on
systemd-ssh-proxy(1), so there's no need to configureany additional networking on the host-side.
Suggested-by: Ryan Lahfa masterancpp@gmail.com
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.