Reboot, and exit successfully without returning. (Encrypted disk)

[gnat]

Is your feature request related to a problem? Please describe

Is there any way to run reboot and immediately trigger success or exit pyinfra successfully?

Use Case

Encrypted servers. Pyinfra shouldn't expect to come back from a reboot when servers must have their disks decrypted first.

This is an important feature in 2021 because we now have GDPR and other new data protection laws which put a lot of pressure on us to encrypt user data at rest.

Error when you try to reboot to an encrypted disk. ignore_errors=True doesn't solve this unfortunately.

--> Starting operation: /home/USER/Desktop/project/infra/tasks/all/reboot.py | Server/Shell (reboot)
    [SERVER_IP] Error (ignored)

--> Results:
    Groups: production / hosts
    [SERVER_IP]   Successful: 107   Errors: 1   Commands: 107/108   
Traceback (most recent call last):
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra_cli/main.py", line 218, in cli
    _main(*args, **kwargs)
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra_cli/main.py", line 610, in _main
    _exit()
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra_cli/main.py", line 64, in _exit
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USER/.local/bin/pyinfra", line 5, in <module>
    from pyinfra_cli.__main__ import execute_pyinfra
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra_cli/__main__.py", line 43, in <module>
    cli()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra_cli/main.py", line 240, in cli
    disconnect_all(pseudo_state)
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra/api/connect.py", line 48, in disconnect_all
    host.disconnect()  # normally a noop
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra/api/host.py", line 194, in disconnect
    remove_any_sudo_askpass_file(self)
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra/api/connectors/util.py", line 193, in remove_any_sudo_askpass_file
    host.run_shell_command('rm -f {0}'.format(SUDO_ASKPASS_EXE_FILENAME))
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/USER/.local/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 286, in run_shell_command
    stdin_buffer, stdout_buffer, stderr_buffer = host.connection.exec_command(
  File "/home/USER/.local/lib/python3.8/site-packages/paramiko/client.py", line 508, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/USER/.local/lib/python3.8/site-packages/paramiko/transport.py", line 875, in open_session
    return self.open_channel(
  File "/home/USER/.local/lib/python3.8/site-packages/paramiko/transport.py", line 969, in open_channel
    raise SSHException("SSH session not active")
paramiko.ssh_exception.SSHException: SSH session not active

[gnat]

An option to immediately trigger success of a task is the quick and easy solution that is workable for most users right now, but I'm also open to the idea of pyinfra gaining the ability to decrypt the server itself (SSH into dropbear, run cryptroot-unlock) but that may be more of a future wishlist item.

[gnat]

Been thinking about decryption a little more for the wishlist item in the future. There's quite a few ways to decrypt at boot. Thought I'd add my notes here.

Non-Interactive:

• echo -ne "password" > /lib/cryptsetup/passfifo
• ssh -i key user@IP "echo -ne \"password\" > /lib/cryptsetup/passfifo"

Interactive:

• /lib/cryptsetup/askpass "Enter password:" > /lib/cryptsetup/passfifo
• /bin/cryptroot-unlock
  • This wrapper script seems to be slightly unreliable compared to the above methods in my experience.

[Fizzadar]

Hi @gnat apologies for taking ages to check in on this one; first up I'd like ignore_errors to work as expected here, even if it waits for the SSH port to return and then errors I think that's better than nothing, also gives some indication the server has come back.

Regarding decryption - does that mean one can echo the password into /lib/cryptsetup/passfifo, then reboot, and have the system automatically decrypt itself on boot?

I'm not familiar with disk encryption on Linux so am going to setup a test VM and play around a bit - I'd definitely like to provide support for this!

[Fizzadar]

(added bug label for ignore_errors not working on server.reboot in some situations).

[gnat]

Hi @gnat apologies for taking ages to check in on this one; first up I'd like ignore_errors to work as expected here, even if it waits for the SSH port to return and then errors I think that's better than nothing

@Fizzadar Definitely agreed. Thank you for looking into this!

I'm not familiar with disk encryption on Linux so am going to setup a test VM and play around a bit - I'd definitely like to provide support for this!

This method is applicable to the default LUKS (dm-crypt) encryption in most Linux distros.

For remote unlock, there is generally a boot-time ssh daemon added to initramfs (in practice this is usually dropbear, but could also just be sshd). Thankfully, for simplicity, you interact with this exactly the same way as sshd from pyinfra's perspective.

All of that said, this one-liner will do everything required to support decryption: ssh -i key user@IP "echo -ne \"password\" > /lib/cryptsetup/passfifo"

Regarding decryption - does that mean one can echo the password into /lib/cryptsetup/passfifo, then reboot, and have the system automatically decrypt itself on boot?

It really is as simple as an echo into /lib/cryptsetup/passfifo

On success:

1. Your current SSH session will close.
2. Boot process automatically continues as normal.
3. The normal SSH daemon (sshd) becomes available to log in as normal.

There's no reboot involved post-decryption, just to be crystal clear.

[Fizzadar]

v1.4.9 (released today) should fix the exception raised above; fixed by 4758930.