sshd not starting after recent patch

[ellisonpatterson]

Describe the bug

Jul 01 06:56:14 op3 (sshd)[1612]: sshd.service: Referenced but unset environment variable evaluates to an empty string: CRYPTO_POLICY, OPTIONS, SSHD_OPTS
Jul 01 06:56:14 op3 sshd[1612]: /usr/lib64/misc/sshd-session does not exist or is not executable
Jul 01 06:56:14 op3 systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 01 06:56:14 op3 systemd[1]: sshd.service: Failed with result 'exit-code'.
Jul 01 06:56:14 op3 systemd[1]: Failed to start OpenSSH server daemon.
Jul 01 06:56:33 op3 systemd[1]: Stopped OpenSSH server daemon.
Jul 01 06:56:39 op3 systemd[1]: Starting OpenSSH server daemon...
Jul 01 06:56:39 op3 systemd[1]: Started OpenSSH server daemon.

Dracut shell with debug enabled

Jul 01 07:42:46 op3 systemd[1]: sshd.service: Service restart interval 42s expired, scheduling restart.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Trying to enqueue job sshd.service/start/restart-dependencies
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Installed new job sshd.service/start as 324
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Enqueued job sshd.service/start as 324
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Scheduled restart job, restart counter is at 4.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Changed auto-restart -> auto-restart-queued
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Will spawn child (service_enter_start): /usr/sbin/sshd
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Passing 0 fds to service
Jul 01 07:42:46 op3 systemd[1]: sshd.service: About to execute: /usr/sbin/sshd -D "\$SSHD_OPTS" "\$OPTIONS" "\$CRYPTO_POLICY"
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Forked /usr/sbin/sshd as 1639
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Changed auto-restart-queued -> start
Jul 01 07:42:46 op3 systemd[1]: Starting OpenSSH server daemon...
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Child 1639 belongs to sshd.service.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Failed with result 'exit-code'.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Service will restart (restart setting)
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Changed start -> failed-before-auto-restart
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Job 324 sshd.service/start finished, result=failed
Jul 01 07:42:46 op3 systemd[1]: Failed to start OpenSSH server daemon.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Unit entered failed state.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Consumed 4ms CPU time.
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Next restart interval calculated as: 42s
Jul 01 07:42:46 op3 systemd[1]: sshd.service: Changed failed-before-auto-restart -> auto-restart

Distribution used
Gentoo
Kernel 6.9.7

Dracut version
sys-kernel/dracut-102

openssh-server version
net-misc/openssh-9.8_p1

Init system
SystemD

To Reproduce
Build initramfs with recent openssh-server patch.

Additional context
Add any other context you like about the problem here.

[ellisonpatterson]

Just confirmed in dracut shell that /usr/lib64/misc doesn't even exist.

[MartinX3]

I am not sure if it is connected to dracut.
I just updated it on my server without a restart.
I can't connect to my arch linux server again.

Here is an issue ticket about it
https://gitlab.archlinux.org/archlinux/packaging/packages/openssh/-/issues/5

It seems that at least if you didn't restart your server a restart of the sshd service in the same session you did the update will fix the access for your server access.
Of course I am too scared now to restart the server since I don't know if this will break the access forever.
(So better doing a downgrade).

[ghen2]

Jul 01 06:56:14 op3 sshd[1612]: /usr/lib64/misc/sshd-session does not exist or is not executable

Your environment is missing the new sshd-session binary.

From https://www.openssh.com/releasenotes.html#9.8p1:

 * sshd(8): the server has been split into a listener binary, sshd(8),
   and a per-session binary "sshd-session". This allows for a much
   smaller listener binary, as it no longer needs to support the SSH
   protocol. As part of this work, support for disabling privilege
   separation (which previously required code changes to disable) and
   disabling re-execution of sshd(8) has been removed. Further
   separation of sshd-session into additional, minimal binaries is
   planned for the future.

[ellisonpatterson]

Could we just include that file for now?
install_optional_items+=" /usr/lib64/misc/sshd-session "

[ellisonpatterson]

That resolves the issue for me, so it is related to dracut-sshd.

[MartinX3]

@ellisonpatterson please reopen the issue.
It needs to be fixed in dracut since ssh support is a buildin feature.

[ellisonpatterson]

@ellisonpatterson please reopen the issue. It needs to be fixed in dracut since ssh support is a buildin feature.

I apologize, it is re-open now.

[ellisonpatterson]

@ellisonpatterson please reopen the issue. It needs to be fixed in dracut since ssh support is a buildin feature.

Wait, the only module using OpenSSH is 95ssh-client, I don't believe the client needs sshd-session?

[ghen2]

Correct, it's a server binary.

[ellisonpatterson]

Correct, it's a server binary.

Well then for the dracut-sshd module I just made a PR incase anyone else is using it.

gsauthof/dracut-sshd#79