Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sshd.socket enters failed state when sshd connection is gracefully terminated #29897

Closed
sshedi opened this issue Nov 7, 2023 · 3 comments
Closed

sshd.socket enters failed state when sshd connection is gracefully terminated #29897

sshedi opened this issue Nov 7, 2023 · 3 comments
Labels
bug 🐛 Programming errors, that need preferential fixing pid1

Comments

@sshedi
Copy link
Contributor

sshedi commented Nov 7, 2023

systemd version the issue has been seen with

254.1

Used distribution

Photon OS

Linux kernel version used

6.1.28

CPU architectures issue was seen on

x86_64

Component

systemctl

Expected behaviour you didn't see

sshd connections should be terminated properly and systemctl --failed should not show failed sshd entries.

Unexpected behaviour you saw

Nov 07 20:33:07 phdev sshd[773]: debug1: do_cleanup
Nov 07 20:33:07 phdev sshd[773]: debug1: PAM: cleanup
Nov 07 20:33:07 phdev sshd[773]: debug1: PAM: closing session
Nov 07 20:33:07 phdev sshd[773]: pam_unix(sshd:session): session closed for user root
Nov 07 20:33:07 phdev sshd[773]: debug1: PAM: deleting credentials
Nov 07 20:33:07 phdev sshd[773]: debug3: PAM: sshpam_thread_cleanup entering
Nov 07 20:33:07 phdev systemd[1]: sshd@0-10.197.103.248:22-10.104.65.96:53997.service: Main process exited, code=exited, status=255/EXCEPTION
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit sshd@0-10.197.103.248:22-10.104.65.96:53997.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 255.
Nov 07 20:33:07 phdev systemd[1]: sshd@0-10.197.103.248:22-10.104.65.96:53997.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd

And later when I did systemctl --failed

root@phdev [ ~ ]# systemctl --failed
  UNIT                                                LOAD   ACTIVE SUB    DESCRIPTION                              >
● sshd@0-10.197.103.248:22-10.104.65.96:53997.service loaded failed failed SSH Per-Connection Server (10.104.65.96:5>

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.

Here is my sshd config.

root@phdev [ ~ ]# sshd -T
port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
maxauthtries 4
maxsessions 10
clientaliveinterval 0
clientalivecountmax 2
requiredrsasize 1024
streamlocalbindmask 0177
unusedconnectiontimeout none
permitrootlogin yes
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
gssapiauthentication no
gssapicleanupcredentials yes
passwordauthentication yes
kbdinteractiveauthentication yes
printmotd yes
printlastlog yes
x11forwarding no
x11uselocalhost yes
permittty yes
permituserrc yes
strictmodes yes
tcpkeepalive no
permitemptypasswords no
compression no
gatewayports no
usedns no
allowtcpforwarding no
allowagentforwarding no
disableforwarding no
allowstreamlocalforwarding yes
streamlocalbindunlink no
fingerprinthash SHA256
exposeauthinfo no
pidfile /var/run/sshd.pid
modulifile /etc/ssh/moduli
xauthlocation /usr/X11R6/bin/xauth
ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
banner none
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum none
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
casignaturealgorithms ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostbasedacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostkeyalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
pubkeyacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
loglevel DEBUG3
syslogfacility AUTH
authorizedkeysfile .ssh/authorized_keys
hostkey /etc/ssh/ssh_host_rsa_key
hostkey /etc/ssh/ssh_host_ecdsa_key
hostkey /etc/ssh/ssh_host_ed25519_key
authenticationmethods any
channeltimeout none
subsystem sftp /usr/libexec/sftp-server
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
permittunnel no
ipqos af21 cs1
rekeylimit 0 0
permitopen any
permitlisten any
permituserenvironment no
pubkeyauthoptions none

Steps to reproduce the problem

systemctl enable --now sshd.socket
Do ssh and exit using Ctrl-D
Do ssh again, check systemctl --failed

Additional program output to the terminal or log subsystem illustrating the issue

No response
@sshedi sshedi added the bug 🐛 Programming errors, that need preferential fixing label Nov 7, 2023
@sshedi
Copy link
Contributor Author

sshedi commented Nov 7, 2023

And tried same steps on Fedora 38, this doesn't happen. I'm guessing there is some configuration in systemd I'm not able to figure out. Need assistance in finding it out. Thanks.

@sshedi sshedi changed the title sshd.socket enters failed state upon sshd connection is terminated sshd.socket enters failed state upon sshd connection is gracefully terminated Nov 7, 2023
@sshedi sshedi changed the title sshd.socket enters failed state upon sshd connection is gracefully terminated sshd.socket enters failed state when sshd connection is gracefully terminated Nov 7, 2023
@YHNdnzj YHNdnzj added pid1 and removed systemctl labels Nov 7, 2023
@poettering
Copy link
Member

How does your sshd@.service unit template file look like?

Should have a line like this:

ExecStart=-/usr/sbin/sshd -i

the important part is the - after the =

@sshedi
Copy link
Contributor Author

sshedi commented Nov 7, 2023

@poettering Awesome, this worked. - was missing in my unit file, spent a lot of time on this. Thanks for saving my day.

gerrit-photon pushed a commit to vmware/photon that referenced this issue Nov 9, 2023
@sshedi @tapakund
openssh: fix sshd.socket failure issue
d001965 
More info at:
systemd/systemd#29897

Change-Id: Idd1180f21e7e5b14faaca13417bdb4dea7410c55
Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/c/photon/+/22306
Tested-by: gerrit-photon <photon-checkins@vmware.com>
Reviewed-by: Tapas Kundu <tkundu@vmware.com>
gerrit-photon pushed a commit to vmware/photon that referenced this issue Nov 9, 2023
@sshedi @tapakund
openssh: fix sshd.socket failure issue
6a356b6 
More info at:
systemd/systemd#29897

Change-Id: I6481acfcdfc4a8971abed9935e49cd393b7f1086
Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/c/photon/+/22310
Tested-by: gerrit-photon <photon-checkins@vmware.com>
Reviewed-by: Tapas Kundu <tkundu@vmware.com>
gerrit-photon pushed a commit to vmware/photon that referenced this issue Nov 9, 2023
@sshedi @tapakund
openssh: fix sshd.socket failure issue
1c7eb08 
More info at:
systemd/systemd#29897

Fixed a mishap in sshd homedir location. Newer sshd uses
/usr/share/empty.sshd as its home dir.

Change-Id: I6481acfcdfc4a8971abed9935e49cd393b7f1086
Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/c/photon/+/22309
Reviewed-by: Tapas Kundu <tkundu@vmware.com>
Tested-by: Tapas Kundu <tkundu@vmware.com>
@sshedi sshedi mentioned this issue Mar 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐛 Programming errors, that need preferential fixing pid1
Milestone
No milestone
Development

No branches or pull requests
3 participants
@poettering @YHNdnzj @sshedi