Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sssd_pam segfaults during password-based SSH-login #7061

Closed
elpres opened this issue Nov 29, 2023 · 8 comments
Closed

sssd_pam segfaults during password-based SSH-login #7061

elpres opened this issue Nov 29, 2023 · 8 comments
Assignees
@justin-stephenson
Labels
Closed: Fixed Issue was closed as fixed. passkey Issues and PRs related to 'passkey' feature

Comments

@elpres
Copy link

elpres commented Nov 29, 2023

Immediately after upgrading a server from Fedora 38 to 39 SSH started rejecting password-authenticated connection attempts with "Permission denied". Luckily it was still possible to log in with a Kerberos ticket and I discovered that sssd_pam was crashing:

Relevant section from the journal: 
Nov 29 11:11:21 server001.domain001.local krb5_child[362908]: Pre-authentication failed: Invalid argument
Nov 29 11:11:21 server001.domain001.local audit[362884]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=362884 comm="sssd_pam" exe="/usr/libexec/sssd/sssd_pam" sig=11 res=1
Nov 29 11:11:21 server001.domain001.local kernel: sssd_pam[362884]: segfault at 0 ip 000055c7f7510723 sp 00007ffc8acc8db0 error 4 in sssd_pam[55c7f74ec000+28000] likely on CPU 57 (core 1, socket 3)
Nov 29 11:11:21 server001.domain001.local kernel: Code: c2 48 8d 05 77 67 00 00 48 89 45 80 48 89 c7 31 c0 e8 f1 d7 fd ff 48 c7 85 70 ff ff ff 01 00 00 00 49 8b 44 24 18 48 8b 5d a0 <48> 8b 30 44 0f b6 7b 48 48 89 df e8 3d cc fd ff 48 c7 45 98 00 00
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=217 op=LOAD
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=218 op=LOAD
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=219 op=LOAD
Nov 29 11:11:21 server001.domain001.local systemd[1]: Started systemd-coredump@3-362909-0.service - Process Core Dump (PID 362909/UID 0).
Nov 29 11:11:21 server001.domain001.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@3-362909-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 29 11:11:21 server001.domain001.local systemd-coredump[362910]: Process 362884 (sssd_pam) of user 0 dumped core.

Module tdb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module skel.so from rpm libldb-2.8.0-1.fc39.x86_64
Module server_sort.so from rpm libldb-2.8.0-1.fc39.x86_64
Module sample.so from rpm libldb-2.8.0-1.fc39.x86_64
Module rdn_name.so from rpm libldb-2.8.0-1.fc39.x86_64
Module paged_searches.so from rpm libldb-2.8.0-1.fc39.x86_64
Module memberof.so from rpm sssd-2.9.3-1.fc39.x86_64
Module mdb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module liblmdb.so.0.0.0 from rpm lmdb-0.9.31-2.fc39.x86_64
Module libldb-tdb-err-map.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-key-value.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-mdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-tdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64
Module ldb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-2.fc39.x86_64
Module libssl.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libevent-2.1.so.7 from rpm libevent-2.1.12-9.fc39.x86_64
Module ldap.so from rpm libldb-2.8.0-1.fc39.x86_64
Module asq.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libpath_utils.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libz.so.1 from rpm zlib-1.2.13-4.fc39.x86_64
Module libzstd.so.1 from rpm zstd-1.5.5-4.fc39.x86_64
Module liblzma.so.5 from rpm xz-5.4.4-1.fc39.x86_64
Module liblz4.so.1 from rpm lz4-1.9.4-4.fc39.x86_64
Module libcap.so.2 from rpm libcap-2.48-7.fc39.x86_64
Module libsss_cert.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libcollection.so.4 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libref_array.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libini_config.so.5 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc39.2.x86_64
Module libunistring.so.5 from rpm libunistring-1.1-5.fc39.x86_64
Module libdbus-1.so.3 from rpm dbus-1.14.10-1.fc39.x86_64
Module libcrypto.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libkeyutils.so.1 from rpm keyutils-1.6.1-7.fc39.x86_64
Module libkrb5support.so.0 from rpm krb5-1.21.2-2.fc39.x86_64
Module libcom_err.so.2 from rpm e2fsprogs-1.47.0-2.fc39.x86_64
Module libk5crypto.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libkrb5.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libeconf.so.0 from rpm libeconf-0.5.2-1.fc39.x86_64
Module libaudit.so.1 from rpm audit-3.1.2-5.fc39.x86_64
Module libtalloc.so.2 from rpm libtalloc-2.4.1-1.fc39.x86_64
Module libtevent.so.0 from rpm libtevent-0.15.0-1.fc39.x86_64
Module libdhash.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libsss_sbus.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_iface.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsystemd.so.0 from rpm systemd-254.5-2.fc39.x86_64
Module libsss_debug.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_child.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_crypt.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libtdb.so.1 from rpm libtdb-1.4.9-1.fc39.x86_64
Module libselinux.so.1 from rpm libselinux-3.5-5.fc39.x86_64
Module libldb.so.2 from rpm libldb-2.8.0-1.fc39.x86_64
Module libpopt.so.0 from rpm popt-1.19-3.fc39.x86_64
Module libsss_util.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_certmap.so.0 from rpm sssd-2.9.3-1.fc39.x86_64
Module libgssapi_krb5.so.2 from rpm krb5-1.21.2-2.fc39.x86_64
Module libpam.so.0 from rpm pam-1.5.3-3.fc39.x86_64
Module sssd_pam from rpm sssd-2.9.3-1.fc39.x86_64
Stack trace of thread 362884:
#0  0x000055c7f7510723 pam_passkey_auth_send.isra.0 (sssd_pam + 0x2d723)
#1  0x000055c7f75118a0 pam_passkey_get_user_done (sssd_pam + 0x2e8a0)
#2  0x000055c7f750ea11 pam_passkey_get_mapping_done (sssd_pam + 0x2ba11)
#3  0x00007f9e3f364e40 tevent_common_invoke_immediate_handler (libtevent.so.0 + 0xbe40)
#4  0x00007f9e3f364ea2 tevent_common_loop_immediate (libtevent.so.0 + 0xbea2)
#5  0x00007f9e3f368a22 epoll_event_loop_once (libtevent.so.0 + 0xfa22)
#6  0x00007f9e3f360894 std_event_loop_once (libtevent.so.0 + 0x7894)
#7  0x00007f9e3f362e1b _tevent_loop_once (libtevent.so.0 + 0x9e1b)
#8  0x00007f9e3f362f6b tevent_common_loop_wait (libtevent.so.0 + 0x9f6b)
#9  0x00007f9e3f360914 std_event_loop_wait (libtevent.so.0 + 0x7914)
#10 0x00007f9e3f5e9a6f server_loop (libsss_util.so + 0x50a6f)
#11 0x000055c7f74f1a63 main (sssd_pam + 0xea63)
#12 0x00007f9e3f18914a __libc_start_call_main (libc.so.6 + 0x2814a)
#13 0x00007f9e3f18920b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820b)
#14 0x000055c7f74f2245 _start (sssd_pam + 0xf245)
ELF object binary architecture: AMD x86-64

Nov 29 11:11:21 server001.domain001.local audit[362896]: USER_AUTH pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=PAM:authentication grantors=? acct="user001" exe="/usr/sbin/sshd" hostname=192.168.2.69 addr=192.168.2.69 terminal=ssh res=failed'
Nov 29 11:11:21 server001.domain001.local audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@3-362909-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 29 11:11:21 server001.domain001.local systemd[1]: systemd-coredump@3-362909-0.service: Deactivated successfully.
Nov 29 11:11:21 server001.domain001.local sssd_pam[362919]: Starting up
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=219 op=UNLOAD
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=218 op=UNLOAD
Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=217 op=UNLOAD
Nov 29 11:11:22 server001.domain001.local abrt-server[362925]: Deleting problem directory ccpp-2023-11-29-11:11:21.843487-362884 (dup of ccpp-2023-11-28-13:33:41.300493-2042)
Nov 29 11:11:22 server001.domain001.local abrt-notification[363023]: Process 2042 (sssd_pam) crashed in pam_passkey_auth_send.isra.0()
Nov 29 11:11:24 server001.domain001.local sshd[362896]: Failed password for user001 from 192.168.2.69 port 47794 ssh2
Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=session fp=? direction=both spid=362897 suid=74 rport=47794 laddr=192.168.2.88 lport=22  exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success'
Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=server fp=SHA256:40:c8:ad:c8:51:db:93:78:c3:08:4a:11:ad:e8:f3:d9:16:05:16:8f:8c:89:a1:68:f3:c8:2f:db:e0:ae:79:05 direction=? spid=362897 suid=74  exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success'
Nov 29 11:11:26 server001.domain001.local sshd[362896]: Connection closed by authenticating user user001 192.168.2.69 port 47794 [preauth]
Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=server fp=SHA256:40:c8:ad:c8:51:db:93:78:c3:08:4a:11:ad:e8:f3:d9:16:05:16:8f:8c:89:a1:68:f3:c8:2f:db:e0:ae:79:05 direction=? spid=362896 suid=0  exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success'
Nov 29 11:11:26 server001.domain001.local audit[362896]: USER_LOGIN pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=login acct="user001" exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=ssh res=failed'

Also, while trying to understand what's wrong I tried logging in from a local account into an AD-backed one, and instead of being asked for a password I'm getting this message that I've never seen before:

Kerberos TGT will not be granted upon login, user experience will be affected.
Insert your passkey device, then press ENTER.

Communication with AD seems to be fine, at least things like getent passwd $USERNAME behave as expected and I can get a ticket with kinit. Both cases (the stack trace and the login message) refer to passkeys, although nothing related to them was ever configured on the server or is desired at the moment. I also manually removed sssd-passkey which seems to have been installed during the upgrade to F39, but with no effect.

sssd.conf.txt
sssd_pam.log

What other logs should I provide?
@alexey-tikhonov
Copy link
Member

alexey-tikhonov commented Nov 29, 2023 •
edited

Hi,

What other logs should I provide?

'rpm -q sssd' output and coredump would be great.

@alexey-tikhonov alexey-tikhonov added the passkey Issues and PRs related to 'passkey' feature label Nov 29, 2023
@elpres
Copy link
Author

elpres commented Nov 29, 2023 

# rpm -q sssd
sssd-2.9.3-1.fc39.x86_64

# rpm -qa | grep sssd-
sssd-common-pac-2.9.3-1.fc39.x86_64
sssd-krb5-common-2.9.3-1.fc39.x86_64
sssd-proxy-2.9.3-1.fc39.x86_64
sssd-ad-2.9.3-1.fc39.x86_64
sssd-krb5-2.9.3-1.fc39.x86_64
sssd-ldap-2.9.3-1.fc39.x86_64
sssd-ipa-2.9.3-1.fc39.x86_64
sssd-2.9.3-1.fc39.x86_64
sssd-nfs-idmap-2.9.3-1.fc39.x86_64
sssd-client-2.9.3-1.fc39.x86_64
sssd-common-2.9.3-1.fc39.x86_64
sssd-dbus-2.9.3-1.fc39.x86_64
sssd-tools-2.9.3-1.fc39.x86_64
sssd-idp-2.9.3-1.fc39.x86_64
sssd-kcm-2.9.3-1.fc39.x86_64

I scrubbed sensitive data from the log above, but it will probably be fully readable in the coredump. How can I make it available in a secure way?

@justin-stephenson
Copy link
Contributor

Hi,

You can try adding pam_passkey_auth = False to the [pam] section of sssd.conf as a workaround for now. The new passkey feature allows storing passkey data in the AD altSecurityIdentities attribute so there is an issue here we will investigate.

I scrubbed sensitive data from the log above, but it will probably be fully readable in the coredump. How can I make it available in a secure way?

You can email our sssd developer team list sssd-maint@redhat.com (or me directly at jstephen@redhat.com) if it's okay for you.

@elpres
Copy link
Author

elpres commented Nov 29, 2023

@justin-stephenson Thanks for the tip about pam_passkey_auth = False, it indeed resolved the problem! Also, I emailed the coredump to you.

@justin-stephenson justin-stephenson self-assigned this Dec 1, 2023
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 1, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
bc66d7b 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, of ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

Resolves: SSSD#7061
@justin-stephenson justin-stephenson mentioned this issue Dec 1, 2023
Closed
@justin-stephenson
Copy link
Contributor

Hi @elpres thank you for the coredump. Would you be able to test the fix (removing pam_passkey_auth = False) from the COPR build provided in #7066 ?

@alexey-tikhonov
Copy link
Member

( https://copr.fedorainfracloud.org/coprs/g/sssd/pr7066/ )

@elpres
Copy link
Author

elpres commented Dec 4, 2023

The fix looks good to me; logins went back to normal. Thank you for the quick work on this!

justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 4, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
d501cd6 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

Resolves: SSSD#7061
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 6, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
0832cd4 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

Resolves: SSSD#7061
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 6, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
b9e4643 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

Resolves: SSSD#7061
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 7, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
e5c649d 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

:relnote: PAM passkey processing now correctly checks for and skips
handling any non-passkey data.

Resolves: SSSD#7061
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 7, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
bb0f33f 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

:relnote: Fixes a crash when PAM passkey processing incorrectly
handles non-passkey data.

Resolves: SSSD#7061
justin-stephenson added a commit to justin-stephenson/sssd that referenced this issue Dec 7, 2023
@justin-stephenson
passkey: Skip processing non-passkey mapping data
a12f902 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

:relnote: Fixes a crash when PAM passkey processing incorrectly
handles non-passkey data.

Resolves: SSSD#7061
alexey-tikhonov pushed a commit that referenced this issue Dec 12, 2023
@justin-stephenson @alexey-tikhonov
passkey: Skip processing non-passkey mapping data
4d01e11 
In the AD case, the user altSecurityIdentities attribute can
store passkey, smartcard, or ssh public key mapping data. Check
to ensure we are handling passkey data before continuing in
PAM passkey processing.

:relnote: Fixes a crash when PAM passkey processing incorrectly
handles non-passkey data.

Resolves: #7061

Reviewed-by: Iker Pedrosa <ipedrosa@redhat.com>
Reviewed-by: Sumit Bose <sbose@redhat.com>
(cherry picked from commit 6ed1eff)
@alexey-tikhonov
Copy link
Member

Pushed PR: #7066

  • master
    • 6ed1eff - passkey: Skip processing non-passkey mapping data
  • sssd-2-9
    • 4d01e11 - passkey: Skip processing non-passkey mapping data

@alexey-tikhonov alexey-tikhonov added the Closed: Fixed Issue was closed as fixed. label Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@justin-stephenson justin-stephenson

Labels
Closed: Fixed Issue was closed as fixed. passkey Issues and PRs related to 'passkey' feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

passkey: Skip processing non-passkey mapping data justin-stephenson/sssd
3 participants
@elpres @justin-stephenson @alexey-tikhonov