c2s segfaulting after SSH connection to host!?!

[scottchiefbaker]

I can reliably c2s crash by SSHing to the parent host. It's the weirdest thing I've ever seen. I have a Fedora 25 (jabberd-2.6.1-1.fc25.x86_64) host I'm trying to upgrade to Fedora 27 (jabberd-2.6.1-4.fc27.x86_64). I can start jabber and everything will run smoothly until someone initiates an SSH connection to the host. I have no idea why the two are related, but I can reproduce it.

Here is the appropriate section of syslog:

Nov 21 08:45:51 perturb jabberd/c2s[2902]: [11] [65.182.224.20, port=42454] connect
Nov 21 08:45:51 perturb audit[2902]: ANOM_ABEND auid=4294967295 uid=993 gid=990 ses=4294967295 pid=2902 comm="c2s" exe="/usr/bin/c2s" sig=11 res=1
Nov 21 08:45:51 perturb kernel: c2s[2902]: segfault at 7ffd00000327 ip 00007f882e369d7c sp 00007ffd892289f0 error 4 in libcrypto.so.1.1.0f[7f882e2bb000+25b000]
Nov 21 08:45:51 perturb audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@7-2906-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:51 perturb systemd[1]: Started Process Core Dump (PID 2906/UID 0).
Nov 21 08:45:51 perturb jabberd/router[2900]: [127.0.0.1, port=33694] disconnect
Nov 21 08:45:51 perturb jabberd/router[2900]: [c2s] offline
Nov 21 08:45:51 perturb systemd[1]: jabberd-c2s.service: Main process exited, code=dumped, status=11/SEGV
Nov 21 08:45:51 perturb systemd[1]: jabberd-c2s.service: Unit entered failed state.
Nov 21 08:45:51 perturb systemd[1]: jabberd-c2s.service: Failed with result 'core-dump'.
Nov 21 08:45:51 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jabberd-c2s comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Nov 21 08:45:51 perturb systemd[1]: Stopped Jabber Server.
Nov 21 08:45:51 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jabberd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:51 perturb jabberd/s2s[2903]: shutting down
Nov 21 08:45:51 perturb systemd[1]: Stopping Jabber IM Session Manager...
Nov 21 08:45:51 perturb jabberd/s2s[2903]: connection to router closed
Nov 21 08:45:51 perturb systemd[1]: Stopping Jabber Server To Server Connector...
Nov 21 08:45:51 perturb jabberd/router[2900]: [127.0.0.1, port=33692] disconnect
Nov 21 08:45:51 perturb jabberd/router[2900]: [s2s] default route offline
Nov 21 08:45:51 perturb jabberd/router[2900]: [s2s] offline
Nov 21 08:45:51 perturb systemd[1]: Stopped Jabber Server To Server Connector.
Nov 21 08:45:51 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jabberd-s2s comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:51 perturb jabberd/router[2900]: shutting down
Nov 21 08:45:51 perturb systemd[1]: Stopping Jabber Router XML Packet Distributior...
Nov 21 08:45:51 perturb jabberd/sm[2901]: session ended: jid=user1@jabber.my-domain.com/83ae7aa7ec867c4d82205132aa6d8d300ee0022b
Nov 21 08:45:51 perturb jabberd/sm[2901]: user unloaded jid=user1@jabber.my-domain.com
Nov 21 08:45:51 perturb jabberd/sm[2901]: session ended: jid=user2@jabber.my-domain.com/Work
Nov 21 08:45:51 perturb jabberd/sm[2901]: user unloaded jid=user2@jabber.my-domain.com
Nov 21 08:45:51 perturb systemd-coredump[2907]: Process 2902 (c2s) of user 993 dumped core.#012#012Stack trace of thread 2902:#012#0  0x00007f882e369d7c BN_set_word (libcrypto.so.1.1)#012#1  0x00007f882e36f4b0 BN_dec2bn (libcrypto.so.1.1)#012#2  0x000055ac87d884aa _sx_ssl_tmp_dh_callback (c2s)#012#3  0x00007f882e786d27 tls_construct_server_key_exchange (libssl.so.1.1)#012#4  0x00007f882e77b0be state_machine (libssl.so.1.1)#012#5  0x00007f882e773781 SSL_do_handshake (libssl.so.1.1)#012#6  0x000055ac87d87005 _sx_ssl_handshake (c2s)#012#7  0x000055ac87d87b4e _sx_ssl_rio (c2s)#012#8  0x000055ac87d8b88d _sx_chain_io_read (c2s)#012#9  0x000055ac87d81b45 sx_can_read (c2s)#012#10 0x000055ac87d7b295 _c2s_client_mio_callback (c2s)#012#11 0x000055ac87d8bd5d _mio_run (c2s)#012#12 0x000055ac87d77e04 main (c2s)#012#13 0x00007f882cddc03a __libc_start_main (libc.so.6)#012#14 0x000055ac87d789da _start (c2s)
Nov 21 08:45:51 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@7-2906-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:52 perturb jabberd/sm[2901]: session ended: jid=scott@jabber.my-domain.com/Work
Nov 21 08:45:52 perturb jabberd/sm[2901]: user unloaded jid=scott@jabber.my-domain.com
Nov 21 08:45:52 perturb jabberd/sm[2901]: shutting down
Nov 21 08:45:52 perturb jabberd/sm[2901]: connection to router closed
Nov 21 08:45:52 perturb systemd[1]: Stopped Jabber IM Session Manager.
Nov 21 08:45:52 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jabberd-sm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:52 perturb jabberd/router[2900]: [127.0.0.1, port=33690] disconnect
Nov 21 08:45:52 perturb jabberd/router[2900]: [sm] offline
Nov 21 08:45:52 perturb jabberd/router[2900]: [jabber.my-domain.com] offline
Nov 21 08:45:52 perturb systemd[1]: Stopped Jabber Router XML Packet Distributior.
Nov 21 08:45:52 perturb audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jabberd-router comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 08:45:52 perturb abrt-server[2915]: Deleting problem directory ccpp-2017-11-21-08:45:52.202321-2902 (dup of ccpp-2017-11-21-08:30:39.274462-1593)
Nov 21 08:45:53 perturb abrt-notification[2956]: Process 1593 (c2s) crashed in BN_set_word()

I'm not fully sure how to read this, but it looks like there is something wrong with BN_set_word() in libcrypto.so.1.1?

[adrianreber]

This is related to the openssl-1.1 patch which had a bug. The pull request #129 is already fixed thanks to @OlegGirko. The Fedora package has also already been updated.

[scottchiefbaker]

I opened a bug with RedHat also and they pointed me to a new testing version that includes the above mentioned patch and does fix my problem.