imapd/pop3d crashing on TLS shutdown

[citrus-it]

I recently upgraded the OS on my mail servers from a version that shipped Cyrus IMAP 3.6.1 and OpenSSL 3.0 so one that incoudes Cyrus IMAP 3.8.1 and OpenSSL 3.1.
I immediately started seeing crashes of both imapd and pop3d during TLS session shutdown.

The following analysis is from a core file created while the application was running under the illumos libumem memory allocator, with debugging turned on.

The crash occurs when attempting to free an SSL structure via SSL_free()

fffffaffffdf8a60 libcrypto.so.3`CRYPTO_free_ex_data+0x4c()
fffffaffffdf8a80 libssl.so.3`SSL_SESSION_free+0x44()
fffffaffffdf8ab0 libssl.so.3`SSL_free+0x14e()
fffffaffffdf8ae0 libcyrus_imap.so.0.0.0`tls_reset_servertls+0x2b(46fcd0)
fffffaffffdf8b60 service_main+0x438(2, a34f20, fffffaffffdfb8a0)
fffffaffffdfb830 main+0x71c(2, fffffaffffdfb888, fffffaffffdfb8a0)
fffffaffffdfb860 _start_crt+0x87()
fffffaffffdfb870 _start+0x18()

The SSL object concerned was originally allocated by SSL_new(), as can be seen by looking at the allocation stack trace:

> 46fcd0/K | ::whatis
d87010 is d87000+10, allocated from umem_alloc_8192:
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          d8aee0           d87000  10cdb04daa6b9b7                1
                           9d7028           8f3c00                0
                 libumem.so.1`umem_cache_alloc_debug+0x213
                 libumem.so.1`umem_cache_alloc+0xe3
                 libumem.so.1`umem_alloc+0x9f
                 libumem.so.1`umem_malloc+0x46
                 libcrypto.so.3`CRYPTO_zalloc+0x11
                 libssl.so.3`SSL_new+0x3b
                 libcyrus_imap.so.0.0.0`tls_start_servertls+0xaf
                 cmd_starttls+0x82
                 service_main+0x684
                 main+0x71c
                 _start_crt+0x87
                 _start+0x18

The crash occurs because SSL->session was previously freed in ssl3_send_alert():

> 46fcd0/K | ::print SSL session | ::whatis
df9b50 is df9b40+10, freed from umem_alloc_1152:
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          dfae00           df9b40  10cdb04e39240fa                1
                           9b8028           7e1c00                0
                 libumem.so.1`umem_cache_free_debug+0x16e
                 libumem.so.1`umem_cache_free+0x48
                 libumem.so.1`umem_free+0xb4
                 libumem.so.1`process_free+0x111
                 libumem.so.1`umem_malloc_free+0x1d
                 libssl.so.3`SSL_CTX_remove_session+0xee
                 libssl.so.3`ssl3_send_alert+0xdd
                 libssl.so.3`ossl_statem_fatal+0xb0
                 libssl.so.3`ssl3_read_n+0x405
                 libssl.so.3`ssl3_get_record+0xb5
                 libssl.so.3`ssl3_read_bytes+0x248
                 libssl.so.3`ssl3_read+0x60
                 libssl.so.3`SSL_read+0x29
                 libcyrus.so.0.0.0`prot_fill+0x19e
                 service_main+0x3b9

Based on this, I opened openssl/openssl#23031 against openssl, thinking that the problem was there. However, after a bit of investigation and further testing, and comments from the openssl developers over in that issue, it appears that the problem originates from a3523d4

I reverted that and have experienced no crashes for 24 hours.

Quoting from the openssl issue I created:

The SSL_SESSION objects are refcounted but the refcount is not bumped when remove_session_cb is called, so there should be no SSL_SESSION_free() call done.

[dilyanpalauzov]

I also do see such crashes, which produce coredumps. Looking in the coredumps, the problematic place is shutting down the TLS connection. So I have compiled OpenSSL and Cyrus IMAP with Address Sanitizer and wait for this to happen again. Fortunately or unfortunately this anomaly does not happen very often. Can you describe how to reproduce the problem?

[dilyanpalauzov]

I compiled cyrus-imapd 3.8-patched and OpenSSL 3.0.12 with

-g -O2 -fsanitize=address -fno-omit-frame-pointer -fno-common -fsanitize-recover=address

and then execute it with the environment variables ASAN_OPTIONS=log_to_syslog=false:log_path=/asan/cyrus-asan.log:halt_on_error=true:detect_leaks=0 LSAN_OPTIONS=verbosity=0:log_threads=0.

At some moment the file below is produced. My reading is that tls_start_servertls() calls indirectly
state_machine() from ssl/statem/statem.c:442 . The latter does allocate and deallocate the space for the session, without zeroing the respective struct member. Then tls.c:1344 calls again SSL_SESSION *session = SSL_get_session(tls_conn); if (session) SSL_CTX_remove_session(s_ctx, session);.

Right now I cannot say more, as I have no idea about TLS/OpenSSL internals. But I came independently to the conclusion mentioned at openssl/openssl#23031.

=================================================================
==2805649==ERROR: AddressSanitizer: heap-use-after-free on address 0x61900000bbd0 at pc 0x7f59072ccfc2 bp 0x7ffd28036150 sp 0x7ffd28036148
READ of size 8 at 0x61900000bbd0 thread T0
    #0 0x7f59072ccfc1 in remove_session_lock ssl/ssl_sess.c:799
    #1 0x7f59072ccf73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #2 0x7f5907bae9a3 in tls_start_servertls ../imap/tls.c:1344
    #3 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #4 0x46a993 in service_main ../imap/imapd.c:965
    #5 0x40fa0e in main ../master/service.c:644
    #6 0x7f5903ab6676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #7 0x7f5903ab6734 in __libc_start_main_impl ../csu/libc-start.c:360
    #8 0x410b20 in _start ../sysdeps/x86_64/start.S:115

0x61900000bbd0 is located 592 bytes inside of 928-byte region [0x61900000b980,0x61900000bd20)
freed by thread T0 here:
    #0 0x7f5907e7f288 in __interceptor_free ../../../../libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x7f5906c7836d in CRYPTO_free crypto/mem.c:263
    #2 0x7f5906c783bd in CRYPTO_clear_free crypto/mem.c:272
    #3 0x7f59072cd641 in SSL_SESSION_free ssl/ssl_sess.c:853
    #4 0x7f5907bad41f in remove_session_cb ../imap/tls.c:637
    #5 0x7f59072cd191 in remove_session_lock ssl/ssl_sess.c:815
    #6 0x7f59072ccf73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #7 0x7f590728c078 in ssl3_send_alert ssl/s3_msg.c:59
    #8 0x7f590733e94d in ossl_statem_send_fatal ssl/statem/statem.c:128
    #9 0x7f590733ea7d in ossl_statem_fatal ssl/statem/statem.c:145
    #10 0x7f5907300e4b in ssl3_read_n ssl/record/rec_layer_s3.c:303
    #11 0x7f590730c211 in ssl3_get_record ssl/record/ssl3_record.c:210
    #12 0x7f590730790e in ssl3_read_bytes ssl/record/rec_layer_s3.c:1350
    #13 0x7f590736b68c in tls_get_message_header ssl/statem/statem_lib.c:1172
    #14 0x7f5907340727 in read_state_machine ssl/statem/statem.c:587
    #15 0x7f590733fe2f in state_machine ssl/statem/statem.c:442
    #16 0x7f590733f21c in ossl_statem_accept ssl/statem/statem.c:270
    #17 0x7f59072b8694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #18 0x7f59072ad4ba in SSL_accept ssl/ssl_lib.c:1749
    #19 0x7f5907bae81c in tls_start_servertls ../imap/tls.c:1207
    #20 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #21 0x46a993 in service_main ../imap/imapd.c:965
    #22 0x40fa0e in main ../master/service.c:644
    #23 0x7f5903ab6676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x7f5907e805bf in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f5906c7813c in CRYPTO_malloc crypto/mem.c:196
    #2 0x7f5906c78167 in CRYPTO_zalloc crypto/mem.c:203
    #3 0x7f59072c9511 in SSL_SESSION_new ssl/ssl_sess.c:129
    #4 0x7f59072cb006 in ssl_get_new_session ssl/ssl_sess.c:434
    #5 0x7f590737bb44 in tls_early_post_process_client_hello ssl/statem/statem_srvr.c:1799
    #6 0x7f590737ec8f in tls_post_process_client_hello ssl/statem/statem_srvr.c:2189
    #7 0x7f5907378490 in ossl_statem_server_post_process_message ssl/statem/statem_srvr.c:1236
    #8 0x7f5907340fee in read_state_machine ssl/statem/statem.c:675
    #9 0x7f590733fe2f in state_machine ssl/statem/statem.c:442
    #10 0x7f590733f21c in ossl_statem_accept ssl/statem/statem.c:270
    #11 0x7f59072b8694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #12 0x7f59072ad4ba in SSL_accept ssl/ssl_lib.c:1749
    #13 0x7f5907bae81c in tls_start_servertls ../imap/tls.c:1207
    #14 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #15 0x46a993 in service_main ../imap/imapd.c:965
    #16 0x40fa0e in main ../master/service.c:644
    #17 0x7f5903ab6676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free ssl/ssl_sess.c:799 in remove_session_lock
Shadow bytes around the buggy address:
  0x61900000b900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000b980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bb00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61900000bb80: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x61900000bc00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bc80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bd00: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000bd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000be00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2805649==ABORTING

[dilyanpalauzov]

With this change

diff --git a/imap/tls.c b/imap/tls.c
index bd318b5ea..63160e09d 100644
--- a/imap/tls.c
+++ b/imap/tls.c
@@ -1213,6 +1213,7 @@ EXPORTED int tls_start_servertls(int readfd, int writefd, int timeout,

         /* Check the error code */
         err = SSL_get_error(tls_conn, sts);
+        syslog(LOG_CRIT, "SSL_accept() failed -> err=%i, sts=%i", err, sts);
         switch (err) {
         case SSL_ERROR_WANT_READ:
         case SSL_ERROR_WANT_WRITE:

Cyrus IMAP sends to syslog and ASAN logs:

• 1st case “no shared cipher in SSL_accept() -> fail”

Feb 02 01:07:14 mail cyrus/imap993[111111]: SSL_accept() failed -> err=1, sts=-1
Feb 02 01:07:14 mail cyrus/imap993[111111]: no shared cipher in SSL_accept() -> fail
Feb 02 01:08:26 mail cyrus/master[355460]: process type:SERVICE name:imap993 path:/prefix/libexec/imapd age:92.073s pid:111111 exited, status 1


=================================================================
==111111==ERROR: AddressSanitizer: heap-use-after-free on address 0x61900000bbd0 at pc 0x7f81b2f13fc2 bp 0x7ffc4392f360 sp 0x7ffc4392f358
READ of size 8 at 0x61900000bbd0 thread T0
    #0 0x7f81b2f13fc1 in remove_session_lock ssl/ssl_sess.c:799
    #1 0x7f81b2f13f73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #2 0x7f81b37f6a7b in tls_start_servertls ../imap/tls.c:1345
    #3 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #4 0x46a993 in service_main ../imap/imapd.c:965
    #5 0x40fa0e in main ../master/service.c:644
    #6 0x7f81af6fd676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #7 0x7f81af6fd734 in __libc_start_main_impl ../csu/libc-start.c:360
    #8 0x410b20 in _start ../sysdeps/x86_64/start.S:115

0x61900000bbd0 is located 592 bytes inside of 928-byte region [0x61900000b980,0x61900000bd20)
freed by thread T0 here:
    #0 0x7f81b3a95288 in __interceptor_free ../../../../libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x7f81b28bf36d in CRYPTO_free crypto/mem.c:263
    #2 0x7f81b28bf3bd in CRYPTO_clear_free crypto/mem.c:272
    #3 0x7f81b2f14641 in SSL_SESSION_free ssl/ssl_sess.c:853
    #4 0x7f81b37f541f in remove_session_cb ../imap/tls.c:637
    #5 0x7f81b2f14191 in remove_session_lock ssl/ssl_sess.c:815
    #6 0x7f81b2f13f73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #7 0x7f81b2ed3078 in ssl3_send_alert ssl/s3_msg.c:59
    #8 0x7f81b2f8594d in ossl_statem_send_fatal ssl/statem/statem.c:128
    #9 0x7f81b2f85a7d in ossl_statem_fatal ssl/statem/statem.c:145
    #10 0x7f81b2fc61da in tls_post_process_client_hello ssl/statem/statem_srvr.c:2220
    #11 0x7f81b2fbf490 in ossl_statem_server_post_process_message ssl/statem/statem_srvr.c:1236
    #12 0x7f81b2f87fee in read_state_machine ssl/statem/statem.c:675
    #13 0x7f81b2f86e2f in state_machine ssl/statem/statem.c:442
    #14 0x7f81b2f8621c in ossl_statem_accept ssl/statem/statem.c:270
    #15 0x7f81b2eff694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #16 0x7f81b2ef44ba in SSL_accept ssl/ssl_lib.c:1749
    #17 0x7f81b37f6858 in tls_start_servertls ../imap/tls.c:1207
    #18 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #19 0x46a993 in service_main ../imap/imapd.c:965
    #20 0x40fa0e in main ../master/service.c:644
    #21 0x7f81af6fd676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x7f81b3a965bf in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f81b28bf13c in CRYPTO_malloc crypto/mem.c:196
    #2 0x7f81b28bf167 in CRYPTO_zalloc crypto/mem.c:203
    #3 0x7f81b2f10511 in SSL_SESSION_new ssl/ssl_sess.c:129
    #4 0x7f81b2f12006 in ssl_get_new_session ssl/ssl_sess.c:434
    #5 0x7f81b2fc2b44 in tls_early_post_process_client_hello ssl/statem/statem_srvr.c:1799
    #6 0x7f81b2fc5c8f in tls_post_process_client_hello ssl/statem/statem_srvr.c:2189
    #7 0x7f81b2fbf490 in ossl_statem_server_post_process_message ssl/statem/statem_srvr.c:1236
    #8 0x7f81b2f87fee in read_state_machine ssl/statem/statem.c:675
    #9 0x7f81b2f86e2f in state_machine ssl/statem/statem.c:442
    #10 0x7f81b2f8621c in ossl_statem_accept ssl/statem/statem.c:270
    #11 0x7f81b2eff694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #12 0x7f81b2ef44ba in SSL_accept ssl/ssl_lib.c:1749
    #13 0x7f81b37f6858 in tls_start_servertls ../imap/tls.c:1207
    #14 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #15 0x46a993 in service_main ../imap/imapd.c:965
    #16 0x40fa0e in main ../master/service.c:644
    #17 0x7f81af6fd676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free ssl/ssl_sess.c:799 in remove_session_lock
Shadow bytes around the buggy address:
  0x61900000b900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000b980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bb00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61900000bb80: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x61900000bc00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bc80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bd00: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000bd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000be00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==111111==ABORTING

• 2nd case “unexpected eof while reading in SSL_accept() -> fail”

Feb 02 01:48:55 mail cyrus/imap993[222222]: SSL_accept() failed -> err=2, sts=-1
Feb 02 01:48:55 mail cyrus/imap993[222222]: SSL_accept() incomplete -> wait
Feb 02 01:48:55 mail cyrus/imap993[222222]: SSL_accept() failed -> err=1, sts=-1
Feb 02 01:48:55 mail cyrus/imap993[222222]: unexpected eof while reading in SSL_accept() -> fail
Feb 02 01:49:56 mail cyrus/master[355460]: process type:SERVICE name:imap993 path:/prefix/libexec/imapd age:178.797s pid:222222 exited, status 1


=================================================================
==222222==ERROR: AddressSanitizer: heap-use-after-free on address 0x61900000bbd0 at pc 0x7f0ee7a98fc2 bp 0x7fff10281700 sp 0x7fff102816f8
READ of size 8 at 0x61900000bbd0 thread T0
    #0 0x7f0ee7a98fc1 in remove_session_lock ssl/ssl_sess.c:799
    #1 0x7f0ee7a98f73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #2 0x7f0ee837ba7b in tls_start_servertls ../imap/tls.c:1345
    #3 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #4 0x46a993 in service_main ../imap/imapd.c:965
    #5 0x40fa0e in main ../master/service.c:644
    #6 0x7f0ee4282676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #7 0x7f0ee4282734 in __libc_start_main_impl ../csu/libc-start.c:360
    #8 0x410b20 in _start ../sysdeps/x86_64/start.S:115

0x61900000bbd0 is located 592 bytes inside of 928-byte region [0x61900000b980,0x61900000bd20)
freed by thread T0 here:
    #0 0x7f0ee861a288 in __interceptor_free ../../../../libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x7f0ee744436d in CRYPTO_free crypto/mem.c:263
    #2 0x7f0ee74443bd in CRYPTO_clear_free crypto/mem.c:272
    #3 0x7f0ee7a99641 in SSL_SESSION_free ssl/ssl_sess.c:853
    #4 0x7f0ee837a41f in remove_session_cb ../imap/tls.c:637
    #5 0x7f0ee7a99191 in remove_session_lock ssl/ssl_sess.c:815
    #6 0x7f0ee7a98f73 in SSL_CTX_remove_session ssl/ssl_sess.c:791
    #7 0x7f0ee7a58078 in ssl3_send_alert ssl/s3_msg.c:59
    #8 0x7f0ee7b0a94d in ossl_statem_send_fatal ssl/statem/statem.c:128
    #9 0x7f0ee7b0aa7d in ossl_statem_fatal ssl/statem/statem.c:145
    #10 0x7f0ee7acce4b in ssl3_read_n ssl/record/rec_layer_s3.c:303
    #11 0x7f0ee7ad8211 in ssl3_get_record ssl/record/ssl3_record.c:210
    #12 0x7f0ee7ad390e in ssl3_read_bytes ssl/record/rec_layer_s3.c:1350
    #13 0x7f0ee7b3768c in tls_get_message_header ssl/statem/statem_lib.c:1172
    #14 0x7f0ee7b0c727 in read_state_machine ssl/statem/statem.c:587
    #15 0x7f0ee7b0be2f in state_machine ssl/statem/statem.c:442
    #16 0x7f0ee7b0b21c in ossl_statem_accept ssl/statem/statem.c:270
    #17 0x7f0ee7a84694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #18 0x7f0ee7a794ba in SSL_accept ssl/ssl_lib.c:1749
    #19 0x7f0ee837b858 in tls_start_servertls ../imap/tls.c:1207
    #20 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #21 0x46a993 in service_main ../imap/imapd.c:965
    #22 0x40fa0e in main ../master/service.c:644
    #23 0x7f0ee4282676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x7f0ee861b5bf in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f0ee744413c in CRYPTO_malloc crypto/mem.c:196
    #2 0x7f0ee7444167 in CRYPTO_zalloc crypto/mem.c:203
    #3 0x7f0ee7a95511 in SSL_SESSION_new ssl/ssl_sess.c:129
    #4 0x7f0ee7a97006 in ssl_get_new_session ssl/ssl_sess.c:434
    #5 0x7f0ee7b47b44 in tls_early_post_process_client_hello ssl/statem/statem_srvr.c:1799
    #6 0x7f0ee7b4ac8f in tls_post_process_client_hello ssl/statem/statem_srvr.c:2189
    #7 0x7f0ee7b44490 in ossl_statem_server_post_process_message ssl/statem/statem_srvr.c:1236
    #8 0x7f0ee7b0cfee in read_state_machine ssl/statem/statem.c:675
    #9 0x7f0ee7b0be2f in state_machine ssl/statem/statem.c:442
    #10 0x7f0ee7b0b21c in ossl_statem_accept ssl/statem/statem.c:270
    #11 0x7f0ee7a84694 in SSL_do_handshake ssl/ssl_lib.c:3937
    #12 0x7f0ee7a794ba in SSL_accept ssl/ssl_lib.c:1749
    #13 0x7f0ee837b858 in tls_start_servertls ../imap/tls.c:1207
    #14 0x460887 in cmd_starttls ../imap/imapd.c:9246
    #15 0x46a993 in service_main ../imap/imapd.c:965
    #16 0x40fa0e in main ../master/service.c:644
    #17 0x7f0ee4282676 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free ssl/ssl_sess.c:799 in remove_session_lock
Shadow bytes around the buggy address:
  0x61900000b900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000b980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000ba80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bb00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61900000bb80: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x61900000bc00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bc80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61900000bd00: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000bd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61900000be00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==222222==ABORTING

[mosvald]

It looks like we are hitting the same also on Fedora with 3.8.1, it didn't happen with 3.6.0:

Bug 2256083 - Sporadic segv in imapd after upgrading to FC39

[daleghent]

FWIW I'm also seeing this on OmniOS:

[root@xenon]/local/cores#  mdb imapd.1032
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> ::status
debugging core file of imapd (64-bit) from xenon
file: /opt/ooce/cyrus/libexec/imapd
initial argv: imapd -s
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=1032 uid=89 code=-1
> ::stack
libc.so.1`_lwp_kill+0xa()
libc.so.1`raise+0x22(6)
libumem.so.1`umem_do_abort+0x44()
libumem.so.1`umem_err_recoverable+0xfe(fffffc7feee17251)
libumem.so.1`process_free+0xd4(b5b050, 1, 0)
libumem.so.1`umem_malloc_free+0x1d(b5b050)
libssl.so.3`SSL_free+0x14e()
libcyrus_imap.so.0.0.0`tls_reset_servertls+0x2b(46fcd0)
service_main+0x438(2, b2af20, fffffc7fffdf25f0)
main+0x71c(2, fffffc7fffdf25d8, fffffc7fffdf25f0)
_start_crt+0x87()
_start+0x18()