squid crash on exit

[beldmit]

When testing a Red Hat default config, squid crashes on exit when both default and legacy providers are activated.

Command:
squid -k parse -f /etc/squid/squid.conf.default

#0  0x00007ffff726d19c in pthread_rwlock_rdlock@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x00007ffff7addf3d in CRYPTO_THREAD_read_lock (lock=<optimized out>) at crypto/threads_pthread.c:85
#2  0x00007ffff7ad6126 in ossl_lib_ctx_get_data (ctx=0x7ffff7d4a540 <default_context_int.lto_priv>, index=1, meth=0x7ffff7d03a00 <provider_store_method.lto_priv>) at crypto/context.c:398
#3  0x00007ffff7ae3bea in get_provider_store (libctx=<optimized out>) at crypto/provider_core.c:334
#4  ossl_provider_deregister_child_cb (handle=0x555555e2f3b0) at crypto/provider_core.c:1752
#5  0x00007ffff7ad6f2f in ossl_provider_deinit_child (ctx=0x555555ddb2f0) at crypto/provider_child.c:279
#6  OSSL_LIB_CTX_free (ctx=0x555555ddb2f0) at crypto/context.c:283
#7  OSSL_LIB_CTX_free (ctx=0x555555ddb2f0) at crypto/context.c:276
#8  0x00007ffff6da7af6 in legacy_teardown (provctx=0x555555e2f6d0) at providers/legacyprov.c:168
#9  0x00007ffff7ae401b in ossl_provider_teardown (prov=0x555555e2f3b0) at crypto/provider_core.c:1477
#10 ossl_provider_free (prov=0x555555e2f3b0) at crypto/provider_core.c:683
#11 0x00007ffff7aae956 in ossl_provider_free (prov=<optimized out>) at crypto/provider_core.c:668
#12 evp_cipher_free_int (cipher=0x555555eb3510) at crypto/evp/evp_enc.c:1632
#13 EVP_CIPHER_free (cipher=0x555555eb3510) at crypto/evp/evp_enc.c:1647
#14 0x00007ffff7d87c1d in ssl_evp_cipher_free (cipher=0x555555eb3510) at ssl/ssl_lib.c:5925
#15 ssl_evp_cipher_free (cipher=0x555555eb3510) at ssl/ssl_lib.c:5915
#16 SSL_CTX_free (a=0x555555e35940) at ssl/ssl_lib.c:3455
#17 SSL_CTX_free (a=0x555555e35940) at ssl/ssl_lib.c:3392
#18 0x00005555559b7c34 in Security::PeerOptions::convertContextFromRawPtr<ssl_ctx_st*>(ssl_ctx_st*) const::{lambda(ssl_ctx_st*)#1}::operator()(ssl_ctx_st*) const (__closure=<optimized out>, p=0x555555e35940)
    at security/../../src/security/PeerOptions.h:114
#19 std::_Sp_counted_deleter<ssl_ctx_st*, Security::PeerOptions::convertContextFromRawPtr<ssl_ctx_st*>(ssl_ctx_st*) const::{lambda(ssl_ctx_st*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=<optimized out>)
    at /usr/include/c++/11/bits/shared_ptr_base.h:442
#20 0x000055555593ee9a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x555555ee29b0) at /usr/include/c++/11/bits/shared_ptr_base.h:168
#21 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/shared_ptr_base.h:705
#22 std::__shared_ptr<ssl_ctx_st, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/shared_ptr_base.h:1154
#23 std::shared_ptr<ssl_ctx_st>::~shared_ptr (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/shared_ptr.h:122
#24 SquidConfig::{unnamed type#27}::~SquidConfig() (this=<optimized out>, this=<optimized out>) at /usr/src/debug/squid-5.5-3.el9_1.x86_64/src/SquidConfig.h:515
#25 SquidConfig::~SquidConfig (this=<optimized out>, this=<optimized out>) at /usr/src/debug/squid-5.5-3.el9_1.x86_64/src/SquidConfig.h:73
#26 0x00007ffff721f455 in __run_exit_handlers () from /lib64/libc.so.6
#27 0x00007ffff721f5d0 in exit () from /lib64/libc.so.6
#28 0x00007ffff7207eb7 in __libc_start_call_main () from /lib64/libc.so.6
#29 0x00007ffff7207f60 in __libc_start_main_impl () from /lib64/libc.so.6
#30 0x00005555556b4f05 in _start () at mem/../../src/Debug.h:77

I believe this is an issue of vanilla OpenSSL, we didn't introduce any changes here in Red Hat.

Original bug: https://bugzilla.redhat.com/show_bug.cgi?id=2155867

[steven-bellock]

@mattcaswell Is the root cause of this bug known? I might be hitting something similar.

[terry2000s]

I am a Samsung engineer. I had a similiar experience.

In my case, this type of error happened when a ssl context is stored in a static variable. The call sequece is like below.

1. main() function ends
2. OPENSSL_cleanup() is called
3. The ssl context stored in static varailbe is destored. During this destruction, a crash happened and its call stack is very similiar with this this.

I think some legacy provider didn't expect this situation.

[nhorman]

that looks to be the case roughly from the backtrace. Given that this was reported a year ago, and theres been no update, save for @terry2000s comment above, I'm assuming this is resolved