Hangs and consuming all CPU cycles in CRYPTO_THREAD_run_once

[leiyu-bytedance]

Describe the bug
bmcweb hangs and consuming all CPU cycles.

Environment
The bmcweb revision is 19a8815

To Reproduce
We could not find stable reproduce steps, the BMC is being used for a few weeks, various operations are done via WebUI.
When the issue occurs, bmcweb does not accept new requests, and it consumes all CPU cycles.

Manually trigger a coredump, and the stack backtrace shows below:

#0  __GI___pthread_once (once_control=once_control@entry=0x4bf89230 <ex_data_init>, init_routine=0x4becc290 <do_ex_data_init_ossl_>) at pthread_once.c:140
#1  0x4bf0b284 in CRYPTO_THREAD_run_once (once=once@entry=0x4bf89230 <ex_data_init>, init=<optimized out>) at ../openssl-1.1.1h/crypto/threads_pthread.c:118
#2  0x4becc224 in get_and_lock (class_index=class_index@entry=12) at ../openssl-1.1.1h/crypto/ex_data.c:59
#3  0x4becc99c in CRYPTO_free_ex_data (class_index=class_index@entry=12, obj=obj@entry=0xa45400, ad=ad@entry=0xa45448) at ../openssl-1.1.1h/crypto/ex_data.c:329
#4  0x4be2f314 in BIO_free (a=0xa45400) at ../openssl-1.1.1h/crypto/bio/bio_lib.c:134
#5  BIO_free (a=0xa45400) at ../openssl-1.1.1h/crypto/bio/bio_lib.c:110
#6  0x00255ecc in boost::asio::ssl::detail::engine::~engine() [clone .isra.0] (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/detail/impl/engine.ipp:78
#7  0x0019b5c8 in boost::asio::ssl::detail::stream_core::~stream_core (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/detail/stream_core.hpp:90
#8  boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >::~stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/stream.hpp:135
#9  boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::~flat_stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/beast/core/flat_stream.hpp:125
#10 std::default_delete<boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > > >::operator()(boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >*) const [clone .part.0] [clone .lto_priv.0] (__ptr=0x9c8fe0, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/unique_ptr.h:85
#11 0x00180344 in boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >::~ssl_stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/beast/ssl/ssl_stream.hpp:73
#12 crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>::~Connection (this=<optimized out>, this=<optimized out>)
    at ../git/http/http_connection.hpp:273
#13 __gnu_cxx::new_allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >::destroy<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > (__p=0xa5cc48, this=0xa5cc48)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/ext/new_allocator.h:156
#14 std::allocator_traits<std::allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > >::destroy<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > (__p=0xa5cc48, __a=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/alloc_traits.h:531
#15 std::_Sp_counted_ptr_inplace<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>, std::allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0xa5cc38)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:560
#16 0x000ac170 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xa5cc38)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:158
#17 0x000a22d8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:733
#18 0x0013f644 in std::__shared_ptr<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:1183
#19 std::shared_ptr<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >::~shared_ptr (this=<optimized out>,
    this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr.h:121
#20 crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}::~error_code()
    (this=<optimized out>, this=<optimized out>) at ../git/http/http_server.hpp:192
#21 boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>::~binder1() (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/bind_handler.hpp:33
#22 boost::asio::detail::executor_function::complete<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, std::allocator<void> >(boost::asio::detail::executor_function::impl_base*, bool) (base=<optimized out>,
    call=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/executor_function.hpp:109
#23 0x00179df8 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::executor_function, boost::asio::detail::executor_function> (context=..., function=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_invoke_helpers.hpp:51
#24 boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>::dispatch<boost::asio::detail::executor_function, std::allocator<void> > (a=..., f=..., this=0x74ef14)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/io_context.hpp:350
--Type <RET> for more, q to quit, c to continue without paging--
#25 boost::asio::executor::impl<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>, std::allocator<void> >::dispatch (this=this@entry=0x74ef08, f=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/executor.hpp:94
#26 0x00123768 in boost::asio::executor::dispatch<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, std::allocator<void> >(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&&, std::allocator<void> const&) const (
    this=0x7ebbf584, this=0x7ebbf584, a=..., f=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/executor.hpp:263
#27 boost::asio::detail::handler_work_base<boost::asio::executor, void, boost::asio::io_context, boost::asio::executor, void>::dispatch<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}>(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}&) (handler=...,
    function=..., this=0x7ebbf584)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_work.hpp:275
#28 boost::asio::detail::handler_work<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::asio::executor, void>::complete<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code> >(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}&) (handler=..., function=..., this=0x7ebbf584)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_work.hpp:429
#29 boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::executor>, boost::asio::ip::tcp, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::asio::executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned int) (owner=0x7ebbf5cc, base=0x7ebbf580)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:148
#30 0x00020320 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x6b7678, this=0x8dcd98)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/scheduler_operation.hpp:40
#31 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x6b7678)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/impl/scheduler.ipp:481
#32 boost::asio::detail::scheduler::run (ec=..., this=0x6b7678)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/impl/scheduler.ipp:204
#33 boost::asio::io_context::run (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/io_context.ipp:63
#34 main () at ../git/src/webserver_main.cpp:134

[edtanous]

Please answer the bug template as it's written. Namely, you omit what system you're on.

[leiyu-bytedance]

Please answer the bug template as it's written. Namely, you omit what system you're on.

It's a downstream system so I did not put the name.
I think the bmcweb revision and the stack backtrace already gives enough information, is it?

[edtanous]

Closing the bug given that this isn't an upstream system. In open source we don't know what code or configuration changes you've made in your downstream, so we can't support you in this bug. If this reproduces on an upstream system, or you submit patches to upstream your system, feel free to reopen.

[leiyu-bytedance]

I do not think I could not open a bug based on a downstream system, because:

• It's stated clearly that the bmcweb's revision (which is in upstream) is used, and no downstream patches are applied;
• The stack backtrace is provided.

[edtanous]

bmcweb doesn't run in isolation, and is part of openbmc, which you don't include a revision for (presumably because you're operating on your fork). Considering your crash looks like it occurred somewhere in OpenSSL, it's quite possible and likely it's a system issue. If this reproduces on an upstream machine that we can inspect the configuration of, let me know, but it's too much to debug issues on everyones forks.

If you're interested in resubmitting this, please upstream your system, then if it still reproduces, we can reopen this.