Segmentation fault on node configured to use client encryption #14299

fruch · 2023-06-19T13:53:42Z

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Node-1 keep crashing since boot, with the following callstack:

2023-06-19 11:51:34.068 <2023-06-19 11:51:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=2ef4c3ed-b7e5-4b45-bcee-ad371e2f65f6: type=SEGMENTATION regex=segmentation line_number=87205 node=longevity-100gb-4h-master-db-node-54f56776-1
2023-06-19T11:51:33+00:00 longevity-100gb-4h-master-db-node-54f56776-1     !INFO | scylla[24832]: Segmentation fault on shard 1.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
(inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:811
(inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:841
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:853
(inlined by) seastar::sigsegv_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3964
(inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3945
(inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3941
?? ??:0
seastar::shared_ptr<seastar::tls::certificate_credentials>::operator->() const at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:613
(inlined by) session at ./build/release/seastar/./seastar/src/net/tls.cc:1002
session at ./build/release/seastar/./seastar/src/net/tls.cc:1048
seastar::lw_shared_ptr<seastar::tls::session> seastar::lw_shared_ptr<seastar::tls::session>::make<seastar::tls::session::type, seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::tls::session::type&&, seastar::shared_ptr<seastar::tls::certificate_credentials>&&, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:282
seastar::lw_shared_ptr<seastar::tls::session> seastar::make_lw_shared<seastar::tls::session, seastar::tls::session::type, seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::tls::session::type&&, seastar::shared_ptr<seastar::tls::certificate_credentials>&&, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:442
(inlined by) seastar::tls::wrap_client(seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>) at ./build/release/seastar/./seastar/src/net/tls.cc:1889
operator() at ./build/release/seastar/./seastar/src/net/tls.cc:1847
(inlined by) seastar::future<seastar::connected_socket> std::__invoke_impl<seastar::future<seastar::connected_socket>, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(std::__invoke_other, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
std::__invoke_result<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>::type std::__invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) std::invoke_result<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>::type std::invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
(inlined by) auto seastar::internal::future_invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1157
(inlined by) operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1458
(inlined by) void seastar::futurize<seastar::future<seastar::connected_socket> >::satisfy_with_result_of<seastar::future<seastar::connected_socket>::then_impl_nrvo<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket> >(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1991
operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1454
(inlined by) seastar::continuation<seastar::internal::promise_base_with_type<seastar::connected_socket>, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket>::then_impl_nrvo<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket> >(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&)#1}, seastar::connected_socket>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:741
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
(inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
(inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
(inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke(std::_Any_data const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
(inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:90
?? ??:0
?? ??:0

2023-06-19T09:18:58+00:00 longevity-100gb-4h-master-db-node-54f56776-1     !INFO | kernel: reactor-2[16692]: segfault at 8 ip 00000000059cc84f sp 00007fb4f85d2c90 error 4
2023-06-19T09:18:58+00:00 longevity-100gb-4h-master-db-node-54f56776-1     !INFO | kernel: Code: 41 55 41 54 53 48 83 ec 18 89 f5 48 89 fb 48 c7 07 00 00 00 00 89 77 08 48 8b 01 48 89 47 10 48 c7 01 00 00 00 00 48 8b 4a 08 <48> 8b 41 08 0f 10 41 08 0f 11 47 18 48 85 c0 74 04 48 ff 40 08 41
2023-06-19T09:18:58+00:00 longevity-100gb-4h-master-db-node-54f56776-1     !INFO | systemd[1]: Started Process Core Dump (PID 16755/UID 0).

2023-06-19 09:12:07.407 <2023-06-19 09:06:12.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=86b061eb-69c1-43a8-a7d5-aa8e9654b9c7 node=Node longevity-100gb-4h-master-db-node-54f56776-1 [44.202.192.33 | 10.12.1.107] (seed: True)
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000.gz
backtrace=           PID: 15844 (scylla)
           UID: 112 (scylla)
           GID: 118 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Mon 2023-06-19 09:06:12 UTC (3min 34s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 25 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: af19e64d0a7b4792a06ef87b91a1d73f
    Machine ID: c45dd525146e4ed58d6b616ce0a3e46c
      Hostname: longevity-100gb-4h-master-db-node-54f56776-1
       Storage: /var/lib/systemd/coredump/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000 (present)
     Disk Size: 110.8G
       Message: Process 15844 (scylla) of user 112 dumped core.
                
                Stack trace of thread 15847:
                #0  0x00000000059cc84f _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEESt10unique_ptrINS_3net21connected_socket_implESt14default_deleteIS8_EENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc84f)
                #1  0x00000000059cc775 _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc775)
                #2  0x00000000059cc682 _ZN7seastar13lw_shared_ptrINS_3tls7sessionEE4makeIJNS2_4typeENS_10shared_ptrINS1_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEEEEES3_DpOT_ (scylla + 0x57cc682)
                #3  0x00000000059ab02c _ZN7seastar3tls11wrap_clientENS_10shared_ptrINS0_23certificate_credentialsEEEONS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57ab02c)
                #4  0x00000000059cc596 _ZSt13__invoke_implIN7seastar6futureINS0_16connected_socketEEERZNS0_3tls15tls_socket_impl7connectENS0_14socket_addressES6_NS0_9transportEEUlS2_E_JS2_EET_St14__invoke_otherOT0_DpOT1_ (scylla + 0x57cc596)
                #5  0x00000000059cc4d6 _ZN7seastar8futurizeINS_6futureINS_16connected_socketEEEE22satisfy_with_result_ofIZZNS3_14then_impl_nrvoIZNS_3tls15tls_socket_impl7connectENS_14socket_addressES9_NS_9transportEEUlS2_E_S3_EET0_OT_ENKUlONS_8internal22promise_base_with_typeIS2_EERSB_ONS_12future_stateIS2_EEE_clESI_SJ_SM_EUlvE_EEvSI_SE_ (scylla + 0x57cc4d6)
                #6  0x00000000059cc3f2 _ZN7seastar12continuationINS_8internal22promise_base_with_typeINS_16connected_socketEEEZNS_3tls15tls_socket_impl7connectENS_14socket_addressES7_NS_9transportEEUlS3_E_ZNS_6futureIS3_E14then_impl_nrvoIS9_SB_EET0_OT_EUlOS4_RS9_ONS_12future_stateIS3_EEE_S3_E15run_and_disposeEv (scylla + 0x57cc3f2)
                #7  0x000000000563af15 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x543af15)
                #8  0x000000000563c198 _ZN7seastar7reactor6do_runEv (scylla + 0x543c198)
                #9  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #10 0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #11 0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #12 0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15864:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15858:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15862:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15859:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15863:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15861:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15870:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15844:
                #0  0x00007f5bd9ff82bf __write (libc.so.6 + 0xfb2bf)
                #1  0x000000000566509c _ZN7seastar9file_desc5writeEPKvm (scylla + 0x546509c)
                #2  0x000000000565dca1 _ZZN7seastar18syscall_work_queue11submit_itemESt10unique_ptrINS0_9work_itemESt14default_deleteIS2_EEEN4$_55clENS_6futureIvEE (scylla + 0x545dca1)
                #3  0x000000000563d77f _ZN7seastar18syscall_work_queue11submit_itemESt10unique_ptrINS0_9work_itemESt14default_deleteIS2_EE (scylla + 0x543d77f)
                #4  0x00000000055db0c3 _ZN7seastar18syscall_work_queue6submitINS_14syscall_resultIiEEEENS_6futureIT_EENS_20noncopyable_functionIFS5_vEEE (scylla + 0x53db0c3)
                #5  0x0000000005630403 _ZN7seastar7reactor14open_directoryESt17basic_string_viewIcSt11char_traitsIcEE (scylla + 0x5430403)
                #6  0x00000000055d196a _ZN7seastar14open_directoryESt17basic_string_viewIcSt11char_traitsIcEE (scylla + 0x53d196a)
                #7  0x0000000005a082f6 _ZN7seastar14sync_directoryESt17basic_string_viewIcSt11char_traitsIcEE (scylla + 0x58082f6)
                #8  0x000000000355839d _ZN2db9commitlog15segment_manager10named_file6renameESt17basic_string_viewIcSt11char_traitsIcEE.resume (scylla + 0x335839d)
                #9  0x0000000001295a6b _ZN7seastar8internal21coroutine_traits_baseIvE12promise_type15run_and_disposeEv (scylla + 0x1095a6b)
                #10 0x000000000563af15 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x543af15)
                #11 0x000000000563c198 _ZN7seastar7reactor6do_runEv (scylla + 0x543c198)
                #12 0x000000000563b4da _ZN7seastar7reactor3runEv (scylla + 0x543b4da)
                #13 0x00000000055df3f6 _ZN7seastar12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla + 0x53df3f6)
                #14 0x00000000055de569 _ZN7seastar12app_template3runEiPPcOSt8functionIFNS_6futureIiEEvEE (scylla + 0x53de569)
                #15 0x000000000121d554 _ZL11scylla_mainiPPc (scylla + 0x101d554)
                #16 0x000000000121f0e1 _ZNKSt8functionIFiiPPcEEclEiS1_ (scylla + 0x101f0e1)
                #17 0x000000000121b978 main (scylla + 0x101b978)
                #18 0x00007f5bd9f24510 __libc_start_call_main (libc.so.6 + 0x27510)
                #19 0x00007f5bd9f245c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x275c9)
                #20 0x0000000001219525 _start (scylla + 0x1019525)
                
                Stack trace of thread 15865:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15868:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15871:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15867:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15846:
                #0  0x00000000059cc84f _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEESt10unique_ptrINS_3net21connected_socket_implESt14default_deleteIS8_EENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc84f)
                #1  0x00000000059cc775 _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc775)
                #2  0x00000000059cc682 _ZN7seastar13lw_shared_ptrINS_3tls7sessionEE4makeIJNS2_4typeENS_10shared_ptrINS1_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEEEEES3_DpOT_ (scylla + 0x57cc682)
                #3  0x00000000059ab02c _ZN7seastar3tls11wrap_clientENS_10shared_ptrINS0_23certificate_credentialsEEEONS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57ab02c)
                #4  0x00000000059cc596 _ZSt13__invoke_implIN7seastar6futureINS0_16connected_socketEEERZNS0_3tls15tls_socket_impl7connectENS0_14socket_addressES6_NS0_9transportEEUlS2_E_JS2_EET_St14__invoke_otherOT0_DpOT1_ (scylla + 0x57cc596)
                #5  0x00000000059cc4d6 _ZN7seastar8futurizeINS_6futureINS_16connected_socketEEEE22satisfy_with_result_ofIZZNS3_14then_impl_nrvoIZNS_3tls15tls_socket_impl7connectENS_14socket_addressES9_NS_9transportEEUlS2_E_S3_EET0_OT_ENKUlONS_8internal22promise_base_with_typeIS2_EERSB_ONS_12future_stateIS2_EEE_clESI_SJ_SM_EUlvE_EEvSI_SE_ (scylla + 0x57cc4d6)
                #6  0x00000000059cc3f2 _ZN7seastar12continuationINS_8internal22promise_base_with_typeINS_16connected_socketEEEZNS_3tls15tls_socket_impl7connectENS_14socket_addressES7_NS_9transportEEUlS3_E_ZNS_6futureIS3_E14then_impl_nrvoIS9_SB_EET0_OT_EUlOS4_RS9_ONS_12future_stateIS3_EEE_S3_E15run_and_disposeEv (scylla + 0x57cc3f2)
                #7  0x000000000563af15 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x543af15)
                #8  0x000000000563c198 _ZN7seastar7reactor6do_runEv (scylla + 0x543c198)
                #9  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #10 0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #11 0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #12 0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15851:
                #0  0x000000000563e790 _ZN7seastar17smp_message_queue19process_completionsEj (scylla + 0x543e790)
                #1  0x0000000005647ced _ZN7seastar3smp11poll_queuesEv (scylla + 0x5447ced)
                #2  0x000000000567b62b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x547b62b)
                #3  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #4  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #5  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #6  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #7  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #8  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15869:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15845:
                #0  0x00000000059cc84f _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEESt10unique_ptrINS_3net21connected_socket_implESt14default_deleteIS8_EENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc84f)
                #1  0x00000000059cc775 _ZN7seastar3tls7sessionC2ENS1_4typeENS_10shared_ptrINS0_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57cc775)
                #2  0x00000000059cc682 _ZN7seastar13lw_shared_ptrINS_3tls7sessionEE4makeIJNS2_4typeENS_10shared_ptrINS1_23certificate_credentialsEEENS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEEEEES3_DpOT_ (scylla + 0x57cc682)
                #3  0x00000000059ab02c _ZN7seastar3tls11wrap_clientENS_10shared_ptrINS0_23certificate_credentialsEEEONS_16connected_socketENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x57ab02c)
                #4  0x00000000059cc596 _ZSt13__invoke_implIN7seastar6futureINS0_16connected_socketEEERZNS0_3tls15tls_socket_impl7connectENS0_14socket_addressES6_NS0_9transportEEUlS2_E_JS2_EET_St14__invoke_otherOT0_DpOT1_ (scylla + 0x57cc596)
                #5  0x00000000059cc4d6 _ZN7seastar8futurizeINS_6futureINS_16connected_socketEEEE22satisfy_with_result_ofIZZNS3_14then_impl_nrvoIZNS_3tls15tls_socket_impl7connectENS_14socket_addressES9_NS_9transportEEUlS2_E_S3_EET0_OT_ENKUlONS_8internal22promise_base_with_typeIS2_EERSB_ONS_12future_stateIS2_EEE_clESI_SJ_SM_EUlvE_EEvSI_SE_ (scylla + 0x57cc4d6)
                #6  0x00000000059cc3f2 _ZN7seastar12continuationINS_8internal22promise_base_with_typeINS_16connected_socketEEEZNS_3tls15tls_socket_impl7connectENS_14socket_addressES7_NS_9transportEEUlS3_E_ZNS_6futureIS3_E14then_impl_nrvoIS9_SB_EET0_OT_EUlOS4_RS9_ONS_12future_stateIS3_EEE_S3_E15run_and_disposeEv (scylla + 0x57cc3f2)
                #7  0x000000000563af15 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x543af15)
                #8  0x000000000563c198 _ZN7seastar7reactor6do_runEv (scylla + 0x543c198)
                #9  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #10 0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #11 0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #12 0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15848:
                #0  0x00007f5bda1d6be9 _ZNSt13random_device7_M_initERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (libstdc++.so.6 + 0xdabe9)
                #1  0x00000000026e8ff9 _ZNSt13random_deviceC2Ev (scylla + 0x24e8ff9)
                #2  0x00000000059946af _ZN7seastar3net17posix_socket_impl21find_port_and_connectENS_14socket_addressES2_NS_9transportE (scylla + 0x57946af)
                #3  0x00000000059941c7 _ZN7seastar3net17posix_socket_impl7connectENS_14socket_addressES2_NS_9transportE (scylla + 0x57941c7)
                #4  0x0000000005998096 _ZN7seastar6socket7connectENS_14socket_addressES1_NS_9transportE (scylla + 0x5798096)
                #5  0x00000000059cbfc6 _ZN7seastar3tls15tls_socket_impl7connectENS_14socket_addressES2_NS_9transportE (scylla + 0x57cbfc6)
                #6  0x0000000005998096 _ZN7seastar6socket7connectENS_14socket_addressES1_NS_9transportE (scylla + 0x5798096)
                #7  0x00000000059dec0a _ZN7seastar3rpc6clientC2ERKNS0_6loggerEPvNS0_14client_optionsENS_6socketERKNS_14socket_addressESA_ (scylla + 0x57dec0a)
                #8  0x00000000013de4d8 _ZN4netw17messaging_service14get_rpc_clientENS_14messaging_verbENS_8msg_addrE (scylla + 0x11de4d8)
                #9  0x00000000013b671b _ZN4netw24send_message_cancellableIN7seastar6futureIN7service20direct_fd_ping_replyEEEJN5utils11tagged_uuidIN4raft13server_id_tagEEEEEEDaPNS_17messaging_serviceENS_14messaging_verbENS_8msg_addrERNS1_12abort_sourceEDpOT0_ (scylla + 0x11b671b)
                #10 0x00000000013b657a _ZN3ser14raft_rpc_verbs19send_direct_fd_pingEPN4netw17messaging_serviceENS1_8msg_addrERN7seastar12abort_sourceEN5utils11tagged_uuidIN4raft13server_id_tagEEE (scylla + 0x11b657a)
                #11 0x0000000004343653 _ZN7service16direct_fd_pinger4pingEN5utils4UUIDERN7seastar12abort_sourceE (scylla + 0x4143653)
                #12 0x00000000043fb2d9 _ZN23direct_failure_detectorL17ping_with_timeoutEN5utils4UUIDElRN7seastar12abort_sourceERNS_6pingerERNS_5clockE (scylla + 0x41fb2d9)
                #13 0x00000000043f769a _ZN23direct_failure_detector15endpoint_worker10ping_fiberEv (scylla + 0x41f769a)
                #14 0x00000000043f461f _ZN23direct_failure_detector16failure_detector4impl13create_workerEN5utils4UUIDE (scylla + 0x41f461f)
                #15 0x00000000043fe774 _ZN7seastar17smp_message_queue15async_work_itemIZNS_7shardedIN23direct_failure_detector16failure_detectorEE9invoke_onIZNS4_4impl12add_endpointEN5utils4UUIDEE3$_1JENS_6futureIvEEEET1_jNS_21smp_submit_to_optionsEOT_DpOT0_EUlvE_E15run_and_disposeEv (scylla + 0x41fe774)
                #16 0x000000000563af15 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x543af15)
                #17 0x000000000563c198 _ZN7seastar7reactor6do_runEv (scylla + 0x543c198)
                #18 0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #19 0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #20 0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #21 0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15850:
                #0  0x0000000005685e31 _ZN7seastar8internal13io_pgeteventsEmllPNS0_9linux_abi8io_eventEPK8timespecPK10__sigset_tb (scylla + 0x5485e31)
                #1  0x0000000005681a65 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla + 0x5481a65)
                #2  0x0000000005681eaf _ZN7seastar19reactor_backend_aio23reap_kernel_completionsEv (scylla + 0x5481eaf)
                #3  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #4  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #5  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #6  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #7  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #8  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15849:
                #0  0x0000000005681056 _ZN7seastar19aio_storage_context16reap_completionsEb (scylla + 0x5481056)
                #1  0x0000000005681ec2 _ZN7seastar19reactor_backend_aio23reap_kernel_completionsEv (scylla + 0x5481ec2)
                #2  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #3  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #4  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #5  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #6  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #7  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15856:
                #0  0x00000000055f7901 _ZN7seastar6memory4freeEPv (scylla + 0x53f7901)
                #1  0x00000000055f9d8b free (scylla + 0x53f9d8b)
                #2  0x0000000002326e41 _ZN8sstables2mx27data_consume_rows_context_mINS0_17mp_row_consumer_mEE16do_process_stateEv.resume (scylla + 0x2126e41)
                #3  0x0000000002304217 _ZN13data_consumer24continuous_data_consumerIN8sstables2mx27data_consume_rows_context_mINS2_17mp_row_consumer_mEEEEclEN7seastar16temporary_bufferIcEE (scylla + 0x2104217)
                #4  0x0000000002303717 _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables2mx27data_consume_rows_context_mINS5_17mp_row_consumer_mEEEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x2103717)
                #5  0x0000000002308523 _ZZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEvENKUlvE0_clEv (scylla + 0x2108523)
                #6  0x00000000022d7f1f _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x20d7f1f)
                #7  0x00000000022da17d _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x20da17d)
                #8  0x0000000001f237cb _ZN22mutation_reader_mergerclEv (scylla + 0x1d237cb)
                #9  0x0000000001f2fb55 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x1d2fb55)
                #10 0x0000000001f2e743 _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x1d2e743)
                #11 0x00000000023ff12e _ZN23flat_mutation_reader_v24impl17consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerENS_9no_filterEEEDaT_T0_ (scylla + 0x21ff12e)
                #12 0x00000000023fe43b _ZN23flat_mutation_reader_v217consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerENS_9no_filterEEEDaT_T0_ (scylla + 0x21fe43b)
                #13 0x00000000023f48cb _ZN23flat_mutation_reader_v217consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEDaT_ (scylla + 0x21f48cb)
                #14 0x00000000023f01f1 _ZZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES1_ENUlvE_clEv (scylla + 0x21f01f1)
                #15 0x00000000023eeb06 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x21eeb06)
                #16 0x00000000058e88f7 _ZN7seastar14thread_context4mainEv (scylla + 0x56e88f7)
                
                Stack trace of thread 15857:
                #0  0x00007f5bda00192d syscall (libc.so.6 + 0x10492d)
                #1  0x0000000005680bf1 _ZN7seastar19aio_storage_context11submit_workEv (scylla + 0x5480bf1)
                #2  0x0000000005681f9f _ZN7seastar19reactor_backend_aio18kernel_submit_workEv (scylla + 0x5481f9f)
                #3  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #4  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #5  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #6  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #7  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #8  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15853:
                #0  0x000000000563e978 _ZN7seastar17smp_message_queue16process_incomingEv (scylla + 0x543e978)
                #1  0x0000000005647d5e _ZN7seastar3smp11poll_queuesEv (scylla + 0x5447d5e)
                #2  0x000000000567b62b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x547b62b)
                #3  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #4  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #5  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #6  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #7  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #8  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15852:
                #0  0x000000000563c1a0 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1a0)
                #1  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #2  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #3  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #4  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15866:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15855:
                #0  0x000000000567b6f8 _ZN7seastar7reactor30reap_kernel_completions_pollfn4pollEv (scylla + 0x547b6f8)
                #1  0x000000000565da79 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_52E9_M_invokeERKSt9_Any_data (scylla + 0x545da79)
                #2  0x000000000563c1e2 _ZN7seastar7reactor6do_runEv (scylla + 0x543c1e2)
                #3  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #4  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #5  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #6  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15860:
                #0  0x00007f5bd9ff821c read (libc.so.6 + 0xfb21c)
                #1  0x000000000567fe75 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x547fe75)
                #2  0x0000000005680182 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5480182)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)
                
                Stack trace of thread 15854:
                #0  0x000000000565db78 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_53E10_M_managerERSt9_Any_dataRKS5_St18_Manager_operation (scylla + 0x545db78)
                #1  0x000000000563c259 _ZN7seastar7reactor6do_runEv (scylla + 0x543c259)
                #2  0x000000000565e962 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE4$_96E9_M_invokeERKSt9_Any_data (scylla + 0x545e962)
                #3  0x000000000560b72b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x540b72b)
                #4  0x00007f5bd9f8812d start_thread (libc.so.6 + 0x8b12d)
                #5  0x00007f5bda009bc0 __clone3 (libc.so.6 + 0x10cbc0)

download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.112.af19e64d0a7b4792a06ef87b91a1d73f.15844.1687165572000000.gz

Impact

The node is becoming unavailable at unknown times, and interfere with the test logic, failing all kind of nemesis

How frequently does it reproduce?

So far it has been seen once

Installation details

Kernel Version: 5.15.0-1038-aws
Scylla version (or git commit hash): 5.4.0~dev-20230618.b7627085cb13 with build-id a2d9adc050ce01f3543f876ea72d863b1ca6e615

Cluster size: 6 nodes (i3.4xlarge)

Scylla Nodes used in this run:

longevity-100gb-4h-master-db-node-54f56776-6 (3.216.9.196 | 10.12.2.39) (shards: 14)
longevity-100gb-4h-master-db-node-54f56776-5 (44.200.35.201 | 10.12.0.219) (shards: 14)
longevity-100gb-4h-master-db-node-54f56776-4 (34.205.155.188 | 10.12.1.142) (shards: 14)
longevity-100gb-4h-master-db-node-54f56776-3 (100.24.107.176 | 10.12.3.58) (shards: 14)
longevity-100gb-4h-master-db-node-54f56776-2 (44.192.5.134 | 10.12.3.155) (shards: 14)
longevity-100gb-4h-master-db-node-54f56776-1 (44.202.192.33 | 10.12.1.107) (shards: 14)

OS / Image: `` (aws: undefined_region)

Test: longevity-100gb-4h-test
Test id: 54f56776-c13f-4749-9b6c-913dc109eb46
Test name: scylla-master/longevity/longevity-100gb-4h-test
Test config file(s):

longevity-100gb-4h.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 54f56776-c13f-4749-9b6c-913dc109eb46
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 54f56776-c13f-4749-9b6c-913dc109eb46

Logs:

db-cluster-54f56776.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/db-cluster-54f56776.tar.gz
sct-runner-events-54f56776.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/sct-runner-events-54f56776.tar.gz
sct-54f56776.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/sct-54f56776.log.tar.gz
monitor-set-54f56776.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/monitor-set-54f56776.tar.gz
loader-set-54f56776.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/loader-set-54f56776.tar.gz
parallel-timelines-report-54f56776.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/54f56776-c13f-4749-9b6c-913dc109eb46/20230619_120409/parallel-timelines-report-54f56776.tar.gz

Jenkins job URL
Argus

The text was updated successfully, but these errors were encountered:

mykaul · 2023-06-20T08:21:19Z

And we don't know if it's a regression?

fruch · 2023-06-20T16:38:59Z

And we don't know if it's a regression?

My bad, it's a regression for sure.
I'm not sure if it is easily reproduced.

I do suspect it might be related to recent build changes in seastar
scylladb/seastar@aa1c8b3

But it merged in seastar 27d ago, I don't know when it was introduced into scylla

So I'm not quite sure...

elcallio · 2023-06-21T12:43:16Z

Well, the stack trace suggest we get an empty shared_ptr, and crash on first dereference. It is not 100% clear to me who the topmost caller is though...
What test (?) was this running, and how reproducible is it?

fruch · 2023-06-22T05:56:08Z

happened again:

during a nemesis that restarts scylla, and it keeps on coredumping when scylla starts again.
one thing to note both server encryption and client encryption are enabled

2023-06-21 13:53:33.683 <2023-06-21 13:53:33.000>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=1c050ab7-610c-4e09-8184-b624b23bc21d: type=SEGMENTATION regex=segmentation line_number=15448 node=longevity-100gb-4h-master-db-node-0ed14ec3-2
2023-06-21T13:53:33+00:00 longevity-100gb-4h-master-db-node-0ed14ec3-2     !INFO | scylla[12198]: Segmentation fault on shard 0.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
(inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:811
(inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:841
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:853
(inlined by) seastar::sigsegv_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3964
(inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3945
(inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3941
?? ??:0
seastar::shared_ptr<seastar::tls::certificate_credentials>::operator->() const at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:613
(inlined by) session at ./build/release/seastar/./seastar/src/net/tls.cc:1002
session at ./build/release/seastar/./seastar/src/net/tls.cc:1048
seastar::lw_shared_ptr<seastar::tls::session> seastar::lw_shared_ptr<seastar::tls::session>::make<seastar::tls::session::type, seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::tls::session::type&&, seastar::shared_ptr<seastar::tls::certificate_credentials>&&, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:282
seastar::lw_shared_ptr<seastar::tls::session> seastar::make_lw_shared<seastar::tls::session, seastar::tls::session::type, seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::tls::session::type&&, seastar::shared_ptr<seastar::tls::certificate_credentials>&&, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ./build/release/seastar/./seastar/include/seastar/core/shared_ptr.hh:442
(inlined by) seastar::tls::wrap_client(seastar::shared_ptr<seastar::tls::certificate_credentials>, seastar::connected_socket&&, seastar::basic_sstring<char, unsigned int, 15u, true>) at ./build/release/seastar/./seastar/src/net/tls.cc:1889
operator() at ./build/release/seastar/./seastar/src/net/tls.cc:1847
(inlined by) seastar::future<seastar::connected_socket> std::__invoke_impl<seastar::future<seastar::connected_socket>, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(std::__invoke_other, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
std::__invoke_result<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>::type std::__invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) std::invoke_result<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>::type std::invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
(inlined by) auto seastar::internal::future_invoke<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket>(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::connected_socket&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1157
(inlined by) operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1458
(inlined by) void seastar::futurize<seastar::future<seastar::connected_socket> >::satisfy_with_result_of<seastar::future<seastar::connected_socket>::then_impl_nrvo<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket> >(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1991
operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1454
(inlined by) seastar::continuation<seastar::internal::promise_base_with_type<seastar::connected_socket>, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket>::then_impl_nrvo<seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}, seastar::future<seastar::connected_socket> >(seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::connected_socket>&&, seastar::tls::tls_socket_impl::connect(seastar::socket_address, seastar::socket_address, seastar::transport)::{lambda(seastar::connected_socket)#1}&, seastar::future_state<seastar::connected_socket>&&)#1}, seastar::connected_socket>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:741
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
(inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:167
scylla_main(int, char**) at ./main.cc:627
std::function<int (int, char**)>::operator()(int, char**) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
main at ./main.cc:1994
?? ??:0
?? ??:0
_start at ??:?

alternator_enforce_authorization: false
api_address: 127.0.0.1
api_doc_dir: /opt/scylladb/api/api-doc/
api_port: 10000
api_ui_dir: /opt/scylladb/swagger-ui/dist/
auto_bootstrap: true
batch_size_fail_threshold_in_kb: 1024
batch_size_warn_threshold_in_kb: 128
broadcast_rpc_address: 10.12.3.7
cas_contention_timeout_in_ms: 1000
client_encryption_options:
  certificate: /etc/scylla/ssl_conf/client/test.crt
  enabled: true
  keyfile: /etc/scylla/ssl_conf/client/test.key
  truststore: /etc/scylla/ssl_conf/client/catest.pem
cluster_name: longevity-100gb-4h-master-db-cluster-0ed14ec3
commitlog_segment_size_in_mb: 32
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_total_space_in_mb: -1
consistent_cluster_management: true
enable_ipv6_dns_lookup: false
endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch
experimental: true
force_schema_commit_log: true
hinted_handoff_enabled: true
listen_address: 10.12.3.7
murmur3_partitioner_ignore_msb_bits: 12
native_shard_aware_transport_port: 19042
native_transport_port: 9042
num_tokens: 256
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
prometheus_address: 0.0.0.0
read_request_timeout_in_ms: 5000
rpc_address: 10.12.3.7
rpc_port: 9160
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
  parameters:
  - seeds: 10.12.2.11,10.12.3.7,10.12.3.242,10.12.3.179,10.12.3.253,10.12.1.14
server_encryption_options:
  certificate: /etc/scylla/ssl_conf/db.crt
  internode_encryption: all
  keyfile: /etc/scylla/ssl_conf/db.key
  truststore: /etc/scylla/ssl_conf/cadb.pem
strict_is_not_null_in_views: true
write_request_timeout_in_ms: 2000

Installation details

Kernel Version: 5.15.0-1038-aws
Scylla version (or git commit hash): 5.4.0~dev-20230621.643e69af891c with build-id 5cdc13bb2d3d575b59161e5972318890bb7c74a7

Cluster size: 6 nodes (i3.4xlarge)

Scylla Nodes used in this run:

longevity-100gb-4h-master-db-node-0ed14ec3-6 (3.216.28.223 | 10.12.1.14) (shards: 14)
longevity-100gb-4h-master-db-node-0ed14ec3-5 (3.224.135.239 | 10.12.3.253) (shards: 14)
longevity-100gb-4h-master-db-node-0ed14ec3-4 (18.209.240.210 | 10.12.3.179) (shards: 14)
longevity-100gb-4h-master-db-node-0ed14ec3-3 (3.239.203.92 | 10.12.3.242) (shards: 14)
longevity-100gb-4h-master-db-node-0ed14ec3-2 (44.213.119.239 | 10.12.3.7) (shards: 14)
longevity-100gb-4h-master-db-node-0ed14ec3-1 (52.205.234.107 | 10.12.2.11) (shards: 14)

OS / Image: `` (aws: undefined_region)

Test: longevity-100gb-4h-test
Test id: 0ed14ec3-878d-4c15-9e53-12efc5433574
Test name: scylla-master/longevity/longevity-100gb-4h-test
Test config file(s):

longevity-100gb-4h.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 0ed14ec3-878d-4c15-9e53-12efc5433574
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 0ed14ec3-878d-4c15-9e53-12efc5433574

Logs:

db-cluster-0ed14ec3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/db-cluster-0ed14ec3.tar.gz
sct-runner-events-0ed14ec3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/sct-runner-events-0ed14ec3.tar.gz
sct-0ed14ec3.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/sct-0ed14ec3.log.tar.gz
monitor-set-0ed14ec3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/monitor-set-0ed14ec3.tar.gz
loader-set-0ed14ec3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/loader-set-0ed14ec3.tar.gz
parallel-timelines-report-0ed14ec3.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/0ed14ec3-878d-4c15-9e53-12efc5433574/20230621_144415/parallel-timelines-report-0ed14ec3.tar.gz

Jenkins job URL
Argus

fruch · 2023-06-22T06:00:00Z

Well, the stack trace suggest we get an empty shared_ptr, and crash on first dereference. It is not 100% clear to me who the topmost caller is though... What test (?) was this running, and how reproducible is it?

seems like it's quite easily reproduced, and been spotted also on enterprise

DoronArazii · 2023-06-25T10:42:03Z

@elcallio setting this crash as P1 (regression) to hopefully deal with it quickly on master.

/Cc @eliransin

fruch · 2023-06-25T14:48:17Z

Happened on the 50gb-3d weekly run, and it was in a loop of crashing like that for 2 days...

Installation details

Kernel Version: 5.15.0-1038-aws
Scylla version (or git commit hash): 5.4.0~dev-20230622.c65abb06cdf9 with build-id 1314024b6d44e996cff71aba00b324edc22f6e91

Cluster size: 6 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

longevity-tls-50gb-3d-master-db-node-72f3640d-6 (54.227.22.173 | 10.12.10.70) (shards: 14)
longevity-tls-50gb-3d-master-db-node-72f3640d-5 (54.165.199.83 | 10.12.9.86) (shards: 14)
longevity-tls-50gb-3d-master-db-node-72f3640d-4 (54.91.34.101 | 10.12.8.167) (shards: 14)
longevity-tls-50gb-3d-master-db-node-72f3640d-3 (18.234.170.163 | 10.12.10.238) (shards: 14)
longevity-tls-50gb-3d-master-db-node-72f3640d-2 (54.163.62.85 | 10.12.8.105) (shards: 14)
longevity-tls-50gb-3d-master-db-node-72f3640d-1 (18.208.245.110 | 10.12.9.106) (shards: 14)

OS / Image: `` (aws: undefined_region)

Test: longevity-50gb-3days-test
Test id: 72f3640d-61ff-44d6-8f26-d4e9b0bfe07a
Test name: scylla-master/longevity/longevity-50gb-3days-test
Test config file(s):

longevity-50GB-3days-authorization-and-tls-ssl.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 72f3640d-61ff-44d6-8f26-d4e9b0bfe07a
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 72f3640d-61ff-44d6-8f26-d4e9b0bfe07a

Logs:

db-cluster-72f3640d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/db-cluster-72f3640d.tar.gz
output-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/output-72f3640d.log.tar.gz
debug-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/debug-72f3640d.log.tar.gz
events-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/events-72f3640d.log.tar.gz
normal-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/normal-72f3640d.log.tar.gz
argus-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/argus-72f3640d.log.tar.gz
raw_events-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/raw_events-72f3640d.log.tar.gz
critical-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/critical-72f3640d.log.tar.gz
warning-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/warning-72f3640d.log.tar.gz
summary-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/summary-72f3640d.log.tar.gz
error-72f3640d.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/error-72f3640d.log.tar.gz
2023_06_23__03_08_58_423.sct-72f3640d.log.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/2023_06_23__03_08_58_423.sct-72f3640d.log.gz
2023_06_24__04_05_50_851.sct-72f3640d.log.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/2023_06_24__04_05_50_851.sct-72f3640d.log.gz
monitor-set-72f3640d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/monitor-set-72f3640d.tar.gz
loader-set-72f3640d.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/72f3640d-61ff-44d6-8f26-d4e9b0bfe07a/20230625_092530/loader-set-72f3640d.tar.gz

Jenkins job URL
Argus

mykaul · 2023-06-29T11:23:38Z

Suspect f86dd85

fruch · 2023-06-29T13:32:01Z

Suspect f86dd85

A bit scary that no unit test or dtest caught this.

I'll look into writing a dtest reproducer for this one...

fruch · 2023-06-29T16:24:42Z

Suspect f86dd85

A bit scary that no unit test or dtest caught this.

I'll look into writing a dtest reproducer for this one...

got a dtest reproducer for it:
https://github.com/scylladb/scylla-dtest/pull/3280

seem like it's a combination of raft enabled and server encryption
and that we have zero tests in dtest for server encryption

eliransin · 2023-07-02T11:21:30Z

@elcallio this issue fails our tests that uses tls and it seams like a regression introduced by certificate base authorization.
Would you give this priority please?

This reverts commit f86dd85, reversing changes made to 3ca0c6c. It breaks setups that use client encryption. Fixes #14299.

elcallio · 2023-07-03T11:42:01Z

How would f86dd85 be even related? That code changes literally nothing with setting up TLS connectors, nor underlying TLS infrastructure. It only looks at data once incoming client connections are up. The above trace is literally in socket creation.

mykaul · 2023-07-03T11:46:14Z

How would f86dd85 be even related? That code changes literally nothing with setting up TLS connectors, nor underlying TLS infrastructure. It only looks at data once incoming client connections are up. The above trace is literally in socket creation.

@elcallio - any other ideas of what we need to revert?

elcallio · 2023-07-03T12:08:34Z

bisect? Assuming @fruch :s repro is sound?

avikivity · 2023-07-03T12:11:07Z

It can't be that commit because the report is older than the merge. I'll dequeue the revert.

elcallio · 2023-07-03T12:12:08Z

I love temporal evidence. I'll run @fruch :s repro and see if I can make sense of the crash.

mykaul · 2023-07-03T12:15:54Z

@elcallio - scylladb/seastar@f461641 perhaps?
(but yes, git bisect is best)

avikivity · 2023-07-03T12:21:49Z

It cannot be certificate based authorization.

Merge date: f86dd85 Tue Jun 27 12:52:14 2023 +0300
Report date: Jun 19 2023

elcallio · 2023-07-03T12:40:49Z

@mykaul scylladb/seastar@f461641 only changes things that happen after a connect. And even so only when demanded. I very much doubt it. Let me repro the issue.

elcallio · 2023-07-03T13:46:49Z

So the problem is that direct_failure_detector::failure_detector::impl::update_endpoint_fiber, which is started at main.cc:1200 (roughly) tries to send an RPC message before messaging_service::start_listen has been called. This is less great, because this is where TLS certificates are created.

Adding a neat little assert:

diff --git a/message/messaging_service.cc b/message/messaging_service.cc
index 1e78476417..15f2cd444d 100644
--- a/message/messaging_service.cc
+++ b/message/messaging_service.cc
@@ -866,6 +866,8 @@ shared_ptr<messaging_service::rpc_protocol_client_wrapper> messaging_service::ge
     opts.reuseaddr = true;
     opts.isolation_cookie = _scheduling_info_for_connection_index[idx].isolation_cookie;
 
+    assert(!must_encrypt || _credentials);
+
     auto client = must_encrypt ?
                     ::make_shared<rpc_protocol_client_wrapper>(_rpc->protocol(), std::move(opts),
                                     remote_addr, laddr, _credentials) :

will demonstrate this excellently by transforming the segfault to a much earlier and more traceable crash.

I would suggest probably making message service TLS init earlier iff code requires sending messages before listeners are up.

elcallio · 2023-07-03T13:50:14Z

So if you want a commit to blame, I would suggest 38f65e5

Fixes scylladb#14299 failure_detector can try sending messages to TLS endpoints before start_listen has been called (why?). Need TLS initialized before this. So do on service creation.

fruch · 2023-07-16T08:00:06Z

we need this in 5.3 (upgrade tests from 5.3.0-rc0, are failing cause if it)

DoronArazii · 2023-07-16T08:12:57Z

@fruch do you need a permission to add labels? ^^

fruch · 2023-07-16T09:22:39Z

@fruch do you need a permission to add labels? ^^

I have it, I wasn't sure which one to add, or that 'backport candidate' was enough.

eliransin · 2023-07-30T08:50:35Z

@DoronArazii We need to figure out if this regression might find it's way to 5.2 if there is a chance for backport one day
so we should backport the test right now to catch it (https://github.com/scylladb/scylla-dtest/commit/78a2dcd7f7ef015b9ad42f19599614bb2886710b)
If this test passes in 5.2 I would either way add it to the gating/dtest if not already there.

fruch · 2023-07-30T14:01:53Z

@DoronArazii We need to figure out if this regression might find it's way to 5.2 if there is a chance for backport one day so we should backport the test right now to catch it (scylladb/scylla-dtest@78a2dcd) If this test passes in 5.2 I would either way add it to the gating/dtest if not already there.

the test is gating on master

I'll test it on 5.2 and 2023.1, and if working I'll backport it

fruch · 2023-07-30T14:12:28Z

Tested and it was working on 5.2/2023.1

backported the test as gating to 5.2/2023.1

DoronArazii · 2023-08-01T08:11:50Z

Removing 'backport candidate' label.
/Cc @eliransin

fruch added area/tls triage/master Looking for assignee labels Jun 19, 2023

mykaul assigned elcallio Jun 20, 2023

mykaul added the status/regression label Jun 20, 2023

DoronArazii added this to the 5.4 milestone Jun 25, 2023

DoronArazii added the P1 Urgent label Jun 25, 2023

mykaul mentioned this issue Jun 29, 2023

Got Segmentation fault while starting upgraded Scylla in rolling upgrade test #14438

Closed

2 tasks

eliransin self-assigned this Jul 2, 2023

avikivity added a commit that referenced this issue Jul 3, 2023

Revert "Merge 'Certificate based authorization' from Calle Wilund"

9635529

This reverts commit f86dd85, reversing changes made to 3ca0c6c. It breaks setups that use client encryption. Fixes #14299.

elcallio mentioned this issue Jul 3, 2023

messaging_service: Do TLS init early #14493

Closed

tchaikov mentioned this issue Jul 4, 2023

CoreDumpEvent(Segmentation fault) during rolling upgrade in ami and ami-arm test #14342

Closed

2 tasks

DoronArazii removed the triage/master Looking for assignee label Jul 9, 2023

yarongilor mentioned this issue Jul 11, 2023

memory corruption - "Segmentation fault on shard 2" on 4 nodes after nodetool refresh #14618

Closed

2 tasks

elcallio mentioned this issue Jul 11, 2023

TLS: Segmentation fault on shard 0 when starting scylla #14630

Closed

2 tasks

scylladb-promoter closed this as completed in e1a52af Jul 12, 2023

scylladb-promoter added the Backport candidate label Jul 12, 2023

DoronArazii added the Requires-Backport-to-5.3 label Jul 16, 2023

DoronArazii removed the Requires-Backport-to-5.3 label Jul 30, 2023

DoronArazii removed the Backport candidate label Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault on node configured to use client encryption #14299

Segmentation fault on node configured to use client encryption #14299

fruch commented Jun 19, 2023 •

edited

Logs:

mykaul commented Jun 20, 2023

fruch commented Jun 20, 2023 •

edited

elcallio commented Jun 21, 2023

fruch commented Jun 22, 2023

Logs:

fruch commented Jun 22, 2023

DoronArazii commented Jun 25, 2023

fruch commented Jun 25, 2023

Logs:

mykaul commented Jun 29, 2023

fruch commented Jun 29, 2023

fruch commented Jun 29, 2023

eliransin commented Jul 2, 2023

elcallio commented Jul 3, 2023

mykaul commented Jul 3, 2023

elcallio commented Jul 3, 2023

avikivity commented Jul 3, 2023

elcallio commented Jul 3, 2023

mykaul commented Jul 3, 2023

avikivity commented Jul 3, 2023

elcallio commented Jul 3, 2023

elcallio commented Jul 3, 2023

elcallio commented Jul 3, 2023

fruch commented Jul 16, 2023

DoronArazii commented Jul 16, 2023

fruch commented Jul 16, 2023

eliransin commented Jul 30, 2023

fruch commented Jul 30, 2023

fruch commented Jul 30, 2023

DoronArazii commented Aug 1, 2023

Segmentation fault on node configured to use client encryption #14299

Segmentation fault on node configured to use client encryption #14299

Comments

fruch commented Jun 19, 2023 • edited

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

mykaul commented Jun 20, 2023

fruch commented Jun 20, 2023 • edited

elcallio commented Jun 21, 2023

fruch commented Jun 22, 2023

Installation details

Logs:

fruch commented Jun 22, 2023

DoronArazii commented Jun 25, 2023

fruch commented Jun 25, 2023

Installation details

Logs:

mykaul commented Jun 29, 2023

fruch commented Jun 29, 2023

fruch commented Jun 29, 2023

eliransin commented Jul 2, 2023

elcallio commented Jul 3, 2023

mykaul commented Jul 3, 2023

elcallio commented Jul 3, 2023

avikivity commented Jul 3, 2023

elcallio commented Jul 3, 2023

mykaul commented Jul 3, 2023

avikivity commented Jul 3, 2023

elcallio commented Jul 3, 2023

elcallio commented Jul 3, 2023

elcallio commented Jul 3, 2023

fruch commented Jul 16, 2023

DoronArazii commented Jul 16, 2023

fruch commented Jul 16, 2023

eliransin commented Jul 30, 2023

fruch commented Jul 30, 2023

fruch commented Jul 30, 2023

DoronArazii commented Aug 1, 2023

fruch commented Jun 19, 2023 •

edited

fruch commented Jun 20, 2023 •

edited