gossiper crashed during shutdown

[gleb-cloudius]

trace:

#0  0x00007f77a62af887 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f77a62b0f78 in __GI_abort () at abort.c:89
#2  0x00007f77a62a87e6 in __assert_fail_base (
fmt=0x7f77a63f5be8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
assertion=assertion@entry=0xe8f5ac "local_is_initialized()", 
file=file@entry=0xe8f328 "/home/gleb/work/seastar/seastar/core/sharded.hh", line=line@entry=437, 
function=function@entry=0xf8e840 <seastar::sharded<gms::gossiper>::local()::__PRETTY_FUNCTION__> "Service& seastar::sharded<Service>::local() [with Service = gms::gossiper]") at assert.c:92
#3  0x00007f77a62a8892 in __GI___assert_fail (
assertion=0xe8f5ac "local_is_initialized()", 
file=0xe8f328 "/home/gleb/work/seastar/seastar/core/sharded.hh", line=437, 
function=0xf8e840 <seastar::sharded<gms::gossiper>::local()::__PRETTY_FUNCTION__> "Service& seastar::sharded<Service>::local() [with Service = gms::gossiper]") at assert.c:101
#4  0x0000000000857479 in seastar::shardedgms::gossiper::local (
this=<optimized out>)
at /home/gleb/work/seastar/seastar/core/sharded.hh:437
#5  0x0000000000a34b71 in get_local_gossiper () at ./gms/gossiper.hh:438
#6  operator() (__closure=0x60000007fcd8) at gms/gossiper.cc:231
#7  futurize<future<> >::applygms::gossiper::init_messaging_service_handler()::<lambda(gms::inet_address)::<lambda()>&>(gms::gossiper::<lambda(gms::inet_address)>::<lambda()> &) (func=...)
at /home/gleb/work/seastar/seastar/core/future.hh:1093
#8  0x0000000000a498b0 in submit_togms::gossiper::init_messaging_service_handler()::<lambda(gms::inet_address)::<lambda()> > (t=0, func=<optimized out>)
at /home/gleb/work/seastar/seastar/core/reactor.hh:987
#9  operator() (__closure=<optimized out>, from=...) at gms/gossiper.cc:242
#10 std::_Function_handler<rpc::no_wait_type(gms::inet_address), gms::gossiper::init_messaging_service_handler()::<lambda(gms::inet_address)> >::_M_invoke(const std::_Any_data &, gms::inet_address) (__functor=..., __args#0=...)
at /usr/include/c++/4.9.2/functional:2024
#11 0x0000000000b21b81 in _ZZN3rpc11recv_helperIN3net10serializerENS1_14messaging_verbESt8functionIFNS_12no_wait_typeEN3gms12inet_addressEEES5_IS7_ENS_21dont_want_client_infoEEEDaNS_9signatureIFT2_DpT3_EEEOT1_T4_ENUl13lw_shared_ptrINS_8protocolIS2_S3_E6server10connectionEEl16temporary_bufferIcEE_clESP_lSR_ ()
at /usr/include/c++/4.9.2/functional:2439
#12 0x0000000000b21d45 in _ZNSt17_Function_handlerIFv13lw_shared_ptrIN3rpc8protocolIN3net10serializerENS3_14messaging_verbEE6server10connectionEEl16temporary_bufferIcEEZNS1_11recv_helperIS4_S5_St8functionIFNS1_12no_wait_typeEN3gms12inet_addressEEESF_ISH_ENS1_21dont_want_client_infoEEEDaNS1_9signatureIFT2_DpT3_EEEOT1_T4_EUlS9_lSB_E_E9_M_invokeERKSt9_Any_dataS9_lSB_ (__functor=..., __args#0=...,
__args#1=<optimized out>, __args#2=...)
at /usr/include/c++/4.9.2/functional:2039
#13 0x0000000000ad89ce in std::function<void (lw_shared_ptr<rpc::protocol<net::serializer, net::messaging_verb>::server::connection>, long, temporary_buffer<char>)>::operator()(lw_shared_ptr<rpc::protocol<net::serializer, net::messaging_verb>::server::connection>, long, temporary_buffer<char>) const (
this=this@entry=0x600000f90bd0, __args#0=..., __args#1=__args#1@entry=293, 
__args#2=...) at /usr/include/c++/4.9.2/functional:2439
#14 0x0000000000b0f3a0 in rpc::protocol<net::serializer, net::messaging_verb>::server::connection::process()::{lambda()#2}::operator()()::{lambda(net::messaging_verb, long, temporary_buffer<char>)#1}::operator()(net::messaging_verb, long, temporary_buffer) const (__closure=__closure@entry=0x7fff8556d448,
type=net::GOSSIP_SHUTDOWN, msg_id=293, data=...)
at /home/gleb/work/seastar/seastar/rpc/rpc_impl.hh:574
#15 0x0000000000b0f4be in apply (
args=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x9623d58>, 
func=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x9623d46>) at /home/gleb/work/seastar/seastar/core/apply.hh:34
#16 apply<rpc::protocol<Serializer, MsgType>::server::connection::process()::<lambda()> mutable [with Serializer = net::serializer; MsgType = net::messaging_verb]::<lambda(net::messaging_verb, int64_t, temporary_buffer<char>)>, net::messaging_verb, long int, temporary_buffer<char> > (
args=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x945e0ab>, 
func=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x945e097>) at /home/gleb/work/seastar/seastar/core/apply.hh:42
#17 futurize<void>::apply<rpc::protocol<net::serializer, net::messaging_verb>::server::connection::process()::{lambda()#2}::operator()()::{lambda(net::messaging_verb, long, temporary_buffer<char>)#1}, net::messaging_verb, long, temporary_buffer>(rpc::protocol<net::serializer, net::messaging_verb>::server::connection::process()::{lambda()#2}::operator()()::{lambda(net::messaging_verb, long, temporary_buffer<char>)#1}&&, std::tuple<net::messaging_verb, long, temporary_buffer>&&) (
func=func@entry=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x9623dac>, 
args=args@entry=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x962400d>)
at /home/gleb/work/seastar/seastar/core/future.hh:1062
#18 0x0000000000b0f598 in operator()<future_state<net::messaging_verb, long int, temporary_buffer<char> > > (
state=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x905479b, DIE 0x948c4fa>, __closure=0x600000304c78)
at /home/gleb/work/seastar/seastar/core/future.hh:779
#19 _ZN12continuationIZN6futureIIN3net14messaging_verbEl16temporary_bufferIcEEE4thenIZZN3rpc8protocolINS1_10serializerES2_E6server10connection7processEvENUlvE0_clEvEUlS2_lS4_E_S0_IIEEEET0_OT_EUlOT_E_IS2_lS4_EE3runEv (this=0x600000304c40)
at /home/gleb/work/seastar/seastar/core/future.hh:359
#20 0x0000000000466052 in reactor::run_tasks (this=this@entry=0x6000001da000,
tasks=..., quota=<optimized out>) at core/reactor.cc:1093
#21 0x000000000048cf73 in reactor::run (this=0x6000001da000)
at core/reactor.cc:1190
#22 0x00000000004ddcbf in app_template::run_deprecated(int, char**, std::function<void ()>&&) (this=this@entry=0x7fff8556db70, ac=ac@entry=17,
av=av@entry=0x7fff8556ddb8, 
func=func@entry=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x4f90d4, DIE 0x5a05cd>) at core/app-template.cc:122
#23 0x000000000041e37c in main (ac=17, av=0x7fff8556ddb8) at main.cc:265

[gleb-cloudius]

rpc handler tries to access gossiper after it was stopped. gossiper::stop() need to unregister rpc handlers to avoid this.

[slivne]

lets not start fixing this issues - we have a hack that calle submitted
that we should first test out so its ok to open a bug, but lets not fix
this items, we are going to disable all stop/shutdown

On Tue, Sep 8, 2015 at 2:47 PM, Gleb Natapov notifications@github.com
wrote:

rpc handler tries to access gossiper after it was stopped.
gossiper::stop() need to unregister rpc handlers to avoid this.

—
Reply to this email directly or view it on GitHub
#311 (comment)
.

[gleb-cloudius]

OK. The fix should be trivial though.

[gleb-cloudius]

Actually gossiper already unregisters its handlers, but probably not at the right time, so not so trivial as I thought.

[slivne]

@gleb-cloudius can we close this as we have updated shutdown

[gleb-cloudius]

On Sun, Sep 13, 2015 at 12:05:04AM -0700, slivne wrote:

@gleb-cloudius can we close this as we have updated shutdown

Of course we cannot. The problem was, hopefully, hidden, not fixed. It
will reappear when workaround will be reverted.

        Gleb.

[glommer]

It's been 2 years. Can we close this one ?

[duarten]

I think we can, there have been a bunch of improvements here recently.