assert in streaming::stream_manager during exit

[gleb-cloudius]

Got this after ^C while other node was joining:

scylla: /home/gleb/work/seastar/seastar/core/sharded.hh:439: Service& seastar::sharded<Service>::local() [with Service = streaming::stream_manager]: Assertion `local_is_initialized()' failed.

#0  0x00007f1f6c605877 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f1f6c606f68 in __GI_abort () at abort.c:89
#2  0x00007f1f6c5fe7d6 in __assert_fail_base (
    fmt=0x7f1f6c74bce8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x1217d88 "local_is_initialized()", 
    file=file@entry=0x1217950 "/home/gleb/work/seastar/seastar/core/sharded.hh", line=line@entry=439, 
    function=function@entry=0x15645c0 <_ZZN7seastar7shardedIN9streaming14stream_managerEE5localEvE19__PRETTY_FUNCTION__> "Service& seastar::sharded<Service>::local() [with Service = streaming::stream_manager]") at assert.c:92
#3  0x00007f1f6c5fe882 in __GI___assert_fail (
    assertion=0x1217d88 "local_is_initialized()", 
    file=0x1217950 "/home/gleb/work/seastar/seastar/core/sharded.hh", 
    line=439, 
    function=0x15645c0 <_ZZN7seastar7shardedIN9streaming14stream_managerEE5localEvE19__PRETTY_FUNCTION__> "Service& seastar::sharded<Service>::local() [with Service = streaming::stream_manager]") at assert.c:101
#4  0x0000000000e93e29 in seastar::sharded<streaming::stream_manager>::local (
    this=<optimized out>)
    at /home/gleb/work/seastar/seastar/core/sharded.hh:439
#5  0x0000000000e8bb29 in get_local_stream_manager ()
    at ./streaming/stream_manager.hh:190
---Type <return> to continue, or q <return> to quit---
#6  operator() (__closure=<optimized out>)
    at streaming/stream_transfer_task.cc:114
#7  do_void_futurize_apply<streaming::stream_transfer_task::start()::<lambda(mutation&&)>::<lambda()>::<lambda()> > (func=<optimized out>)
    at /home/gleb/work/seastar/seastar/core/future.hh:1091
#8  apply<streaming::stream_transfer_task::start()::<lambda(mutation&&)>::<lambda()>::<lambda()> > (func=<optimized out>)
    at /home/gleb/work/seastar/seastar/core/future.hh:1139
#9  operator() (__closure=<optimized out>, result=...)
    at /home/gleb/work/seastar/seastar/core/future.hh:916
#10 futurize<future<> >::apply<future<T>::finally(Func&&) [with Func = streaming::stream_transfer_task::start()::<lambda(mutation&&)>::<lambda()>::<lambda()>; T = {}]::<lambda(future<>)>, future<> >(<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0xd1e4e86, DIE 0xd325e6d>, <unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0xd1e4e86, DIE 0xd325e5c>) (
    func=<optimized out>)
    at /home/gleb/work/seastar/seastar/core/future.hh:1156
#11 0x0000000000e8f0e2 in operator()<future_state<> > (
    state=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0xd1e4e86, DIE 0xd34aaca>, __closure=0x60000610a6d0)
    at /home/gleb/work/seastar/seastar/core/future.hh:833
#12 continuation<future<T>::then_wrapped(Func&&) [with Func = future<T>::finally(Func&&) [with Func = streaming::stream_transfer_task::start()::<lambda(mutation---Type <return> to continue, or q <return> to quit---
&&)>::<lambda()>::<lambda()>; T = {}]::<lambda(future<>)>; Result = future<>; T = {}]::<lambda(auto:4&&)> >::run(void) (this=0x60000610a6c0)
    at /home/gleb/work/seastar/seastar/core/future.hh:362
#13 0x000000000048066f in reactor::run_tasks (this=this@entry=0x600000302000, 
    tasks=...) at core/reactor.cc:1222
#14 0x00000000004adcbb in reactor::run (this=0x600000302000)
    at core/reactor.cc:1348
#15 0x0000000000505b33 in app_template::run_deprecated(int, char**, std::function<void ()>&&) (this=this@entry=0x7ffe68c9c7e0, ac=ac@entry=25, 
    av=av@entry=0x7ffe68c9ca28, 
    func=func@entry=<unknown type in /home/gleb/work/seastar/build/release/scylla, CU 0x53f1d6, DIE 0x5eb251>) at core/app-template.cc:123
#16 0x0000000000420a0b in main (ac=25, av=0x7ffe68c9ca28) at main.cc:352

[slivne]

we have a similar unitest that kills the new node dest - did you kill the
src node ?

On Wed, Nov 25, 2015 at 3:19 PM, Gleb Natapov notifications@github.com
wrote:

Got this after ^C while other node was joining:

scylla: /home/gleb/work/seastar/seastar/core/sharded.hh:439: Service&
seastar::sharded::local() [with Service = streaming::stream_manager]:
Assertion `local_is_initialized()' failed.

#0 0x00007f1f6c605877 in

_GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 #1 0x00007f1f6c606f68 in
__GI_abort () at abort.c:89 #2
#2 0x00007f1f6c5fe7d6 in
__assert_fail_base ( fmt=0x7f1f6c74bce8 "%s%s%s:%u: %s%sAssertion `%s'
failed.\n%n", assertion=assertion@entry=0x1217d88 "local_is_initialized()",
file=file@entry=0x1217950
"/home/gleb/work/seastar/seastar/core/sharded.hh", line=line@entry=439,
function=function@entry=0x15645c0
<ZZN7seastar7shardedIN9streaming14stream_managerEE5localEvE19__PRETTY_FUNCTION>
"Service& seastar::sharded::local() [with Service =
streaming::stream_manager]") at assert.c:92
#3 #3 0x00007f1f6c5fe882 in

_GI___assert_fail ( assertion=0x1217d88 "local_is_initialized()",
file=0x1217950 "/home/gleb/work/seastar/seastar/core/sharded.hh", line=439,
function=0x15645c0
<ZZN7seastar7shardedIN9streaming14stream_managerEE5localEvE19__PRETTY_FUNCTION>
"Service& seastar::sharded::local() [with Service =
streaming::stream_manager]") at assert.c:101
#4 #4 0x0000000000e93e29 in
seastar::shardedstreaming::stream_manager::local (
this=)
at /home/gleb/work/seastar/seastar/core/sharded.hh:439
#5 #5 0x0000000000e8bb29 in
get_local_stream_manager ()
at ./streaming/stream_manager.hh:190
---Type to continue, or q to quit---
#6 #6 operator() (__closure=)
at streaming/stream_transfer_task.cc:114
#7 #7
do_void_futurize_applystreaming::stream_transfer_task::start()::<lambda(mutation&&)::::

(func=)
at /home/gleb/work/seastar/seastar/core/future.hh:1091
#8 #8
applystreaming::stream_transfer_task::start()::<lambda(mutation&&):::: >
(func=)
at /home/gleb/work/seastar/seastar/core/future.hh:1139
#9 #9 operator() (__closure=,
result=...)
at /home/gleb/work/seastar/seastar/core/future.hh:916
#10 #10 futurize
::apply::finally(Func&&) [with Func =
streaming::stream_transfer_task::start()::::::; T = {}]::)>, future<> >(, )
(
func=)
at /home/gleb/work/seastar/seastar/core/future.hh:1156
#11 #11 0x0000000000e8f0e2 in
operator() > (
state=, __closure=0x60000610a6d0)
at /home/gleb/work/seastar/seastar/core/future.hh:833
#12 #12
continuation::then_wrapped(Func&&) [with Func = future::finally(Func&&)
[with Func = streaming::stream_transfer_task::start():: to continue, or q
to quit---
&&)>::::; T = {}]::)>; Result = future<>; T = {}]:: >::run(void)
(this=0x60000610a6c0)
at /home/gleb/work/seastar/seastar/core/future.hh:362
#13 #13 0x000000000048066f in
reactor::run_tasks (this=this@entry=0x600000302000,
tasks=...) at core/reactor.cc:1222
#14 #14 0x00000000004adcbb in
reactor::run (this=0x600000302000)
at core/reactor.cc:1348
#15 #15 0x0000000000505b33 in
app_template::run_deprecated(int, char**, std::function&&) (this=this@entry=0x7ffe68c9c7e0,
ac=ac@entry=25,
av=av@entry=0x7ffe68c9ca28,
func=func@entry=) at core/app-template.cc:123
#16 #16 0x0000000000420a0b in
main (ac=25, av=0x7ffe68c9ca28) at main.cc:352

—
Reply to this email directly or view it on GitHub
#608.

[slivne]

@gleb-cloudius - can you please provide the detailed scenario so we can add a test for regression

[gleb-cloudius]

On Sun, Nov 29, 2015 at 03:07:51AM -0800, slivne wrote:

@gleb-cloudius - can you please provide the detailed scenario so we can add a test for regression

Run 3-node cluster. Start c-s write against it. Start one more node.
While node is joining kill one of original nodes with ^C. The one that
is stopped will crash.

        Gleb.

[slivne]

Add a test update_cluster_layout_tests.py:TestUpdateClusterLayout.simple_kill_streaming_node_while_bootstrapping_test to test this out,

Please note that this test passes with Cassandra as on Ctrl+C streaming continues - in our case we know that shutdown does not behave in the same manner

[slivne]

@asias is this still happening - did you resolve this ?

[asias]

I do not see the crash in simple_kill_streaming_node_while_bootstrapping_test test.