Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CQL: double-free in get_query_state() #329

Closed
pdziepak opened this issue Sep 9, 2015 · 3 comments
Closed

CQL: double-free in get_query_state() #329

pdziepak opened this issue Sep 9, 2015 · 3 comments
Labels
bug
Milestone
anglerfish - scyl...

Comments

@pdziepak
Copy link
Contributor

pdziepak commented Sep 9, 2015 

==19261==ERROR: AddressSanitizer: attempting double-free on 0x610000e4b540 in thread T1:
    #0 0x7f2ccc16354f in __interceptor_free (/lib64/libasan.so.1+0x5754f)
    #1 0x5fbfd1 in __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::deallocate(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/ext/new_allocator.h:110
    #2 0x5e2e90 in std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::deallocate(std::allocator<std::__detail::_Hash_node_base*>&, std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/alloc_traits.h:383
    #3 0x17c7db1 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false> > >::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/hashtable_policy.h:2012
    #4 0x17bba54 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/hashtable.h:356
    #5 0x17adcc8 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_deallocate_buckets() /usr/include/c++/4.9.2/bits/hashtable.h:361
    #6 0x17d9e88 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) /usr/include/c++/4.9.2/bits/hashtable.h:2026
    #7 0x17d1193 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) /usr/include/c++/4.9.2/bits/hashtable.h:1980
    #8 0x17c96eb in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false>*) /usr/include/c++/4.9.2/bits/hashtable.h:1617
    #9 0x17bc9d9 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<unsigned short&, service::client_state&>(std::integral_constant<bool, true>, unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:1564
    #10 0x17aed91 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:718
    #11 0x17a1df2 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::unordered_map<unsigned short, transport::cql_query_state, std::hash<unsigned short>, std::equal_to<unsigned short>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> > >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/unordered_map.h:341
    #12 0x17232ab in transport::cql_server::connection::get_query_state(unsigned short) transport/server.cc:465
    #13 0x1723f3e in transport::cql_server::connection::process_prepare(unsigned short, temporary_buffer<char>)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() (/home/pdziepak/urchin/build/debug/scylla+0x1723f3e)
    #14 0x1748b57 in apply<transport::cql_server::connection::process_prepare(uint16_t, temporary_buffer<char>)::<lambda(unsigned int)> mutable::<lambda()>&> /home/pdziepak/urchin/seastar/core/future.hh:1072
    #15 0x17907b5 in process /home/pdziepak/urchin/seastar/core/reactor.hh:502
    #16 0x42d479 in smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}::operator()(smp_message_queue::work_item*) const (/home/pdziepak/urchin/build/debug/scylla+0x42d479)
    #17 0x453cca in process_queue<2ul, smp_message_queue::process_incoming()::<lambda(smp_message_queue::work_item*)> > core/reactor.cc:1423
    #18 0x42d699 in smp_message_queue::process_incoming() core/reactor.cc:1451
    #19 0x524033 in smp::poll_queues() (/home/pdziepak/urchin/build/debug/scylla+0x524033)
    #20 0x5776eb in std::unique_ptr<reactor::pollfn, std::default_delete<reactor::pollfn> > reactor::make_pollfn<bool (&)()>(bool (&)())::the_pollfn::poll_and_check_more_work() (/home/pdziepak/urchin/build/debug/scylla+0x5776eb)
    #21 0x42a779 in reactor::poll_once() core/reactor.cc:1221
    #22 0x42a0ec in reactor::run() core/reactor.cc:1200
    #23 0x432195 in smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()() const (/home/pdziepak/urchin/build/debug/scylla+0x432195)
    #24 0x4f28c9 in _M_invoke /usr/include/c++/4.9.2/functional:2039
    #25 0x53c313 in std::function<void ()>::operator()() const /usr/include/c++/4.9.2/functional:2439
    #26 0x720e54 in posix_thread::start_routine(void*) core/posix.cc:51
    #27 0x7f2ccac52529 in start_thread (/lib64/libpthread.so.0+0x7529)
    #28 0x7f2cca98e22c in __clone (/lib64/libc.so.6+0x10022c)

0x610000e4b540 is located 0 bytes inside of 184-byte region [0x610000e4b540,0x610000e4b5f8)
freed by thread T2 here:
    #0 0x7f2ccc16354f in __interceptor_free (/lib64/libasan.so.1+0x5754f)
    #1 0x5fbfd1 in __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::deallocate(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/ext/new_allocator.h:110
    #2 0x5e2e90 in std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::deallocate(std::allocator<std::__detail::_Hash_node_base*>&, std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/alloc_traits.h:383
    #3 0x17c7db1 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false> > >::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/hashtable_policy.h:2012
    #4 0x17bba54 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/4.9.2/bits/hashtable.h:356
    #5 0x17adcc8 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_deallocate_buckets() /usr/include/c++/4.9.2/bits/hashtable.h:361
    #6 0x17d9e88 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) /usr/include/c++/4.9.2/bits/hashtable.h:2026
    #7 0x17d1193 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) /usr/include/c++/4.9.2/bits/hashtable.h:1980
    #8 0x17c96eb in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false>*) /usr/include/c++/4.9.2/bits/hashtable.h:1617
    #9 0x17bc9d9 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<unsigned short&, service::client_state&>(std::integral_constant<bool, true>, unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:1564
    #10 0x17aed91 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:718
    #11 0x17a1df2 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::unordered_map<unsigned short, transport::cql_query_state, std::hash<unsigned short>, std::equal_to<unsigned short>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> > >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/unordered_map.h:341
    #12 0x17232ab in transport::cql_server::connection::get_query_state(unsigned short) transport/server.cc:465
    #13 0x1723f3e in transport::cql_server::connection::process_prepare(unsigned short, temporary_buffer<char>)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() (/home/pdziepak/urchin/build/debug/scylla+0x1723f3e)
    #14 0x1748b57 in apply<transport::cql_server::connection::process_prepare(uint16_t, temporary_buffer<char>)::<lambda(unsigned int)> mutable::<lambda()>&> /home/pdziepak/urchin/seastar/core/future.hh:1072
    #15 0x17907b5 in process /home/pdziepak/urchin/seastar/core/reactor.hh:502
    #16 0x42d479 in smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}::operator()(smp_message_queue::work_item*) const (/home/pdziepak/urchin/build/debug/scylla+0x42d479)
    #17 0x453cca in process_queue<2ul, smp_message_queue::process_incoming()::<lambda(smp_message_queue::work_item*)> > core/reactor.cc:1423
    #18 0x42d699 in smp_message_queue::process_incoming() core/reactor.cc:1451
    #19 0x524033 in smp::poll_queues() (/home/pdziepak/urchin/build/debug/scylla+0x524033)
    #20 0x5776eb in std::unique_ptr<reactor::pollfn, std::default_delete<reactor::pollfn> > reactor::make_pollfn<bool (&)()>(bool (&)())::the_pollfn::poll_and_check_more_work() (/home/pdziepak/urchin/build/debug/scylla+0x5776eb)
    #21 0x42a779 in reactor::poll_once() core/reactor.cc:1221
    #22 0x42a0ec in reactor::run() core/reactor.cc:1200
    #23 0x432195 in smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()() const (/home/pdziepak/urchin/build/debug/scylla+0x432195)
    #24 0x4f28c9 in _M_invoke /usr/include/c++/4.9.2/functional:2039
    #25 0x53c313 in std::function<void ()>::operator()() const /usr/include/c++/4.9.2/functional:2439
    #26 0x720e54 in posix_thread::start_routine(void*) core/posix.cc:51
    #27 0x7f2ccac52529 in start_thread (/lib64/libpthread.so.0+0x7529)

previously allocated by thread T3 here:
    #0 0x7f2ccc1637c7 in malloc (/lib64/libasan.so.1+0x577c7)
    #1 0x7f2ccf636607 in operator new(unsigned long) (/lib64/libstdc++.so.6+0x5f607)
    #2 0x5fbed8 in std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) /usr/include/c++/4.9.2/bits/alloc_traits.h:357
    #3 0x17cfe65 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false> > >::_M_allocate_buckets(unsigned long) /usr/include/c++/4.9.2/bits/hashtable_policy.h:1998
    #4 0x17c79d0 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_allocate_buckets(unsigned long) /usr/include/c++/4.9.2/bits/hashtable.h:347
    #5 0x17d9964 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) /usr/include/c++/4.9.2/bits/hashtable.h:2001
    #6 0x17d1193 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) /usr/include/c++/4.9.2/bits/hashtable.h:1980
    #7 0x17c96eb in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, transport::cql_query_state>, false>*) /usr/include/c++/4.9.2/bits/hashtable.h:1617
    #8 0x17bc9d9 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<unsigned short&, service::client_state&>(std::integral_constant<bool, true>, unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:1564
    #9 0x17aed91 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/hashtable.h:718
    #10 0x17a1df2 in std::pair<std::__detail::_Node_iterator<std::pair<unsigned short const, transport::cql_query_state>, false, false>, bool> std::unordered_map<unsigned short, transport::cql_query_state, std::hash<unsigned short>, std::equal_to<unsigned short>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> > >::emplace<unsigned short&, service::client_state&>(unsigned short&, service::client_state&) /usr/include/c++/4.9.2/bits/unordered_map.h:341
    #11 0x17232ab in transport::cql_server::connection::get_query_state(unsigned short) transport/server.cc:465
    #12 0x17238af in transport::cql_server::connection::process_query(unsigned short, temporary_buffer<char>) transport/server.cc:482
    #13 0x171da3b in transport::cql_server::connection::process_request_one(temporary_buffer<char>, unsigned char, unsigned short)::{lambda()#1}::operator()() (/home/pdziepak/urchin/build/debug/scylla+0x171da3b)
    #14 0x174150b in apply /home/pdziepak/urchin/seastar/core/apply.hh:34
    #15 0x17415e8 in apply<transport::cql_server::connection::process_request_one(temporary_buffer<char>, uint8_t, uint16_t)::<lambda()> > /home/pdziepak/urchin/seastar/core/apply.hh:42
    #16 0x17416cc in apply<transport::cql_server::connection::process_request_one(temporary_buffer<char>, uint8_t, uint16_t)::<lambda()> > /home/pdziepak/urchin/seastar/core/future.hh:1083
    #17 0x1791f57 in _ZZN6futureIIEE4thenIZN9transport10cql_server10connection19process_request_oneE16temporary_bufferIcEhtEUlvE_S0_EET0_OT_ENUlOT_E_clI12future_stateIIEEEEDaSA_ (/home/pdziepak/urchin/build/debug/scylla+0x1791f57)
    #18 0x17920f0 in run /home/pdziepak/urchin/seastar/core/future.hh:359
    #19 0x427058 in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&, unsigned long) core/reactor.cc:1093
    #20 0x429edd in reactor::run() core/reactor.cc:1190
    #21 0x432195 in smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()() const (/home/pdziepak/urchin/build/debug/scylla+0x432195)
    #22 0x4f28c9 in _M_invoke /usr/include/c++/4.9.2/functional:2039
    #23 0x53c313 in std::function<void ()>::operator()() const /usr/include/c++/4.9.2/functional:2439
    #24 0x720e54 in posix_thread::start_routine(void*) core/posix.cc:51
    #25 0x7f2ccac52529 in start_thread (/lib64/libpthread.so.0+0x7529)

Thread T1 created by T0 here:
    #0 0x7f2ccc12fdba in pthread_create (/lib64/libasan.so.1+0x23dba)
    #1 0x7213dd in posix_thread::posix_thread(posix_thread::attr, std::function<void ()>) core/posix.cc:87
    #2 0x720f12 in posix_thread::posix_thread(std::function<void ()>) core/posix.cc:60
    #3 0x4d627c in construct<posix_thread, smp::configure(boost::program_options::variables_map)::<lambda()> > /usr/include/c++/4.9.2/ext/new_allocator.h:120
    #4 0x4acc87 in _S_construct<posix_thread, smp::configure(boost::program_options::variables_map)::<lambda()> > /usr/include/c++/4.9.2/bits/alloc_traits.h:253
    #5 0x47b5ac in construct<posix_thread, smp::configure(boost::program_options::variables_map)::<lambda()> > /usr/include/c++/4.9.2/bits/alloc_traits.h:399
    #6 0x47b707 in _M_emplace_back_aux<smp::configure(boost::program_options::variables_map)::<lambda()> > /usr/include/c++/4.9.2/bits/vector.tcc:416
    #7 0x45550a in emplace_back<smp::configure(boost::program_options::variables_map)::<lambda()> > /usr/include/c++/4.9.2/bits/vector.tcc:101
    #8 0x433418 in smp::configure(boost::program_options::variables_map) core/reactor.cc:1841
    #9 0x776406 in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:108
    #10 0xa6fdc3 in main /home/pdziepak/urchin/main.cc:279
    #11 0x7f2cca8adfdf in __libc_start_main (/lib64/libc.so.6+0x1ffdf)

Unfortunately, I haven't been able to find any better way to reproduce this than running Spark Cassandra connector (#317) integration tests.
@slivne slivne added the bug label Sep 10, 2015
@slivne slivne added this to the anglerfish milestone Sep 10, 2015
@asias
Copy link
Contributor

asias commented Sep 10, 2015

gdb --args ./build/release/scylla --network-stack posix --cluster-name=cluster101 --num-tokens=3 --logger-log-level gossip=debug -c 6 -m 2048M --rpc-address 127.0.0.1 --listen-address 127.0.0.1 --seed-provider-parameters seeds=127.0.0.1 --datadir /home/asias/src/cloudius-systems/urchin/tmp/1 --commitlog-directory /home/asias/src/cloudius-systems/urchin/tmp/1

 (gdb) bt
#0  0x00000000007b69c3 in std::_Hashtable<unsigned short, std::pair<unsigned short const, transport::cql_query_state>, std::allocator<std::pair<unsigned short const, transport::cql_query_state> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node (this=this@entry=0x604000134cb8, __n=0, __k=__k@entry=@0x7ffff63fa33c: 192, __code=192) at /usr/include/c++/5.1.1/bits/hashtable.h:1430
#1  0x00000000007c054b in _M_find_node (__c=192, __key=@0x7ffff63fa33c: 192, __bkt=<optimized out>, this=0x604000134cb8) at /usr/include/c++/5.1.1/bits/hashtable.h:632
#2  find (__k=@0x7ffff63fa33c: 192, this=0x604000134cb8) at /usr/include/c++/5.1.1/bits/hashtable.h:1307
#3  find (__x=@0x7ffff63fa33c: 192, this=0x604000134cb8) at /usr/include/c++/5.1.1/bits/unordered_map.h:577
#4  transport::cql_server::connection::get_query_state (this=0x604000134bd0, stream=192) at transport/server.cc:464
#5  0x00000000007c0614 in operator() (__closure=0x6040009a7ee8) at transport/server.cc:498
#6  apply<transport::cql_server::connection::process_prepare(uint16_t, temporary_buffer<char>)::<lambda(unsigned int)> mutable::<lambda()>&> (func=...) at /home/asias/src/cloudius-systems/urchin/seastar/core/future.hh:1072
#7  smp_message_queue::async_work_item<transport::cql_server::connection::process_prepare(uint16_t, temporary_buffer<char>)::<lambda(unsigned int)> mutable::<lambda()> >::process(void) (this=0x6040009a7ee0)
   at /home/asias/src/cloudius-systems/urchin/seastar/core/reactor.hh:502
#8  0x000000000048a740 in operator() (wi=0x6040009a7ee0, __closure=<synthetic pointer>) at core/reactor.cc:1448
#9  smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::<lambda(smp_message_queue::work_item*)> >(smp_message_queue::lf_queue &, smp_message_queue::<lambda(smp_message_queue::work_item*)>) (q=...,
   process=..., process@entry=..., this=0x60000054f840) at core/reactor.cc:1423
#10 0x00000000004a2dbe in process_incoming (this=0x60000054f840) at core/reactor.cc:1451
#11 smp::poll_queues () at core/reactor.hh:1004
#12 0x0000000000484e52 in poll_once (this=0x6050000c3000) at core/reactor.cc:1221
#13 reactor::run (this=0x6050000c3000) at core/reactor.cc:1200
#14 0x000000000049018a in smp::<lambda()>::operator()(void) const (__closure=0x6000000ae700) at core/reactor.cc:1840
#15 0x00000000004cc07e in operator() (this=<optimized out>) at /usr/include/c++/5.1.1/functional:2271
#16 posix_thread::start_routine (arg=<optimized out>) at core/posix.cc:51
#17 0x0000003f77007555 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003f76902b9d in clone () from /lib64/libc.so.6

@slivne
Copy link
Contributor

slivne commented Sep 10, 2015

@asias does it happen on boot without any request with a clean datadir - or do you need to send some kind of traffic ?

@asias
Copy link
Contributor

asias commented Sep 10, 2015

it happens when I load scylla

@pdziepak pdziepak removed their assignment Sep 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
anglerfish - scylla 0.9
Development

No branches or pull requests
3 participants
@pdziepak @asias @slivne