New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

heap-use-after-free in service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}::operator()() #4107

Closed
bhalevy opened this Issue Jan 20, 2019 · 15 comments

Comments

Projects
None yet
3 participants
@bhalevy
Copy link
Contributor

bhalevy commented Jan 20, 2019

Seen here
node3.log

logged_batch_doesnt_throw_uae_test

==19149==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000196a30 at pc 0x000005ba5b9f bp 0x7f41770a9ae0 sp 0x7f41770a9ad0
READ of size 8 at 0x602000196a30 thread T0
    #0 0x5ba5b9e in service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5ba5b9e)
    #1 0x5c990fa in seastar::apply_helper<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply({lambda()#1}&&, std::tuple<>) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5c990fa)
    #2 0x5c99181 in auto seastar::apply<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>(service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5c99181)
    #3 0x5c992a2 in seastar::future<> seastar::do_void_futurize_helper<void>::apply_tuple<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>(service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5c992a2)
    #4 0x5c5eda4 in seastar::future<> seastar::futurize<void>::apply<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>(service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5c5eda4)
    #5 0x5c0c690 in seastar::futurize<std::result_of<std::decay<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>::type ()>::type>::type seastar::async<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5c0c690)
    #6 0x5cd751b in std::_Function_handler<void (), seastar::futurize<std::result_of<std::decay<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>::type ()>::type>::type seastar::async<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5cd751b)
    #7 0x7b16db in std::function<void ()>::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x7b16db)
    #8 0xd57fe3 in seastar::thread_context::main() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xd57fe3)

0x602000196a30 is located 0 bytes inside of 16-byte region [0x602000196a30,0x602000196a40)
freed by thread T0 here:
    #0 0x7f41910376d8 in operator delete(void*, unsigned long) (/lib64/libasan.so.4+0xe16d8)
    #1 0xb03be4c in auth::auth_migration_listener::~auth_migration_listener() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xb03be4c)
    #2 0x9f6cfaa in std::default_delete<service::migration_listener>::operator()(service::migration_listener*) const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9f6cfaa)
    #3 0x9f3c8ad in std::unique_ptr<service::migration_listener, std::default_delete<service::migration_listener> >::~unique_ptr() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9f3c8ad)
    #4 0x9efab4a in auth::service::~service() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9efab4a)
    #5 0x9fbee16 in seastar::shared_ptr_count_for<auth::service>::~shared_ptr_count_for() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9fbee16)
    #6 0x9fbee85 in seastar::shared_ptr_count_for<auth::service>::~shared_ptr_count_for() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9fbee85)
    #7 0x20c4674 in seastar::shared_ptr<auth::service>::~shared_ptr() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x20c4674)
    #8 0x9d6dfea in seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}::~stop() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d6dfea)
    #9 0x9db41a4 in seastar::future<> seastar::future<>::then<seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, seastar::future<> >(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}::~then() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9db41a4)
    #10 0x9f01c9c in seastar::continuation<seastar::future<> seastar::future<>::then<seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, seastar::future<> >(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::~continuation() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9f01c9c)
    #11 0x9f01d0b in seastar::continuation<seastar::future<> seastar::future<>::then<seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, seastar::future<> >(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::~continuation() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9f01d0b)
    #12 0xa0178c8 in seastar::continuation<seastar::future<> seastar::future<>::then<seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, seastar::future<> >(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<auth::service>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::run_and_dispose() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xa0178c8)
    #13 0x486832 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x486832)
    #14 0x48aedb in seastar::reactor::run_some_tasks() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x48aedb)
    #15 0x491c8b in seastar::reactor::run() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x491c8b)
    #16 0xd110a1 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xd110a1)
    #17 0x1b3eb59 in main (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x1b3eb59)
    #18 0x7f418b4ecfe9 in __libc_start_main (/lib64/libc.so.6+0x20fe9)

previously allocated by thread T0 here:
    #0 0x7f4191036158 in operator new(unsigned long) (/lib64/libasan.so.4+0xe0158)
    #1 0xb032065 in std::_MakeUniq<auth::auth_migration_listener>::__single_object std::make_unique<auth::auth_migration_listener, auth::authorizer&>(auth::authorizer&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xb032065)
    #2 0xaf5600b in auth::service::service(auth::permissions_cache_config, cql3::query_processor&, service::migration_manager&, std::unique_ptr<auth::authorizer, std::default_delete<auth::authorizer> >, std::unique_ptr<auth::authenticator, std::default_delete<auth::authenticator> >, std::unique_ptr<auth::role_manager, std::default_delete<auth::role_manager> >) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xaf5600b)
    #3 0xaf56a7a in auth::service::service(auth::permissions_cache_config, cql3::query_processor&, service::migration_manager&, auth::service_config const&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0xaf56a7a)
    #4 0x9efaede in seastar::shared_ptr_count_for<auth::service>::shared_ptr_count_for<auth::permissions_cache_config, seastar::internal::either_sharded_or_local<cql3::query_processor>, seastar::internal::either_sharded_or_local<service::migration_manager>, auth::service_config>(auth::permissions_cache_config&&, seastar::internal::either_sharded_or_local<cql3::query_processor>&&, seastar::internal::either_sharded_or_local<service::migration_manager>&&, auth::service_config&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9efaede)
    #5 0x9e8fab3 in seastar::shared_ptr<auth::service> seastar::shared_ptr_make_helper<auth::service, false>::make<auth::permissions_cache_config, seastar::internal::either_sharded_or_local<cql3::query_processor>, seastar::internal::either_sharded_or_local<service::migration_manager>, auth::service_config>(auth::permissions_cache_config&&, seastar::internal::either_sharded_or_local<cql3::query_processor>&&, seastar::internal::either_sharded_or_local<service::migration_manager>&&, auth::service_config&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e8fab3)
    #6 0x9e17543 in seastar::shared_ptr<auth::service> seastar::make_shared<auth::service, auth::permissions_cache_config, seastar::internal::either_sharded_or_local<cql3::query_processor>, seastar::internal::either_sharded_or_local<service::migration_manager>, auth::service_config>(auth::permissions_cache_config&&, seastar::internal::either_sharded_or_local<cql3::query_processor>&&, seastar::internal::either_sharded_or_local<service::migration_manager>&&, auth::service_config&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e17543)
    #7 0x9da6670 in seastar::shared_ptr<auth::service> seastar::sharded<auth::service>::create_local_service<auth::permissions_cache_config, seastar::internal::either_sharded_or_local<cql3::query_processor>, seastar::internal::either_sharded_or_local<service::migration_manager>, auth::service_config>(auth::permissions_cache_config&&, seastar::internal::either_sharded_or_local<cql3::query_processor>&&, seastar::internal::either_sharded_or_local<service::migration_manager>&&, auth::service_config&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9da6670)
    #8 0x9d65faf in seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config)#1}::operator()(auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config) const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d65faf)
    #9 0x9d66926 in seastar::apply_helper<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config)#1}, std::tuple<auth::permissions_cache_config, seastar::sharded<cql3::query_processor>&, seastar::sharded<service::migration_manager>&, auth::service_config>&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul> >::apply({lambda()#1}&&, seastar::sharded<cql3::query_processor>&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d66926)
    #10 0x9d66b0b in auto seastar::apply<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config)#1}, auth::permissions_cache_config, seastar::sharded<cql3::query_processor>&, seastar::sharded<service::migration_manager>&, auth::service_config>(seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config)#1}&&, std::tuple<auth::permissions_cache_config, seastar::sharded<cql3::query_processor>&, seastar::sharded<service::migration_manager>&, auth::service_config>&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d66b0b)
    #11 0x9d65b38 in seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d65b38)
    #12 0x9e8fc5d in seastar::future<> seastar::do_void_futurize_helper<void>::apply<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>(seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e8fc5d)
    #13 0x9e1780e in seastar::future<> seastar::futurize<void>::apply<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>(seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e1780e)
    #14 0x9da6ad6 in seastar::futurize<std::result_of<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1} ()>::type>::type seastar::smp::submit_to<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>(unsigned int, seastar::futurize&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9da6ad6)
    #15 0x9d65912 in seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}::operator()(unsigned int) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d65912)
    #16 0x9e90bc2 in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}, unsigned int>(seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}&&, unsigned int&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e90bc2)
    #17 0x9e18a79 in auto seastar::futurize_apply<seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}, unsigned int>(seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}&&, unsigned int&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e18a79)
    #18 0x9e19b83 in seastar::future<> seastar::parallel_for_each<boost::range_detail::integer_iterator<unsigned int>, seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}>(boost::range_detail::integer_iterator<unsigned int>, seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}, seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9e19b83)
    #19 0x9da6f32 in seastar::future<> seastar::parallel_for_each<boost::integer_range<unsigned int>, seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}>(boost::integer_range<unsigned int>&&, seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&)::{lambda(unsigned int)#1}&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9da6f32)
    #20 0x9d68a8a in seastar::future<> seastar::sharded<auth::service>::start<auth::permissions_cache_config, std::reference_wrapper<seastar::sharded<cql3::query_processor> >, std::reference_wrapper<seastar::sharded<service::migration_manager> >, auth::service_config>(auth::permissions_cache_config&&, std::reference_wrapper<seastar::sharded<cql3::query_processor> >&&, std::reference_wrapper<seastar::sharded<service::migration_manager> >&&, auth::service_config&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9d68a8a)
    #21 0x943c8c9 in service::storage_service::join_token_ring(int) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x943c8c9)
    #22 0x9486d58 in service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x9486d58)
    #23 0x97b1c08 in seastar::apply_helper<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply({lambda()#1}&&, std::tuple<>) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x97b1c08)
    #24 0x97b1c8f in auto seastar::apply<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>(service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x97b1c8f)
    #25 0x97b1db0 in seastar::future<> seastar::do_void_futurize_helper<void>::apply_tuple<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>(service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x97b1db0)
    #26 0x96c32e0 in seastar::future<> seastar::futurize<void>::apply<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>(service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}&&, std::tuple<>&&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x96c32e0)
    #27 0x95e7c8c in seastar::futurize<std::result_of<std::decay<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>::type ()>::type>::type seastar::async<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x95e7c8c)
    #28 0x991cd27 in std::_Function_handler<void (), seastar::futurize<std::result_of<std::decay<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>::type ()>::type>::type seastar::async<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<service::storage_service::init_server(int, seastar::bool_class<service::bind_messaging_port_tag>)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x991cd27)
    #29 0x7b16db in std::function<void ()>::operator()() const (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x7b16db)

SUMMARY: AddressSanitizer: heap-use-after-free (/data2/jenkins/.dtest/dtest-qEoHiG/test/node3/bin/scylla+0x5ba5b9e) in service::migration_manager::notify_create_column_family(seastar::lw_shared_ptr<schema const> const&)::{lambda()#1}::operator()() const
Shadow bytes around the buggy address:
  0x0c048002acf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002ad00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002ad10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002ad20: fa fa fa fa fa fa fd fa fa fa fa fa fa fa fa fa
  0x0c048002ad30: fa fa fd fa fa fa fd fd fa fa fa fa fa fa fd fa
=>0x0c048002ad40: fa fa fd fa fa fa[fd]fd fa fa fa fa fa fa fa fa
  0x0c048002ad50: fa fa fa fa fa fa fd fd fa fa fa fa fa fa fa fa
  0x0c048002ad60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002ad70: fa fa fa fa fa fa fa fa fa fa 04 fa fa fa fa fa
  0x0c048002ad80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002ad90: fa fa fd fd fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==19149==ABORTING
@slivne

This comment has been minimized.

Copy link
Contributor

slivne commented Jan 20, 2019

commit id ?

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

bhalevy commented Jan 20, 2019

commit id ?

e212d37

@slivne

This comment has been minimized.

@slivne slivne added the bug label Jan 20, 2019

@slivne slivne added this to the 3.1 milestone Jan 20, 2019

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 21, 2019

I can't reproduce locally (so far) and the above trace doesn't have line numbers. How can I get the binary?

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 21, 2019

The problem seems to be that the auth service doesn't unregister its listener from the migration_manager when destroyed. Looks like an easy fix, however I can't test locally as I can't reproduce.

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

bhalevy commented Jan 21, 2019

Unfortunately I don't think we keep the binaries on Jenkins runs.
Can you try to reproduce this with a unit test?
Regardless, if there's an obvious issue with the auth service as you described above, let's get this fixed and keep monitoring the dtests in debug mode.

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 21, 2019

I managed to decipher the backtrace without line numbers and find the issue, so I don't need the binary after all.

avikivity added a commit that referenced this issue Jan 21, 2019

auth/service: unregister migration listener on stop()
Otherwise any event that triggers notification to this listener would
trigger a heap-use-after-free.

Refs: #4107

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <b6bbd609371a2312aed7571b05119d59c7d103d7.1548067626.git.bdenes@scylladb.com>
@slivne

This comment has been minimized.

Copy link
Contributor

slivne commented Jan 22, 2019

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 22, 2019

Right, I don't know how I missed it (I did check the build artifacts).

@slivne

This comment has been minimized.

Copy link
Contributor

slivne commented Jan 22, 2019

Why Refs and not Fixes ?

@slivne

This comment has been minimized.

Copy link
Contributor

slivne commented Jan 22, 2019

do we need to backport to 3.0 ?

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 22, 2019

I could not reproduce locally so I wanted to close this when we see the test passing.

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 22, 2019

All branches are affected that have Jesse's auth changes (RBA).
Based on the commit which introduced the listener (ba6a41d) both 2.3 and 3.0 are affected.

@denesb

This comment has been minimized.

Copy link
Contributor

denesb commented Jan 22, 2019

I think we should backport. This is a very low-risk backport and it avoids potential crash on shutdown.

avikivity added a commit that referenced this issue Jan 22, 2019

auth/service: unregister migration listener on stop()
Otherwise any event that triggers notification to this listener would
trigger a heap-use-after-free.

Refs: #4107

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <b6bbd609371a2312aed7571b05119d59c7d103d7.1548067626.git.bdenes@scylladb.com>
(cherry picked from commit f229dff)

avikivity added a commit that referenced this issue Jan 22, 2019

auth/service: unregister migration listener on stop()
Otherwise any event that triggers notification to this listener would
trigger a heap-use-after-free.

Refs: #4107

Signed-off-by: Botond Dénes <bdenes@scylladb.com>
Message-Id: <b6bbd609371a2312aed7571b05119d59c7d103d7.1548067626.git.bdenes@scylladb.com>
(cherry picked from commit f229dff)
@slivne

This comment has been minimized.

Copy link
Contributor

slivne commented Jan 22, 2019

closing this issue - its supposed to be fixed (can't reproduce locally)

@slivne slivne closed this Jan 22, 2019

@bhalevy bhalevy added the dtest label Jan 31, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment