Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

inner variable read/write conflict when run executor in parallel. #4844

Closed
Shylock-Hg opened this issue Nov 9, 2022 · 0 comments · Fixed by #4913
Closed

inner variable read/write conflict when run executor in parallel. #4844

Shylock-Hg opened this issue Nov 9, 2022 · 0 comments · Fixed by #4913
Assignees
@jievince
Labels
severity/minor Severity of bug type/bug Type: something is unexpected
Milestone
v3.4.0

Comments

@Shylock-Hg
Copy link
Contributor

Please check the FAQ documentation before raising an issue

Describe the bug (required)
Wen set max_job_size > 1, the inner variable write/read will lead to condition race.

E20221109 18:01:45.060799 2658775 QueryInstance.cpp:139] SyntaxError: syntax error near `M MATCH('
E20221109 18:02:03.555559 2658775 QueryInstance.cpp:139] SyntaxError: syntax error near `M MATCH('
E20221109 18:02:18.386580 2658775 QueryInstance.cpp:139] SemanticError: `product': Unknown tag
E20221109 18:02:30.745103 2658775 QueryInstance.cpp:139] SemanticError: `e': Redefined alias
=================================================================
==2659462==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000114e80 at pc 0x0000022b7655 bp 0x7fede8338e00 sp 0x7fede8338df0
READ of size 1 at 0x61d000114e80 thread T16 (executor-pri3-4)
    #0 0x22b7654 in nebula::graph::Result::Core::Core(nebula::graph::Result::Core&&) src/graph/context/Result.h:85
    #1 0x22b8225 in nebula::graph::Result::Result(nebula::graph::Result&&) src/graph/context/Result.h:20
    #2 0x22b8225 in void __gnu_cxx::new_allocator<nebula::graph::Result>::construct<nebula::graph::Result, nebula::graph::Result>(nebula::graph::Result*, nebula::graph::Result&&) /usr/include/c++/9/ext/new_allocator.h:146
    #3 0x22b8225 in void std::allocator_traits<std::allocator<nebula::graph::Result> >::construct<nebula::graph::Result, nebula::graph::Result>(std::allocator<nebula::graph::Result>&, nebula::graph::Result*, nebula::graph::Result&&) /usr/include/c++/9/bits/alloc_traits.h:483
    #4 0x22b8225 in void std::__relocate_object_a<nebula::graph::Result, nebula::graph::Result, std::allocator<nebula::graph::Result> >(nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result>&) /usr/include/c++/9/bits/stl_uninitialized.h:912
    #5 0x22b8225 in nebula::graph::Result* std::__relocate_a_1<nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result> >(nebula::graph::Result*, nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result>&) /usr/include/c++/9/bits/stl_uninitialized.h:950
    #6 0x22b8225 in nebula::graph::Result* std::__relocate_a<nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result> >(nebula::graph::Result*, nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result>&) /usr/include/c++/9/bits/stl_uninitialized.h:964
    #7 0x22b8225 in std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_S_do_relocate(nebula::graph::Result*, nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result>&, std::integral_constant<bool, true>) /usr/include/c++/9/bits/stl_vector.h:453
    #8 0x22b8225 in std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_S_relocate(nebula::graph::Result*, nebula::graph::Result*, nebula::graph::Result*, std::allocator<nebula::graph::Result>&) /usr/include/c++/9/bits/stl_vector.h:466
    #9 0x22b8225 in void std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_M_realloc_insert<nebula::graph::Result>(__gnu_cxx::__normal_iterator<nebul a::graph::Result*, std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> > >, nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:461
    #10 0x22b8bb3 in nebula::graph::Result& std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::emplace_back<nebula::graph::Result>(nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:121
    #11 0x22b5eaf in nebula::graph::ExecutionContext::setResult(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::graph::Result&&) src/graph/context/ExecutionContext.cpp:25
    #12 0x22b5eaf in nebula::graph::ExecutionContext::setValue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value&&) src/graph/context/ExecutionContext.cpp:20
    #13 0x22b2f7f in nebula::graph::QueryExpressionContext::setVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value) src/graph/context/QueryExpressionContext.cpp:106
    #14 0x514bd0d in nebula::ListComprehensionExpression::eval(nebula::ExpressionContext&) src/common/expression/ListComprehensionExpression.cpp:34
    #15 0x1ce6669 in nebula::graph::ProjectExecutor::handleJob(unsigned long, unsigned long, nebula::graph::Iterator*) src/graph/executor/query/ProjectExecutor.cpp:57
    #16 0x1cebec7 in nebula::graph::ProjectExecutor::execute() src/graph/executor/query/ProjectExecutor.cpp:21
    #17 0x228ca85 in nebula::graph::AsyncMsgNotifyBasedScheduler::execute(nebula::graph::Executor*) const src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:221
    #18 0x228f2d9 in operator()<std::vector<nebula::Status> > src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:158
    #19 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:101
    #20 0x228f984 in wrapResult<folly::futures::detail::wrapInvoke(folly::Try<T>&&, F&&) [with T = std::vector<nebula::Status>; F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>]::<lambda()> > build/third-party/install/include/folly/futures/Future-inl.h:73
    #21 0x228f984 in wrapInvoke<std::vector<nebula::Status>, nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)> > build/third-party/install/include/folly/futures/Future-inl.h:108
    #22 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:991
    #23 0x228f984 in invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:144
    #24 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:149
    #25 0x228f984 in makeTryWithNoUnwrap<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:257
    #26 0x228f984 in makeTryWith<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:279
    #27 0x228f984 in tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:149
    #28 0x228f984 in tryInvoke<folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>, folly::futures::detail::CoreCallbackState<nebula::Status, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)> >, std::vector<nebula::Status> > build/third-party/install/include/folly/futures/Future-inl.h:349
    #29 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:436
    #30 0x228f984 in operator() build/third-party/install/include/folly/futures/detail/Core.h:637
    #31 0x228f984 in callSmall<folly::futures::detail::Core<T>::setCallback(F&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation) [with F = folly::futures::detail::FutureBase<T>::thenImplementation(F&&, R, folly::futures::detail::InlineContinuation) [with F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; R = folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; T = std::vector<nebula::Status>]::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> > build/third-party/install/include/folly/Function.h:371
    #32 0x565e93b  (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x565e93b)
    #33 0x5411dd6 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5411dd6)
    #34 0x5551627 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5551627)
    #35 0x555372d in apache::thrift::concurrency::PthreadThread::threadMain(void*) (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x555372d)
    #36 0x7fedfbd53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #37 0x7fedfbc78132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

0x61d000114e80 is located 0 bytes inside of 2048-byte region [0x61d000114e80,0x61d000115680)
freed by thread T13 (executor-pri3-1) here:
    #0 0x7fedfbfef51f in operator delete(void*) ../../../../src/libsanitizer/asan/asan_new_delete.cc:165
    #1 0x22b8648 in __gnu_cxx::new_allocator<nebula::graph::Result>::deallocate(nebula::graph::Result*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
    #2 0x22b8648 in std::allocator_traits<std::allocator<nebula::graph::Result> >::deallocate(std::allocator<nebula::graph::Result>&, nebula::graph::Result*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:469
    #3 0x22b8648 in std::_Vector_base<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_M_deallocate(nebula::graph::Result*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351
    #4 0x22b8648 in void std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_M_realloc_insert<nebula::graph::Result>(__gnu_cxx::__normal_iterator<nebula::graph::Result*, std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> > >, nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:500
    #5 0x22b8bb3 in nebula::graph::Result& std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::emplace_back<nebula::graph::Result>(nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:121
    #6 0x22b5eaf in nebula::graph::ExecutionContext::setResult(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::graph::Result&&) src/graph/context/ExecutionContext.cpp:25
    #7 0x22b5eaf in nebula::graph::ExecutionContext::setValue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value&&) src/graph/context/ExecutionContext.cpp:20
    #8 0x22b2f7f in nebula::graph::QueryExpressionContext::setVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value) src/graph/context/QueryExpressionContext.cpp:106
    #9 0x514bd0d in nebula::ListComprehensionExpression::eval(nebula::ExpressionContext&) src/common/expression/ListComprehensionExpression.cpp:34
    #10 0x1ce6669 in nebula::graph::ProjectExecutor::handleJob(unsigned long, unsigned long, nebula::graph::Iterator*) src/graph/executor/query/ProjectExecutor.cpp:57
    #11 0x1cebec7 in nebula::graph::ProjectExecutor::execute() src/graph/executor/query/ProjectExecutor.cpp:21
    #12 0x228ca85 in nebula::graph::AsyncMsgNotifyBasedScheduler::execute(nebula::graph::Executor*) const src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:221
    #13 0x228f2d9 in operator()<std::vector<nebula::Status> > src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:158
    #14 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:101
    #15 0x228f984 in wrapResult<folly::futures::detail::wrapInvoke(folly::Try<T>&&, F&&) [with T = std::vector<nebula::Status>; F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>]::<lambda()> > build/third-party/install/include/folly/futures/Future-inl.h:73
    #16 0x228f984 in wrapInvoke<std::vector<nebula::Status>, nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)> > build/third-party/install/include/folly/futures/Future-inl.h:108
    #17 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:991
    #18 0x228f984 in invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:144
    #19 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:149
    #20 0x228f984 in makeTryWithNoUnwrap<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:257
    #21 0x228f984 in makeTryWith<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:279
    #22 0x228f984 in tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:149
    #23 0x228f984 in tryInvoke<folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>, folly::futures::detail::CoreCallbackState<nebula::Status, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)> >, std::vector<nebula::Status> > build/third-party/install/include/folly/futures/Future-inl.h:349
    #24 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:436
    #25 0x228f984 in operator() build/third-party/install/include/folly/futures/detail/Core.h:637
    #26 0x228f984 in callSmall<folly::futures::detail::Core<T>::setCallback(F&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation) [with F = folly::futures::detail::FutureBase<T>::thenImplementation(F&&, R, folly::futures::detail::InlineContinuation) [with F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; R = folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; T = std::vector<nebula::Status>]::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> > build/third-party/install/include/folly/Function.h:371
    #27 0x565e93b  (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x565e93b)
    #28 0x5411dd6 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5411dd6)
    #29 0x5551627 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5551627)
    #30 0x555372d in apache::thrift::concurrency::PthreadThread::threadMain(void*) (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x555372d)
    #31 0x7fedfbd53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #32 0x7fedfbc78132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

previously allocated by thread T13 (executor-pri3-1) here:
    #0 0x7fedfbfee587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
    #1 0x22b8142 in __gnu_cxx::new_allocator<nebula::graph::Result>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
    #2 0x22b8142 in std::allocator_traits<std::allocator<nebula::graph::Result> >::allocate(std::allocator<nebula::graph::Result>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:443
    #3 0x22b8142 in std::_Vector_base<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
    #4 0x22b8142 in void std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::_M_realloc_insert<nebula::graph::Result>(__gnu_cxx::__normal_iterator<nebula::graph::Result*, std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> > >, nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:440
    #5 0x22b8bb3 in nebula::graph::Result& std::vector<nebula::graph::Result, std::allocator<nebula::graph::Result> >::emplace_back<nebula::graph::Result>(nebula::graph::Result&&) /usr/include/c++/9/bits/vector.tcc:121
    #6 0x22b5eaf in nebula::graph::ExecutionContext::setResult(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::graph::Result&&) src/graph/context/ExecutionContext.cpp:25
    #7 0x22b5eaf in nebula::graph::ExecutionContext::setValue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value&&) src/graph/context/ExecutionContext.cpp:20
    #8 0x22b2f7f in nebula::graph::QueryExpressionContext::setVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nebula::Value) src/graph/context/QueryExpressionContext.cpp:106
    #9 0x514bd0d in nebula::ListComprehensionExpression::eval(nebula::ExpressionContext&) src/common/expression/ListComprehensionExpression.cpp:34
    #10 0x1ce6669 in nebula::graph::ProjectExecutor::handleJob(unsigned long, unsigned long, nebula::graph::Iterator*) src/graph/executor/query/ProjectExecutor.cpp:57
    #11 0x1cebec7 in nebula::graph::ProjectExecutor::execute() src/graph/executor/query/ProjectExecutor.cpp:21
    #12 0x228ca85 in nebula::graph::AsyncMsgNotifyBasedScheduler::execute(nebula::graph::Executor*) const src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:221
    #13 0x228f2d9 in operator()<std::vector<nebula::Status> > src/graph/scheduler/AsyncMsgNotifyBasedScheduler.cpp:158
    #14 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:101
    #15 0x228f984 in wrapResult<folly::futures::detail::wrapInvoke(folly::Try<T>&&, F&&) [with T = std::vector<nebula::Status>; F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>]::<lambda()> > build/third-party/install/include/folly/futures/Future-inl.h:73
    #16 0x228f984 in wrapInvoke<std::vector<nebula::Status>, nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)> > build/third-party/install/include/folly/futures/Future-inl.h:108
    #17 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:991
    #18 0x228f984 in invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:144
    #19 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:149
    #20 0x228f984 in makeTryWithNoUnwrap<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:257
    #21 0x228f984 in makeTryWith<folly::futures::detail::CoreCallbackState<T, F>::tryInvoke(Args&& ...) [with Args = {folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > >}; T = nebula::Status; F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>]::<lambda()> > build/third-party/install/include/folly/Try-inl.h:279
    #22 0x228f984 in tryInvoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<std::vector<nebula::Status, std::allocator<nebula::Status> > > > build/third-party/install/include/folly/futures/Future-inl.h:149
    #23 0x228f984 in tryInvoke<folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>, folly::futures::detail::CoreCallbackState<nebula::Status, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)> >, std::vector<nebula::Status> > build/third-party/install/include/folly/futures/Future-inl.h:349
    #24 0x228f984 in operator() build/third-party/install/include/folly/futures/Future-inl.h:436
    #25 0x228f984 in operator() build/third-party/install/include/folly/futures/detail/Core.h:637
    #26 0x228f984 in callSmall<folly::futures::detail::Core<T>::setCallback(F&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation) [with F = folly::futures::detail::FutureBase<T>::thenImplementation(F&&, R, folly::futures::detail::InlineContinuation) [with F = folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; R = folly::futures::detail::tryExecutorCallableResult<std::vector<nebula::Status>, folly::Future<T>::thenValue(F&&) && [with F = nebula::graph::AsyncMsgNotifyBasedScheduler::runExecutor(std::vector<folly::Future<nebula::Status> >&&, nebula::graph::Executor*, folly::Executor*) const::<lambda(auto:182&&)>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>, void>; T = std::vector<nebula::Status>]::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<std::vector<nebula::Status> >&&)>; T = std::vector<nebula::Status>]::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> > build/third-party/install/include/folly/Function.h:371
    #27 0x565e93b  (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x565e93b)
    #28 0x5411dd6 in virtual thunk to apache::thrift::concurrency::FunctionRunner::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5411dd6)
    #29 0x5551627 in apache::thrift::concurrency::ThreadManager::Impl::Worker::run() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5551627)
    #30 0x555372d in apache::thrift::concurrency::PthreadThread::threadMain(void*) (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x555372d)
    #31 0x7fedfbd53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #32 0x7fedfbc78132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

Thread T16 (executor-pri3-4) created by T0 here:
    #0 0x7fedfbf19815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x5553594 in apache::thrift::concurrency::PthreadThread::start() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5553594)
    #2 0x5540fcc in apache::thrift::concurrency::ThreadManager::Impl::addWorker(unsigned long) (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5540fcc)
    #3 0x55491cd in apache::thrift::concurrency::PriorityThreadManager::PriorityImpl::start() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x55491cd)
    #4 0x14b375d in nebula::graph::GraphServer::start() src/graph/service/GraphServer.cpp:31

Thread T13 (executor-pri3-1) created by T0 here:
    #0 0x7fedfbf19815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x5553594 in apache::thrift::concurrency::PthreadThread::start() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5553594)
    #2 0x5540fcc in apache::thrift::concurrency::ThreadManager::Impl::addWorker(unsigned long) (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x5540fcc)
    #3 0x55491cd in apache::thrift::concurrency::PriorityThreadManager::PriorityImpl::start() (/home/sunny.liu/nebula-ent-3.1.2-21103/bin/nebula-graphd+0x55491cd)
    #4 0x14b375d in nebula::graph::GraphServer::start() src/graph/service/GraphServer.cpp:31

SUMMARY: AddressSanitizer: heap-use-after-free src/graph/context/Result.h:85 in nebula::graph::Result::Core::Core(nebula::graph::Result::Core&&)
Shadow bytes around the buggy address:
  0x0c3a8001a980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001a990: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8001a9a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8001a9b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3a8001a9c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c3a8001a9d0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001a9e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001a9f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001aa00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001aa10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3a8001aa20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==2659462==ABORTING

Your Environments (required)

  • OS: uname -a
  • Compiler: g++ --version or clang++ --version
  • CPU: lscpu
  • Commit id (e.g. a3ffc7d8)

How To Reproduce(required)

Steps to reproduce the behavior:

  1. Step 1
  2. Step 2
  3. Step 3

Expected behavior

Additional context
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jievince jievince

Labels
severity/minor Severity of bug type/bug Type: something is unexpected
Projects
None yet
Milestone
v3.4.0
Development

Successfully merging a pull request may close this issue.

make evaluation of inner var expr thread-safe jievince/nebula
5 participants
@yixinglu @Shylock-Hg @jievince @Sophie-Xie @jinyingsunny