RateLimiter crash

[Shylock-Hg]

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

(lldb) target create "./bin/nebula-storaged" --core "core.37901"
Core file '/home/shylock.huang/nebula-feature-node-polymorphsim/build/server_2021-09-07T10-53-01/core.37901' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'nebula-storaged', stop reason = signal SIGSEGV
  * frame #0: 0x00000000025a0209 nebula-storaged`std::__detail::_Equal_helper<std::pair<int, int>, std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> >, std::__detail::_Select1st, std::equal_to<std::pair<int, int> >, unsigned long, true>::_S_equals(__eq=0x00007f678a240ea0, __extract=0x00007f678a240ea0, __k=0x00007f678d7eb128, __c=15112025674922533546, __n=0x0000000000000000) at hashtable_policy.h:1450
    frame #1: 0x00000000025a0188 nebula-storaged`std::__detail::_Hashtable_base<std::pair<int, int>, std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> >, std::__detail::_Select1st, std::equal_to<std::pair<int, int> >, std::hash<std::pair<int, int> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<true, false, true> >::_M_equals(this=0x00007f678a240ea0, __k=0x00007f678d7eb128, __c=15112025674922533546, __n=0x0000000000000000) const at hashtable_policy.h:1834
    frame #2: 0x00000000025a00a6 nebula-storaged`std::_Hashtable<std::pair<int, int>, std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> >, std::allocator<std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> > >, std::__detail::_Select1st, std::equal_to<std::pair<int, int> >, std::hash<std::pair<int, int> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node(this=0x00007f678a240ea0, __n=1, __k=0x00007f678d7eb128, __code=15112025674922533546) const at hashtable.h:1545
    frame #3: 0x000000000259ff11 nebula-storaged`std::_Hashtable<std::pair<int, int>, std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> >, std::allocator<std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> > >, std::__detail::_Select1st, std::equal_to<std::pair<int, int> >, std::hash<std::pair<int, int> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_node(this=0x00007f678a240ea0, __bkt=1, __key=0x00007f678d7eb128, __c=15112025674922533546) const at hashtable.h:655
    frame #4: 0x000000000259fe2c nebula-storaged`std::_Hashtable<std::pair<int, int>, std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> >, std::allocator<std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> > >, std::__detail::_Select1st, std::equal_to<std::pair<int, int> >, std::hash<std::pair<int, int> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find(this=0x00007f678a240ea0, __k=0x00007f678d7eb128) at hashtable.h:1419
    frame #5: 0x000000000259fc3b nebula-storaged`std::unordered_map<std::pair<int, int>, folly::BasicTokenBucket<std::chrono::_V2::steady_clock>, std::hash<std::pair<int, int> >, std::equal_to<std::pair<int, int> >, std::allocator<std::pair<std::pair<int, int> const, folly::BasicTokenBucket<std::chrono::_V2::steady_clock> > > >::find(this=0x00007f678a240ea0, __x=0x00007f678d7eb128) at unordered_map.h:921
    frame #6: 0x000000000259f3d5 nebula-storaged`nebula::kvstore::RateLimiter::consume(this=0x00007f678a240ea0, spaceId=248, partId=1, toConsume=212) at RateLimiter.h:44
    frame #7: 0x000000000259eef0 nebula-storaged`nebula::storage::RebuildIndexTask::writeData(this=0x00007f67894d2080, space=248, part=1, data=<unavailable>, batchSize=212) at RebuildIndexTask.cpp:225
    frame #8: 0x00000000025b4b53 nebula-storaged`nebula::storage::RebuildTagIndexTask::buildIndexGlobal(this=0x00007f67894d2080, space=248, part=1, items=size=1) at RebuildTagIndexTask.cpp:144
    frame #9: 0x000000000259bf8e nebula-storaged`nebula::storage::RebuildIndexTask::invoke(this=0x00007f67894d2080, space=248, part=1, items=size=1) at RebuildIndexTask.cpp:100
    frame #10: 0x00000000025aeacc nebula-storaged`nebula::cpp2::ErrorCode std::__invoke_impl<nebula::cpp2::ErrorCode, nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>((null)=__invoke_memfun_deref @ 0x00007f678d7ec0f0, __f=0x00007f67a6c1b8c0, __t=0x00007f67a6c1b8f0, __args=0x00007f67a6c1b8ec, __args=0x00007f67a6c1b8e8, __args=size=1)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&) at invoke.h:73
    frame #11: 0x00000000025ae8df nebula-storaged`std::__invoke_result<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>::type std::__invoke<nebula::cpp2::ErrorCode (__fn=0x00007f67a6c1b8c0, __args=0x00007f67a6c1b8f0, __args=0x00007f67a6c1b8ec, __args=0x00007f67a6c1b8e8, __args=size=1)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>(nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&) at invoke.h:95
    frame #12: 0x00000000025ae857 nebula-storaged`nebula::cpp2::ErrorCode std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::__call<nebula::cpp2::ErrorCode, 0ul, 1ul, 2ul, 3ul>(this=0x00007f67a6c1b8c0, __args=0x00007f678d7ec1d8, (null)=_Index_tuple<0, 1, 2, 3> @ 0x00007f678d7ec1b0) at functional:400
    frame #13: 0x00000000025ae765 nebula-storaged`nebula::cpp2::ErrorCode std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::operator(this=0x00007f67a6c1b8c0)<nebula::cpp2::ErrorCode>() at functional:482
    frame #14: 0x00000000025ae1d6 nebula-storaged`std::_Function_handler<nebula::cpp2::ErrorCode (), std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)> >::_M_invoke(__functor=0x00007f678d7ec4d8) at std_function.h:285
    frame #15: 0x000000000257351e nebula-storaged`std::function<nebula::cpp2::ErrorCode ()>::operator(this=0x00007f678d7ec4d8)() const at std_function.h:690
    frame #16: 0x0000000002572fae nebula-storaged`nebula::storage::AdminSubTask::invoke(this=0x00007f678d7ec4d8) at AdminTask.h:29
    frame #17: 0x0000000002571318 nebula-storaged`nebula::storage::AdminTaskManager::runSubTask(this=0x000000000522dc00, handle=(first = 254, second = 0)) at AdminTaskManager.cpp:163
    frame #18: 0x0000000002588194 nebula-storaged`void std::__invoke_impl<void, void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>((null)=__invoke_memfun_deref @ 0x00007f678d7ec660, __f=0x00007f67a6c00240, __t=0x00007f67a6c00258, __args=0x00007f67a6c00250)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&) at invoke.h:73
    frame #19: 0x000000000258806d nebula-storaged`std::__invoke_result<void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>::type std::__invoke<void (__fn=0x00007f67a6c00240, __args=0x00007f67a6c00258, __args=0x00007f67a6c00250)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>(void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&) at invoke.h:95
    frame #20: 0x0000000002588017 nebula-storaged`void std::_Bind<void (nebula::storage::AdminTaskManager::* (nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::__call<void, 0ul, 1ul>(this=0x00007f67a6c00240, __args=0x00007f678d7ec708, (null)=_Index_tuple<0, 1> @ 0x00007f678d7ec6e0) at functional:400
    frame #21: 0x0000000002587f95 nebula-storaged`void std::_Bind<void (nebula::storage::AdminTaskManager::* (nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::operator(this=0x00007f67a6c00240)<void>() at functional:482
    frame #22: 0x0000000002587d00 nebula-storaged`void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (p=0x00007f67a6c04260))(std::pair<int, int>)> >(folly::detail::function::Data&) at Function.h:385
    frame #23: 0x0000000004185dcd nebula-storaged`folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) + 173
    frame #24: 0x000000000417a33f nebula-storaged`___lldb_unnamed_symbol3962$$nebula-storaged + 63
    frame #25: 0x00000000041ea67e nebula-storaged`bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&) + 158
    frame #26: 0x00000000041eb9dd nebula-storaged`non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short) + 45
    frame #27: 0x00000000042af263 nebula-storaged`___lldb_unnamed_symbol4263$$nebula-storaged + 1363
    frame #28: 0x00000000042af937 nebula-storaged`event_base_loop + 1287
    frame #29: 0x00000000041e67a6 nebula-storaged`folly::EventBase::loopBody(int, bool) + 2134
    frame #30: 0x00000000041e6c3f nebula-storaged`folly::EventBase::loop() + 47
    frame #31: 0x00000000041e8d44 nebula-storaged`folly::EventBase::loopForever() + 20
    frame #32: 0x000000000417acc9 nebula-storaged`folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) + 793
    frame #33: 0x0000000004187257 nebula-storaged`void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::* (folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) + 71
    frame #34: 0x000000000237f433 nebula-storaged`folly::detail::function::FunctionTraits<void ()>::operator(this=0x00007f67a6c38160)() at Function.h:400
    frame #35: 0x00000000023da6fa nebula-storaged`folly::NamedThreadFactory::newThread(this=0x00007f67a6c38160)>&&)::'lambda'()::operator()() at NamedThreadFactory.h:40
    frame #36: 0x00000000023da6ab nebula-storaged`void std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()>((null)=__invoke_other @ 0x00007f678d7ecc60, __f=0x00007f67a6c38160)>&&)::'lambda'()&&) at invoke.h:60
    frame #37: 0x00000000023da65b nebula-storaged`std::__invoke_result<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()>::type std::__invoke<folly::NamedThreadFactory::newThread(__fn=0x00007f67a6c38160)>&&)::'lambda'()>(folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()&&) at invoke.h:95
    frame #38: 0x00000000023da633 nebula-storaged`void std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> >::_M_invoke<0ul>(this=0x00007f67a6c38160, (null)=_Index_tuple<0> @ 0x00007f678d7ecca0) at thread:244
    frame #39: 0x00000000023da603 nebula-storaged`std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> >::operator(this=0x00007f67a6c38160)() at thread:251
    frame #40: 0x00000000023da4a7 nebula-storaged`std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> > >::_M_run(this=0x00007f67a6c38150) at thread:195
    frame #41: 0x00000000046fc0f0 nebula-storaged`execute_native_thread_routine + 16
    frame #42: 0x00007f67d5c2458e libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:486
    frame #43: 0x00007f67d5b53513 libc.so.6`__GI___clone at clone.S:95

Your Environments (must be provided)

• OS: uname -a
• Compliler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id (e.g. a3ffc7d8)

How To Reproduce(must be provided)

Steps to reproduce the behavior:

Run tck

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.