[BUG] another coredump for version 3.4.6-1 #9302

daoxian · 2019-06-20T13:11:38Z

3.4.6-1 , CentOS-7.4, Single mode on metal machine.

Failed to read a valid object file image from memory.
Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000001ef472c in malloc_mutex_lock (mutex=0x40e3740 <je_arenas_lock>, tsdn=0x0) at include/jemalloc/internal/mutex.h:205
205     include/jemalloc/internal/mutex.h: No such file or directory.
(gdb) bt
#0  0x0000000001ef472c in malloc_mutex_lock (mutex=0x40e3740 <je_arenas_lock>, tsdn=0x0) at include/jemalloc/internal/mutex.h:205
#1  je_arena_init (extent_hooks=<optimized out>, ind=0, tsdn=0x0) at src/jemalloc.c:360
#2  malloc_init_hard_a0_locked () at src/jemalloc.c:1270
Cannot access memory at address 0x7f3ca2d973a8

The text was updated successfully, but these errors were encountered:

daoxian · 2019-06-20T13:33:17Z

Here's yet another coredump:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
bt#0  getCollectionIdLocal (name=..., this=0x7f3a0cc1f880) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7f3a0cc1f880) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7f3a0cc1f880, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7f3a49a7c800, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7f3a0e626620, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f39658a59c0, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f3a4a59de48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7f3a4a59de40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f45c70cee00, this=0x7f395d3c1970)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7f45c70cee00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7f3b76004540) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7f3b76004540) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7f3b76004540) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7f3b6ed34b30) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7f3a4a5a0ae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

daoxian · 2019-06-20T13:36:23Z

And another:

#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
#1  extent_arena_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:49
#2  je_large_dalloc (tsdn=0x7f5b6570af98, extent=0x0) at src/large.c:347
#3  0x00000000010b6ef3 in deallocate (this=0x7f5c6ed0c968, __p=<optimized out>) at /usr/include/c++/6.4.0/ext/new_allocator.h:110
#4  deallocate (__a=..., __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/alloc_traits.h:462
#5  _M_deallocate (this=0x7f5c6ed0c968, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:178
#6  std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > >::_M_range_insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (this=0x7f5c6ed0c968, __position=...,
    __first=..., __last=...) at /usr/include/c++/6.4.0/bits/vector.tcc:685
#7  0x00000000010b7289 in _M_insert_dispatch<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __pos=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1375
#8  insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __position=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1100
#9  arangodb::pregel::ArrayInCache<arangodb::pregel::SenderMessage<double> >::mergeCache (this=0x7f5c3e609d00, config=..., otherCache=0x7f5c3e643580)
    at /work/ArangoDB/arangod/Pregel/IncomingCache.cpp:151
#10 0x0000000000ea0eb7 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_processVertices (this=0x7f5cc5072b10,
    threadId=<optimized out>, vertexIterator=...) at /work/ArangoDB/arangod/Pregel/Worker.cpp:418
#11 0x0000000000eb1710 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_startProcessing()::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f584c79c520) at /work/ArangoDB/arangod/Pregel/Worker.cpp:335
#12 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f5b65709e48) at /usr/include/c++/6.4.0/functional:2127
#13 operator() (__closure=0x7f5b65709e40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#14 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#15 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#16 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#17 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#18 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f69b74e7300, this=0x7f5c7d1038b0)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#19 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#20 run_one (ec=..., this=0x7f69b74e7300) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#21 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#22 arangodb::SchedulerThread::run (this=0x7f69820c1150) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#23 0x0000000001204499 in runMe (this=0x7f69820c1150) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#24 arangodb::Thread::startThread (arg=0x7f69820c1150) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#25 0x00000000012c4559 in ThreadStarter (data=0x7f6981fb1090) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#26 0x00000000027d1a08 in start (p=0x7f5b6570cae0) at src/thread/pthread_create.c:150
#27 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

daoxian · 2019-06-21T05:59:29Z

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000110cb09 in readIntegerNonEmpty<unsigned long> (length=<optimized out>, start=0x0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h:225
225     /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h: No such file or directory.
(gdb) bt
#0  0x000000000110cb09 in readIntegerNonEmpty<unsigned long> (length=<optimized out>, start=0x0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h:225
#1  byteSize (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Slice.h:777
#2  addInternal<arangodb::velocypack::Slice> (sub=..., attrLength=<optimized out>, attrName=0x7ec1c807df10 "H,\016\003", this=0x7f412e5320e0)
    at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:681
#3  addInternal<arangodb::velocypack::Slice> (sub=..., attrName=..., this=0x7f412e5320e0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:645
#4  add (sub=..., attrName=..., this=0x7f412e5320e0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:377
#5  (anonymous namespace)::handleProjections (node=<optimized out>, trxPtr=<optimized out>, slice=..., b=..., useRawDocumentPointers=<optimized out>)
    at /work/ArangoDB/arangod/Aql/DocumentProducingBlock.cpp:59
#6  0x000000000106710e in operator+= (__n=3910498560, this=<synthetic pointer>) at /usr/include/c++/6.4.0/bits/stl_deque.h:218
#7  operator+ (__n=3910498560, this=0x7eb24c484310) at /usr/include/c++/6.4.0/bits/stl_deque.h:228
#8  operator[] (__n=3910498560, this=0x7eb24c484310) at /usr/include/c++/6.4.0/bits/stl_deque.h:244
#9  operator[] (__n=3910498560, this=0x7eb24c484300) at /usr/include/c++/6.4.0/bits/stl_deque.h:1367
#10 operator() (b=..., a=..., this=<synthetic pointer>) at /work/ArangoDB/arangod/Aql/SortBlock.cpp:45
#11 operator()<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned int, unsigned int> > >, const std::pair<unsigned int, unsigned int> >
    (__val=..., __it=..., this=<synthetic pointer>) at /usr/include/c++/6.4.0/bits/predefined_ops.h:147
#12 __lower_bound<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned int, unsigned int> > >, std::pair<unsigned int, unsigned int>, __gnu_cxx::__ops::_Iter_comp_val<(anonymous namespace)::OurLessThan> > (__comp=..., __val=..., __last=..., __first=...) at /usr/include/c++/6.4.0/bits/stl_algobase.h:959
#13 std::__merge_adaptive<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned int, unsigned int> > >, long, std::pair<unsigned int, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<(anonymous namespace)::OurLessThan> > (__first=..., __middle=..., __last=..., __len1=139919698076146, __len2=139304249083264,
    __buffer=0x7eb24c483e60, __buffer_size=139324492732352, __comp=...) at /usr/include/c++/6.4.0/bits/stl_algo.h:2441
#14 0x000000000106ee05 in LoggerStream (this=0x2) at /work/ArangoDB/lib/Logger/LoggerStream.h:43
#15 arangodb::aql::TraversalBlock::shutdown (this=<optimized out>, errorCode=<optimized out>) at /work/ArangoDB/arangod/Aql/TraversalBlock.cpp:248
#16 0x00000000010e28bb in arangodb::SocketSslTcp::sslHandshake (this=0x7eb24c4848b0) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.cpp:79
#17 0x00000000010e3256 in system_category () at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/error_code.ipp:198
#18 get_system_category () at /work/ArangoDB/3rdParty/asio/1.12/include/asio/error.hpp:228
#19 error_wrapper<int> (ec=..., return_value=-792) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/socket_ops.ipp:85
#20 shutdown (ec=..., what=1, s=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/socket_ops.ipp:459
#21 shutdown (this=<optimized out>, impl=..., ec=..., what=asio::socket_base::shutdown_send) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:210
#22 shutdown (ec=..., what=asio::socket_base::shutdown_send, this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/basic_socket.hpp:1622
#23 arangodb::SocketSslTcp::shutdownSend (this=<optimized out>, ec=...) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.h:94
#24 0x00000000010d3a86 in _M_capacity (this=<optimized out>, __capacity=<optimized out>) at /usr/include/c++/6.4.0/bits/basic_string.h:160
#25 _M_construct<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.tcc:220
#26 _M_construct_aux<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.h:196
#27 _M_construct<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.h:215
#28 basic_string (__a=..., __n=<optimized out>, __s=0x7eb1f1c41773 "", this=0x7eb24c484b90) at /usr/include/c++/6.4.0/bits/basic_string.h:447
#29 copyString (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Slice.h:684
#30 arangodb::RestCollectionHandler::<lambda(const std::shared_ptr<arangodb::LogicalCollection>&)>::operator()(const std::shared_ptr<arangodb::LogicalCollection> &) const (
    __closure=0x7eb24c484c50, coll=...) at /work/ArangoDB/arangod/RestHandler/RestCollectionHandler.cpp:423
#31 0x00000000010d3ca0 in ?? () at /usr/include/c++/6.4.0/bits/basic_string.h:180
#32 0x00007f41580eabe0 in ?? ()
#33 0x00000000010e2d62 in arangodb::SocketSslTcp::sslHandshake (this=0x2710) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.cpp:57
---Type <return> to continue, or q <return> to quit---
#34 0x00000000010d4f93 in do_use_service (owner=<optimized out>, factory=0x73a9c0 <asio::detail::service_registry::create<asio::detail::scheduler, asio::execution_context>(void*)>,
    key=<synthetic pointer>, this=0x7f49d4ffd640) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:147
#35 use_service<asio::detail::scheduler> (this=0x7f49d4ffd640) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:29
#36 use_service<asio::detail::scheduler> (e=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/execution_context.hpp:32
#37 epoll_reactor (ctx=..., this=0x7eb214044b20) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:39
#38 create<asio::detail::epoll_reactor, asio::io_context> (owner=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#39 do_use_service (owner=<optimized out>, factory=<optimized out>, key=<synthetic pointer>, this=0x7f49df214000)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:131
#40 use_service<asio::detail::epoll_reactor> (owner=..., this=0x7f49df214000) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:38
#41 use_service<asio::detail::epoll_reactor> (ioc=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.hpp:38
#42 reactive_socket_service_base (io_context=..., this=0x7eb24c484c28) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:33
#43 reactive_socket_service (io_context=..., this=0x7eb24c484c00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#44 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#45 0x00000000010d5245 in unlock (this=0x7f46fb2fbb60) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/posix_mutex.hpp:57
#46 unlock (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scoped_lock.hpp:71
#47 do_use_service (owner=0x7f49e0a94e00, factory=0x73b060 <asio::detail::service_registry::create<asio::detail::epoll_reactor, asio::execution_context>(void*)>,
    key=<synthetic pointer>, this=0x7f49e0a94e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:130
#48 use_service<asio::detail::epoll_reactor> (this=0x7f49e0a94e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:29
#49 use_service<asio::detail::epoll_reactor> (e=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/execution_context.hpp:32
#50 init_task (this=0x31cfdb8 <ssl3_ciphers+3192>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:131
#51 init_task (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:147
#52 reactive_socket_service_base (io_context=..., this=0x45c) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:35
#53 reactive_socket_service (io_context=..., this=0x434) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#54 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#55 0x00000000010d543b in throw_error (location=0x2980711 "mutex", err=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/throw_error.hpp:41
#56 posix_mutex (this=0x0) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/posix_mutex.ipp:36
#57 conditionally_enabled_mutex (enabled=144, this=0x7eb24c484d80) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/conditionally_enabled_mutex.hpp:108
#58 epoll_reactor (ctx=..., this=0x7eb24c484d50) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:46
#59 create<asio::detail::epoll_reactor, asio::io_context> (owner=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#60 do_use_service (owner=<optimized out>, factory=<optimized out>, key=<synthetic pointer>, this=0xffffffffffffffff)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:131
#61 use_service<asio::detail::epoll_reactor> (owner=..., this=0xffffffffffffffff) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:38
#62 use_service<asio::detail::epoll_reactor> (ioc=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.hpp:38
#63 reactive_socket_service_base (io_context=..., this=0x7eb24c484d68) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:33
#64 reactive_socket_service (io_context=..., this=0x7eb24c484d40) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#65 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#66 0x00000000008c83ad in unlock (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/posix_static_mutex.hpp:49
#67 ~scoped_lock (this=<optimized out>, __in_chrg=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scoped_lock.hpp:53
#68 add (ec=..., signal_number=15, impl=..., this=0x7f49df214000) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/signal_set_service.ipp:256
#69 add (signal_number=15, this=0x7eb24c484d90) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/signal_set.hpp:256
#70 arangodb::SchedulerFeature::buildControlCHandler (this=0x7eb24c484eb0) at /work/ArangoDB/arangod/Scheduler/SchedulerFeature.cpp:335
---Type <return> to continue, or q <return> to quit---
#71 0x00000000008c2e5b in operator|= (__i=9223372036854775808, this=0x7f4031c9d308) at /usr/include/c++/6.4.0/bits/atomic_base.h:336
#72 setStopping (this=0x7f4031c9d300) at /work/ArangoDB/arangod/Scheduler/Scheduler.h:112
#73 arangodb::rest::Scheduler::beginShutdown (this=0x7f4031c9d300) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:636
#74 0x00000000011f7709 in arangodb::basics::StringUtils::replace (sourceStr=..., fromStr=..., toStr=...) at /work/ArangoDB/lib/Basics/StringUtils.cpp:1144
#75 0x00000000012b76e9 in ~MutexLocker (this=<optimized out>, __in_chrg=<optimized out>) at /work/ArangoDB/lib/Basics/MutexLocker.h:103
#76 arangodb::ScriptLoader::defineScript (this=0x0, name=..., script=...) at /work/ArangoDB/lib/Utilities/ScriptLoader.cpp:84
#77 0x00000000027c5a9c in tan (x=<unavailable>) at src/math/tan.c:50
#78 0x00000000027c707d in getnameinfo (sa=0x7ec1c807df10, sl=503, node=0x2 <Address 0x2 out of bounds>, nodelen=1, serv=0x520b <Address 0x520b out of bounds>, servlen=3776104640,
    flags=0) at src/network/getnameinfo.c:138
#79 0x0000000000000000 in ?? ()

daoxian · 2019-06-21T07:43:24Z

Could you plz analyze and fix these crashes ?

goedderz · 2019-06-21T11:53:11Z

Hi @daoxian,

I've had a look at the last backtrace (#9302 (comment)), and it looks broken. Would it be possible for you to provide us with the coredump?

Best,

Tobias

daoxian · 2019-06-21T13:46:08Z

Hi @daoxian,

I've had a look at the last backtrace (#9302 (comment)), and it looks broken. Would it be possible for you to provide us with the coredump?

Best,

Tobias

Thank you for your quick response. Do you mean the coredump files?

goedderz · 2019-06-21T13:49:55Z

Yes, exactly.

daoxian · 2019-06-21T15:01:59Z

Yes, exactly.

OK, wait for hours please, for the files are very large.

daoxian · 2019-06-24T02:35:02Z

Yes, exactly.

@goedderz I've uploaded relating coredump files:

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.222.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935585&Signature=GqX84VeOv0QkAG6DaqqQDuK5tT4%3D

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.49.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935615&Signature=9H7JThkwYyU8iLcR1q5%2FIvSU678%3D

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.158.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935627&Signature=91MUOi9pDS0Tlr270Fi6nosGKDM%3D

Thanks!

goedderz · 2019-06-24T07:28:58Z

Hi @daoxian, thanks a lot for sharing. I am currently downloading the files. We will have a look and then come back to you.

goedderz · 2019-06-26T10:31:02Z

Hi @daoxian,
gdb says that two of the core files are truncated (namely core.11626 and core.1301). These are the sizes in bytes of the files I got - are these consistent with yours?

703720357888 core.11626
868124520448 core.1301
547165097984 core.66237
102406086656 core.70242

I'm also calculating SHA256 sums, but this is going to take a few hours.

goedderz · 2019-06-26T11:18:07Z

Independent of that, the backtraces of all 4 look like they point to the same problem, which is the same as in #9301, as @mpoeter already pointed out. @graetzer will look into it.

daoxian · 2019-06-26T12:12:52Z

Hi @daoxian,
gdb says that two of the core files are truncated (namely core.11626 and core.1301). These are the sizes in bytes of the files I got - are these consistent with yours?
703720357888 core.11626
868124520448 core.1301
547165097984 core.66237
102406086656 core.70242
I'm also calculating SHA256 sums, but this is going to take a few hours.

Yes they're the same size as those on my local machine.

daoxian · 2019-06-26T12:16:54Z

@goedderz Everyday I can see new coredump files. Two of them listed as follows:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000b0a79e in arangodb::RocksDBIndex::blackListKey (this=this@entry=0x7f6647cd7710,
    data=0x7f52a5526077 "site_shikoku_np_co_jp__2Fen/6753de861d88c1e6f3c5835033162d1e48a9bf5155e651352e279c0f2K_Y3LDEQK-AnGversionG6224570Fvertexxfdf341d57f9cc7fcad17c30860df63a9ea7594c92b52da52c4b017ddDpathC/enKdomain_namePs"..., len=<optimized out>) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp:289
289     /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp: No such file or directory.
(gdb) bt
#0  0x0000000000b0a79e in arangodb::RocksDBIndex::blackListKey (this=this@entry=0x7f6647cd7710,
    data=0x7f52a5526077 "site_shikoku_np_co_jp__2Fen/6753de861d88c1e6f3c5835033162d1e48a9bf5155e651352e279c0f2K_Y3LDEQK-AnGversionG6224570Fvertexxfdf341d57f9cc7fcad17c30860df63a9ea7594c92b52da52c4b017ddDpathC/enKdomain_namePs"..., len=<optimized out>) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp:289
#1  0x0000000000fcf9e6 in blackListKey (ref=..., this=0x7f6647cd7710) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.h:147
#2  arangodb::RocksDBEdgeIndex::insertInternal (this=0x7f6647cd7710, trx=0x7f5517bee260, mthd=0x7f54cab0d1e0, documentId=..., doc=..., mode=arangodb::Index::normal)
    at /work/ArangoDB/arangod/RocksDBEngine/RocksDBEdgeIndex.cpp:605
#3  0x0000000000fb3e21 in arangodb::RocksDBCollection::insertDocument (this=this@entry=0x7f6649b85800, trx=trx@entry=0x7f5517bee260, documentId=..., doc=..., options=...)
    at /work/ArangoDB/arangod/RocksDBEngine/RocksDBCollection.cpp:1296
#4  0x0000000000fb4060 in arangodb::RocksDBCollection::insert(arangodb::transaction::Methods*, arangodb::velocypack::Slice, arangodb::ManagedDocumentResult&, arangodb::OperationOptions&, unsigned long&, bool, unsigned long&, arangodb::KeyLockInfo*, std::function<arangodb::Result ()>) (this=0x7f6649b85800, trx=0x7f5517bee260, slice=..., mdr=..., options=...,
    resultMarkerTick=@0x7f5517bed3f0: 0, revisionId=@0x7f5517bed3f8: 1637397464475500713, callbackDuringLock=...) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBCollection.cpp:797
#5  0x00000000009dfa85 in arangodb::LogicalCollection::insert(arangodb::transaction::Methods*, arangodb::velocypack::Slice, arangodb::ManagedDocumentResult&, arangodb::OperationOptions&, unsigned long&, bool, unsigned long&, arangodb::KeyLockInfo*, std::function<arangodb::Result ()>) (this=<optimized out>, trx=<optimized out>, slice=..., slice@entry=..., result=...,
    options=..., resultMarkerTick=@0x7f5517bed3f0: 0, lock=false, revisionId=@0x7f5517bed3f8: 1637397464475500713, keyLockInfo=0x7f5517bed760, callbackDuringLock=...)
    at /work/ArangoDB/arangod/VocBase/LogicalCollection.cpp:934
#6  0x0000000000911fd0 in arangodb::transaction::Methods::<lambda(arangodb::transaction::Methods::VPackSlice)>::operator()(arangodb::transaction::Methods::VPackSlice) const (
    __closure=__closure@entry=0x7f5517bed7b0, value=...) at /work/ArangoDB/arangod/Transaction/Methods.cpp:1787
#7  0x0000000000917838 in arangodb::transaction::Methods::insertLocal (this=this@entry=0x7f5517bee260, collectionName=..., value=..., options=...)
    at /work/ArangoDB/arangod/Transaction/Methods.cpp:1849
#8  0x0000000000917b7b in arangodb::transaction::Methods::insert (this=this@entry=0x7f5517bee260, collectionName=..., value=..., options=...)
    at /work/ArangoDB/arangod/Transaction/Methods.cpp:1602
#9  0x00000000007fe8da in arangodb::RestImportHandler::performImport (this=this@entry=0x7f55393f2400, trx=..., result=..., collectionName=..., babies=..., complete=false, opOptions=...)
    at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:884
#10 0x0000000000800482 in arangodb::RestImportHandler::createFromJson (this=this@entry=0x7f55393f2400, type=...) at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:511
#11 0x000000000080362e in arangodb::RestImportHandler::execute (this=0x7f55393f2400) at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:101
#12 0x00000000006fda5e in arangodb::rest::RestHandler::executeEngine (this=this@entry=0x7f55393f2400, isContinue=isContinue@entry=false)
    at /work/ArangoDB/arangod/GeneralServer/RestHandler.cpp:427
#13 0x00000000006fe92a in arangodb::rest::RestHandler::runHandlerStateMachine (this=this@entry=0x7f55393f2400) at /work/ArangoDB/arangod/GeneralServer/RestHandler.cpp:269
#14 0x0000000001113f41 in runHandler (cb=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=<optimized out>)
    at /work/ArangoDB/arangod/GeneralServer/RestHandler.h:75
#15 arangodb::rest::GeneralCommTask::handleRequest (this=<optimized out>, doLock=<optimized out>, handler=...) at /work/ArangoDB/arangod/GeneralServer/GeneralCommTask.cpp:482
#16 0x000000000111402d in operator() (isDirect=<optimized out>, __closure=<optimized out>) at /work/ArangoDB/arangod/GeneralServer/GeneralCommTask.cpp:450
#17 std::_Function_handler<void(bool), arangodb::rest::GeneralCommTask::handleRequestSync(std::shared_ptr<arangodb::rest::RestHandler>)::<lambda(bool)> >::_M_invoke(const std::_Any_data &, <unknown type in /usr/lib/debug/usr/sbin/arangod.debug, CU 0x4b59ce, DIE 0x4b59e2>) (__functor=..., __args#0=<optimized out>) at /usr/include/c++/6.4.0/functional:1731
#18 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f5517beee48) at /usr/include/c++/6.4.0/functional:2127
#19 operator() (__closure=0x7f5517beee40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#20 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#21 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#22 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#23 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
---Type <return> to continue, or q <return> to quit---
#24 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f7720bb7e00, this=0x7f5539414ec0)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#25 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#26 run_one (ec=..., this=0x7f7720bb7e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#27 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#28 arangodb::SchedulerThread::run (this=0x7f58b61dec70) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#29 0x0000000001204499 in runMe (this=0x7f58b61dec70) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#30 arangodb::Thread::startThread (arg=0x7f58b61dec70) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#31 0x00000000012c4559 in ThreadStarter (data=0x7f63ee5104a0) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#32 0x00000000027d1a08 in start (p=0x7f5517bf1ae0) at src/thread/pthread_create.c:150
#33 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  getCollectionIdLocal (name=..., this=0x7f4ca5f3e280) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7f4ca5f3e280) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7f4ca5f3e280, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7f4c97cd4980, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7f9adb3e16e0, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f4ca5e24d40, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f4777b7de48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7f4777b7de40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7fba1ce8ae00, this=0x7f4acc270890)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7fba1ce8ae00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7fa6eac217a0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7fa6eac217a0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7fa6eac217a0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7fa6eabf3d80) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7f4777b80ae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

If you need coredump files, I can also supply.

goedderz · 2019-06-26T14:07:45Z

Thanks a lot. For now we're fine and probably have enough information to fix it, otherwise we will ask you for more.

Best,

Tobias

daoxian · 2019-06-26T14:21:44Z

Thanks， and please let me know when this issue gets fixed

graetzer · 2019-06-26T14:39:03Z

now I am curious did you enable the collection cache ?

daoxian · 2019-06-27T01:36:56Z

now I am curious did you enable the collection cache ?

No, the startup cmd is:
arangodb --starter.mode=single --server.storage-engine=rocksdb --starter.data-dir=/home/admin/arangodb/data_3.4.6_singlemode_partition --all.rocksdb.block-cache-size=1000000000 --all.rocksdb.enforce-block-cache-size-limit=true --all.rocksdb.total-write-buffer-size=1000000000 --all.rocksdb.write-buffer-size=100000000 --all.rocksdb.max-write-buffer-number=100000 --all.rocksdb.min-write-buffer-number-to-merge=1 --all.rocksdb.max-total-wal-size=83886080 --all.server.maximal-queue-size=40960000 --all.server.prio1-size=1048576000

daoxian · 2019-06-27T01:40:17Z

Btw, there're over 10,000 tables in one database. And all the tables are created dynamically.

daoxian · 2019-06-28T12:42:17Z

now I am curious did you enable the collection cache ?

Hi @graetzer, I think I've found the coredump reason: After bulk import, I called collection.unload() method, while pregel on this collection is running. This is related to the coredump info "getCollectionIdLocal".

daoxian · 2019-07-01T03:47:34Z

Hi @graetzer , I think you've fixed that, which version or git branch should I use to verify this fix ?

daoxian · 2019-07-02T11:16:48Z

Thanks for your patch. But I've found new coredumps after disable unload. Maybe my previous guess was wrong.

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  getCollectionIdLocal (name=..., this=0x7ef6cc930780) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7ef6cc930780) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7ef6cc930780, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7ef701207f80, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7ef8379f2120, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7ef70120b7c0, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7ea3779dbe48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7ea3779dbe40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f2847e63e00, this=0x7ef6f9a13a40)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7f2847e63e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7efe72040db0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7efe72040db0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7efe72040db0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7f1514fc9b80) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7ea3779deae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21

graetzer · 2019-07-02T13:29:06Z

is there a chance you are using empty edge collection names ?

daoxian · 2019-07-03T02:05:49Z

is there a chance you are using empty edge collection names ?

Do you mean an edge collection that doen't exists or an empty edge collection containing no edges? An empty edge collection is very likely to be used in my codes, this is a common case. And when an exception occurs while creating a collection, there IS a chance that the subsequent codes using a graph whose edge collection doesn't exist.

daoxian · 2019-07-03T09:04:00Z

Btw, can vertex or edge collection be modified (adding or removing vertex/edge) when pregel is running on that graph?

daoxian · 2019-07-04T03:50:29Z

I've tried skipping empty edge collections and broken graph, still coredump:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
55      include/jemalloc/internal/atomic.h: No such file or directory.
(gdb) bt
#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
#1  extent_arena_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:49
#2  je_large_dalloc (tsdn=0x7f961a882f98, extent=0x0) at src/large.c:347
#3  0x00000000010b6ef3 in deallocate (this=0x7f9816750a48, __p=<optimized out>) at /usr/include/c++/6.4.0/ext/new_allocator.h:110
#4  deallocate (__a=..., __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/alloc_traits.h:462
#5  _M_deallocate (this=0x7f9816750a48, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:178
#6  std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > >::_M_range_insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (this=0x7f9816750a48, __position=...,
    __first=..., __last=...) at /usr/include/c++/6.4.0/bits/vector.tcc:685
#7  0x00000000010b7289 in _M_insert_dispatch<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __pos=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1375
#8  insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __position=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1100
#9  arangodb::pregel::ArrayInCache<arangodb::pregel::SenderMessage<double> >::mergeCache (this=0x7f9b13091480, config=..., otherCache=0x7f9b13091b80)
    at /work/ArangoDB/arangod/Pregel/IncomingCache.cpp:151
#10 0x0000000000ea0eb7 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_processVertices (this=0x7f9a85dbf010,
    threadId=<optimized out>, vertexIterator=...) at /work/ArangoDB/arangod/Pregel/Worker.cpp:418
#11 0x0000000000eb1710 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_startProcessing()::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f93fc36f140) at /work/ArangoDB/arangod/Pregel/Worker.cpp:335
#12 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f961a881e48) at /usr/include/c++/6.4.0/functional:2127
#13 operator() (__closure=0x7f961a881e40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#14 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#15 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#16 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#17 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#18 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7fd2f854de00, this=0x7f975ce6d950)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#19 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#20 run_one (ec=..., this=0x7fd2f854de00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#21 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#22 arangodb::SchedulerThread::run (this=0x7fbfbd103bd0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#23 0x0000000001204499 in runMe (this=0x7fbfbd103bd0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#24 arangodb::Thread::startThread (arg=0x7fbfbd103bd0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#25 0x00000000012c4559 in ThreadStarter (data=0x7fbfc1b3e130) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#26 0x00000000027d1a08 in start (p=0x7f961a884ae0) at src/thread/pthread_create.c:150
#27 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

Look at the coredump files, I don't know how long can I bear all the stuff, maybe switching to a alternative db is a good choise.

mpoeter · 2019-07-04T07:50:09Z

This is yet another occurrence of the same issue as in #9302 (comment) and #9301 (comment)

As already mentioned, this looks very much like a race on ArrayInCache::_shardMap to me (or more specifically on the entries of that map). It seems that ArrayInCache<M>::_set, ArrayInCache<M>::clear and ArrayInCache<M>::erase modify the _shardMap or entries of that map without acquiring the _bucketLocker. I am not familiar enough with the code to judge whether this is indeed a problem, or if other synchronization mechanisms are used somewhere else, but maybe someone from the ArangoDB team can take a look at it.

sourabhedake-mm · 2023-01-04T06:20:05Z

@dothebart Is there any update on the core dump mentioned in - #9302 (comment)

I am facing a similar core in the prod -

(gdb) bt 
#0  0x0000556d1d209d0a in arangodb::velocypack::readIntegerNonEmpty<unsigned long> (start=0x7f364e208ae8 <error: Cannot access memory at address 0x7f364e208ae8>, length=4)
    at /c8db/3rdParty/velocypack/include/velocypack/velocypack-common.h:230
#1  0x0000556d1d209622 in arangodb::velocypack::Slice::getNthOffset (this=0x7f35ee9f5820, index=0) at /c8db/3rdParty/velocypack/src/Slice.cpp:495
#2  0x0000556d1be88709 in arangodb::velocypack::Slice::getNthKeyUntranslated (this=0x7f35ee9f5820, index=0) at /c8db/3rdParty/velocypack/include/velocypack/Slice.h:1080
#3  0x0000556d1bee4ea9 in arangodb::velocypack::ObjectIterator::operator* (this=0x7f35ee9f5820) at /c8db/3rdParty/velocypack/include/velocypack/Iterator.h:242
#4  0x0000556d1e8d222e in rice::needStrip (document=...) at /c8db/arangod/Rice/Rice.cpp:514
#5  0x0000556d1e8d22a9 in rice::strip (document=..., didStrip=0x7f35ee9f5a67, copyIfNotStrip=false) at /c8db/arangod/Rice/Rice.cpp:524
#6  0x0000556d1c78c3e7 in arangodb::RocksDBAllIndexIterator::nextDocument(std::function<void (arangodb::LocalDocumentId const&, arangodb::velocypack::Slice)> const&, unsigned long, bool) (this=0x7f358729c520, 
    cb=..., limit=1000, raw=false) at /c8db/arangod/RocksDBEngine/RocksDBIterators.cpp:128
#7  0x0000556d1c5d3f55 in arangodb::OperationCursor::nextDocument(std::function<void (arangodb::LocalDocumentId const&, arangodb::velocypack::Slice)> const&, unsigned long, bool) (this=0x7f353c7f0960, 
    callback=..., batchSize=1000, raw=false) at /c8db/arangod/Utils/OperationCursor.cpp:86
#8  0x0000556d1d00b60f in arangodb::aql::EnumerateCollectionExecutor::produceRows (this=0x7f35b28586a8, output=...) at /c8db/arangod/Aql/EnumerateCollectionExecutor.cpp:148
#9  0x0000556d1c9f9f0a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::EnumerateCollectionExecutor>::getSomeWithoutTrace (this=0x7f35b2858400, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:169
#10 0x0000556d1c9f9a47 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::EnumerateCollectionExecutor>::getSome (this=0x7f35b2858400, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#11 0x0000556d1cfdf462 in arangodb::aql::DependencyProxy<true>::prefetchBlock (this=0x7f35f9608a08, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#12 0x0000556d1cfdf1c8 in arangodb::aql::DependencyProxy<true>::fetchBlockForPassthrough (this=0x7f35f9608a08, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:211
#13 0x0000556d1cb87d73 in arangodb::aql::SingleRowFetcher<true>::fetchBlockForPassthrough (this=0x7f35f9608ae8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:66
#14 0x0000556d1ca2062c in arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0>::fetchBlockForPassthrough (this=0x7f35f9608bb0, atMost=1000) at /c8db/arangod/Aql/CalculationExecutor.h:114
#15 0x0000556d1ca19bc9 in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)1>::run<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> > (executor=..., 
    engine=..., nrItems=1000, nrRegs=2) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:572
#16 0x0000556d1c9f5b5a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::requestWrappedBlock (this=0x7f35f9608900, nrItems=1000, nrRegs=2)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#17 0x0000556d1c9f578c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSomeWithoutTrace (this=0x7f35f9608900, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#18 0x0000556d1c9f5461 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSome (this=0x7f35f9608900, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#19 0x0000556d1cfdfd62 in arangodb::aql::DependencyProxy<false>::prefetchBlock (this=0x7f3601fe2008, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#20 0x0000556d1cfdf702 in arangodb::aql::DependencyProxy<false>::fetchBlock (this=0x7f3601fe2008, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:78
#21 0x0000556d1cb87be0 in arangodb::aql::SingleRowFetcher<false>::fetchBlock (this=0x7f3601fe20e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:52
#22 0x0000556d1ca16a29 in arangodb::aql::SingleRowFetcher<false>::preFetchNumberOfRows (this=0x7f3601fe20e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.h:98
#23 0x0000556d1ca743de in arangodb::aql::FilterExecutor::expectedNumberOfRows (this=0x7f3601fe2170, atMost=1000) at /c8db/arangod/Aql/FilterExecutor.cpp:94
#24 0x0000556d1ca1b30a in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)2>::run<arangodb::aql::FilterExecutor> (executor=..., engine=..., nrItems=1000, nrRegs=2)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:629
#25 0x0000556d1c9fb99c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::requestWrappedBlock (this=0x7f3601fe1f00, nrItems=1000, nrRegs=2) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#26 0x0000556d1c9fb594 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::getSomeWithoutTrace (this=0x7f3601fe1f00, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#27 0x0000556d1c9fb301 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::skipSome (this=0x7f3601fe1f00, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:320
#28 0x0000556d1cfdfc04 in arangodb::aql::DependencyProxy<false>::skipSome (this=0x7f35f9608408, toSkip=1000000000) at /c8db/arangod/Aql/DependencyProxy.cpp:178
#29 0x0000556d1cb87b1a in arangodb::aql::SingleRowFetcher<false>::skipRows (this=0x7f35f96084e8, atMost=1000000000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:74
#30 0x0000556d1c9f4808 in arangodb::aql::CountCollectExecutor::produceRows (this=0x7f35f9608570, output=...) at /c8db/arangod/Aql/CountCollectExecutor.h:106
#31 0x0000556d1c9f8786 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CountCollectExecutor>::getSomeWithoutTrace (this=0x7f35f9608300, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:169
#32 0x0000556d1c9f826b in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CountCollectExecutor>::getSome (this=0x7f35f9608300, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#33 0x0000556d1cfdf462 in arangodb::aql::DependencyProxy<true>::prefetchBlock (this=0x7f35f9608708, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#34 0x0000556d1cfdf1c8 in arangodb::aql::DependencyProxy<true>::fetchBlockForPassthrough (this=0x7f35f9608708, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:211
#35 0x0000556d1cb87d73 in arangodb::aql::SingleRowFetcher<true>::fetchBlockForPassthrough (this=0x7f35f96087e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:66
#36 0x0000556d1ca2062c in arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0>::fetchBlockForPassthrough (this=0x7f35f96088b0, atMost=1000) at /c8db/arangod/Aql/CalculationExecutor.h:114
#37 0x0000556d1ca19bc9 in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)1>::run<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> > (executor=..., 
    engine=..., nrItems=1000, nrRegs=4) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:572
#38 0x0000556d1c9f5b5a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::requestWrappedBlock (this=0x7f35f9608600, nrItems=1000, nrRegs=4)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#39 0x0000556d1c9f578c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSomeWithoutTrace (this=0x7f35f9608600, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#40 0x0000556d1c9f5461 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSome (this=0x7f35f9608600, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#41 0x0000556d1ca3d946 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::IdExecutor<void> >::getSome (this=0x7f35f961a3c0, atMost=1000) at /c8db/arangod/Aql/IdExecutor.h:92
--Type <RET> for more, q to quit, c to continue without paging--c
#42 0x0000556d1ca279f6 in arangodb::aql::ExecutionEngine::getSome (this=0x7f35b2853300, atMost=1000) at /c8db/arangod/Aql/ExecutionEngine.cpp:519
#43 0x0000556d1c18f9fd in arangodb::aql::Query::execute (this=0x7f35ee9f7b00, registry=0x7f360f5d4640, queryResult=...) at /c8db/arangod/Aql/Query.cpp:746
#44 0x0000556d1c191bfd in arangodb::aql::Query::executeSync (this=0x7f35ee9f7b00, registry=0x7f360f5d4640) at /c8db/arangod/Aql/Query.cpp:1027
#45 0x0000556d1c02e00e in c8db_db::queryFetch (aql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", vars=std::shared_ptr<arangodb::velocypack::Builder> (empty) = {...}, vocbase=0x7f35c7d82200) at /c8db/arangod/C8DBFeatures/query_helper.cpp:84
#46 0x0000556d1c02d4eb in c8db_db::queryFetchSingle (aql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", vars=std::shared_ptr<arangodb::velocypack::Builder> (empty) = {...}, builder=..., vocbase=0x7f35c7d82200) at /c8db/arangod/C8DBFeatures/query_helper.cpp:51
#47 0x0000556d1c52f10e in getCount (c8ql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", tenant="qa_amol_test.com", database="QATest23_24") at /c8db/arangod/Statistics/StatisticsWorker.cpp:262
#48 0x0000556d1c53a626 in arangodb::StatisticsWorker::updateDatabaseStats (this=0x7f360f451500, tenant="qa_amol_test.com", database="QATest23_24") at /c8db/arangod/Statistics/StatisticsWorker.cpp:1592
#49 0x0000556d1c5372cc in arangodb::StatisticsWorker::updateTenantStatistics (this=0x7f360f451500) at /c8db/arangod/Statistics/StatisticsWorker.cpp:1252
#50 0x0000556d1c53be05 in arangodb::StatisticsWorker::run (this=0x7f360f451500) at /c8db/arangod/Statistics/StatisticsWorker.cpp:1691
#51 0x0000556d1d2684ab in arangodb::Thread::runMe (this=0x7f360f451500) at /c8db/lib/Basics/Thread.cpp:317
#52 0x0000556d1d265f8b in arangodb::Thread::startThread (arg=0x7f360f451500) at /c8db/lib/Basics/Thread.cpp:96
#53 0x0000556d1d30d3fa in ThreadStarter (data=0x7f360e22b330) at /c8db/lib/Basics/threads-posix.cpp:78

mpoeter mentioned this issue Jun 21, 2019

[BUG] coredump in 3.4.6-1 #9301

Closed

markuspf mentioned this issue Jun 21, 2019

Fix potential segfault #9308

Merged

9 tasks

maxkernbach added 1 Analyzing 3 Core labels Jun 24, 2019

goedderz self-assigned this Jun 24, 2019

goedderz assigned graetzer Jun 26, 2019

This was referenced Jul 1, 2019

Add lock on unload #9367

Merged

use a lock when calling unload #9375

Merged

daoxian mentioned this issue Jul 9, 2019

ERROR {pregel} Could not resolve target shard of edge #9403

Open

dothebart added 1 Bug and removed 1 Analyzing labels Sep 2, 2020

sourabhedake-mm mentioned this issue Jan 12, 2023

[BUG] New coredump observed in Slice #17930

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] another coredump for version 3.4.6-1 #9302

[BUG] another coredump for version 3.4.6-1 #9302

daoxian commented Jun 20, 2019

daoxian commented Jun 20, 2019

daoxian commented Jun 20, 2019

daoxian commented Jun 21, 2019

daoxian commented Jun 21, 2019

goedderz commented Jun 21, 2019

daoxian commented Jun 21, 2019

goedderz commented Jun 21, 2019

daoxian commented Jun 21, 2019

daoxian commented Jun 24, 2019 •

edited

Loading

goedderz commented Jun 24, 2019

goedderz commented Jun 26, 2019

goedderz commented Jun 26, 2019

daoxian commented Jun 26, 2019

daoxian commented Jun 26, 2019 •

edited

Loading

goedderz commented Jun 26, 2019

daoxian commented Jun 26, 2019 •

edited

Loading

graetzer commented Jun 26, 2019

daoxian commented Jun 27, 2019

daoxian commented Jun 27, 2019

daoxian commented Jun 28, 2019 •

edited

Loading

daoxian commented Jul 1, 2019 •

edited

Loading

daoxian commented Jul 2, 2019

graetzer commented Jul 2, 2019

daoxian commented Jul 3, 2019 •

edited

Loading

daoxian commented Jul 3, 2019

daoxian commented Jul 4, 2019 •

edited

Loading

mpoeter commented Jul 4, 2019

sourabhedake-mm commented Jan 4, 2023

[BUG] another coredump for version 3.4.6-1 #9302

[BUG] another coredump for version 3.4.6-1 #9302

Comments

daoxian commented Jun 20, 2019

daoxian commented Jun 20, 2019

daoxian commented Jun 20, 2019

daoxian commented Jun 21, 2019

daoxian commented Jun 21, 2019

goedderz commented Jun 21, 2019

daoxian commented Jun 21, 2019

goedderz commented Jun 21, 2019

daoxian commented Jun 21, 2019

daoxian commented Jun 24, 2019 • edited Loading

goedderz commented Jun 24, 2019

goedderz commented Jun 26, 2019

goedderz commented Jun 26, 2019

daoxian commented Jun 26, 2019

daoxian commented Jun 26, 2019 • edited Loading

goedderz commented Jun 26, 2019

daoxian commented Jun 26, 2019 • edited Loading

graetzer commented Jun 26, 2019

daoxian commented Jun 27, 2019

daoxian commented Jun 27, 2019

daoxian commented Jun 28, 2019 • edited Loading

daoxian commented Jul 1, 2019 • edited Loading

daoxian commented Jul 2, 2019

graetzer commented Jul 2, 2019

daoxian commented Jul 3, 2019 • edited Loading

daoxian commented Jul 3, 2019

daoxian commented Jul 4, 2019 • edited Loading

mpoeter commented Jul 4, 2019

sourabhedake-mm commented Jan 4, 2023

daoxian commented Jun 24, 2019 •

edited

Loading

daoxian commented Jun 26, 2019 •

edited

Loading

daoxian commented Jun 26, 2019 •

edited

Loading

daoxian commented Jun 28, 2019 •

edited

Loading

daoxian commented Jul 1, 2019 •

edited

Loading

daoxian commented Jul 3, 2019 •

edited

Loading

daoxian commented Jul 4, 2019 •

edited

Loading