jemalloc lock and crash problem

[huzhiguang]

hi,I have 2 jemalloc problem in hhvm.
First problem is jemalloc crash in hhvm.
Second problem is jemalloc hang(lock) in hhvm.

we use jemalloc version is 3.4.0, we want to update jemalloc version(3.6.0) to slove this problem.
Beacause we can't stable emersion this two questions, so I want to know whether you encountered such a problem.

first problem crash stack:

#0  HPHP::bt_handler(int) at crash-reporter.cpp:0
#1  __restore_rt at sigaction.c:0
#2  HPHP::JIT::FixupMap::fixup(HPHP::ExecutionContext*) const at bin/hhvm_bin:0
#3  HPHP::JIT::MCGenerator::syncWork() at bin/hhvm_bin:0
#4  HPHP::ExecutionContext::debugBacktrace(bool, bool, bool, HPHP::VMParserFrame*, bool, int) at bin/hhvm_bin:0
#5  HPHP::debug_string_backtrace(bool, bool, int) at bin/hhvm_bin:0
#6  HPHP::bt_handler(int) at crash-reporter.cpp:0
#7  __restore_rt at sigaction.c:0
#8  arena_dalloc_bin_locked at :0
#9  tcache_bin_flush_small at :0
#10 tcache_event_hard at :0
#11 malloc at /home/../lib/libjemalloc.so.1:0
#12 mbfl_convert_filter_new at bin/hhvm_bin:0
#13 mbfl_buffer_converter_new at bin/hhvm_bin:0
#14 HPHP::php_mb_convert_encoding(char const*, unsigned long, char const*, char const*, unsigned int*) at ext_mb.cpp:0
#15 HPHP::f_mb_convert_encoding(HPHP::String const&, HPHP::String const&, HPHP::Variant const&) at bin/hhvm_bin:0

second problem hang stack (gdb attach)

#0  0x00007f53eb551e14 in __lll_lock_wait ()
   from /home/lib/libpthread.so.0
#1  0x00007f53eb54d6b7 in _L_lock_535 ()
   from /home/lib/libpthread.so.0
#2  0x00007f53eb54d4e0 in pthread_mutex_lock ()
   from /home/lib/libpthread.so.0
#3  0x00007f53ed43c30f in arena_dalloc_large ()
   from /home/lib/libjemalloc.so.1
#4  0x00000000009dc42c in HPHP::MemoryManager::resetAllocator() ()
#5  0x0000000000a11ed7 in HPHP::/home() ()
#6  0x0000000000ea02b5 in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) ()
#7  0x0000000000e6bb45 in HPHP::ServerWorker<std::shared_ptr<HPHP::LibEventJob>, HPHP::LibEventTransportTraits>::doJob
Impl(std::shared_ptr<HPHP::LibEventJob>, bool) ()
#8  0x0000000000e6bd98 in HPHP::ServerWorker<std::shared_ptr<HPHP::LibEventJob>, HPHP::LibEventTransportTraits>::doJob
(std::shared_ptr<HPHP::LibEventJob>) ()
#9  0x0000000000e6aac8 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::LibEventJob>, HPHP::Server*, true, false, HPHP::J
obQueueDropVMStack>::start() ()
#10 0x000000000153b527 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()
#11 0x00007f53eb54b1c1 in start_thread ()
   from /home/lib/libpthread.so.0
#12 0x00007f53eaa610ad in clone ()

Thanks.

[paulbiss]

Does updating jemalloc resolve this crash?

[huzhiguang]

This crash is not often appear,I have been updated, but this problem still exists.

[JoelMarcey]

@huzhiguang Can you provide a piece of code that demonstrates the crash?

cc: @jasone

[huzhiguang]

@JoelMarcey The question is not easy repetition, can appear only production environment.

[apaloma]

I think I am also experiencing this.

I'm on Arch linux with jemalloc 3.6.0 installed.

I have the below two stacks and also the occasional hang.

[Thread 0x7fae1e7fb700 (LWP 12900) exited]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fae693ff700 (LWP 10920)]
0x00007fae94d12bf1 in ?? () from /usr/lib/libjemalloc.so.1
(gdb) bt
#0  0x00007fae94d12bf1 in ?? () from /usr/lib/libjemalloc.so.1
#1  0x00007fae94d1411d in ?? () from /usr/lib/libjemalloc.so.1
#2  0x0000000000a075dc in HPHP::MemoryManager::resetAllocator() ()
#3  0x00000000009c266c in HPHP::hphp_session_exit() ()
#4  0x0000000000b954c2 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) ()
#5  0x0000000000b95982 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() ()
#6  0x00000000016bb4b7 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()
#7  0x00007fae93520314 in start_thread () from /usr/lib/libpthread.so.0
#8  0x00007fae92a343ed in clone () from /usr/lib/libc.so.6

[Switching to Thread 0x7fd792bff700 (LWP 11947)]
0x00007fd7c0484bf1 in ?? () from /usr/lib/libjemalloc.so.1
(gdb) bt
#0  0x00007fd7c0484bf1 in ?? () from /usr/lib/libjemalloc.so.1
#1  0x00007fd7c048611d in ?? () from /usr/lib/libjemalloc.so.1
#2  0x0000000001145dd7 in ?? ()
#3  0x0000000001147759 in ?? ()
#4  0x0000000000bf1f15 in void HPHP::Native::callFunc<false, false>(HPHP::Func const*, void*, HPHP::TypedValue*, HPHP::TypedValue&) ()
#5  0x0000000000bf2168 in HPHP::TypedValue* HPHP::Native::methodWrapper<false, false>(HPHP::ActRec*) ()
#6  0x0000000002e00397 in ?? ()
#7  0x0000000000ef017e in enterTCHelper ()
#8  0x0000000000ee0629 in HPHP::jit::x64::BackEnd::enterTCHelper(unsigned char*, HPHP::jit::TReqInfo&) ()
#9  0x0000000000e0ec27 in HPHP::jit::MCGenerator::enterTC(unsigned char*, void*) ()
#10 0x0000000000c7e2ee in HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) ()
#11 0x0000000000c7e55b in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) ()
#12 0x0000000000c7e7a9 in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) ()
#13 0x0000000000a5f911 in ?? ()
#14 0x0000000000a62301 in HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) ()
#15 0x00000000009c1aea in HPHP::hphp_invoke(HPHP::ExecutionContext*, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()
#16 0x0000000000b70549 in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()
#17 0x0000000000b71730 in HPHP::HttpRequestHandler::handleRequestImpl(HPHP::Transport*) ()
#18 0x0000000000b723f9 in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) ()
#19 0x0000000000b954c2 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) ()
#20 0x0000000000b95982 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() ()
#21 0x00000000016bb4b7 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()
#22 0x00007fd7bec92314 in start_thread () from /usr/lib/libpthread.so.0
#23 0x00007fd7be1a63ed in clone () from /usr/lib/libc.so.6

This is a production environment, so I don't know which PHP code was running at the time. Let me know how I might collect more information. I'd be happy to provide it.

[Orvid]

This issue has been waiting for more info for more than 2 weeks. Closing for now, feel free to re-open it if you can provide more info.

https://github.com/facebook/hhvm/wiki/Human-Timeouts