hhvm hangs in phpmyadmin with high cpu usage until gateway times out

[r3wt]

unsure whats causing this. i restart hhvm and its fine for a few requests then starts shitting the bed

[r3wt]

nevermind, its still doing it

HipHop VM 3.10.1 (rel)
Compiler: tags/HHVM-3.10.1-0-g689b4969a141620ee5a282ce0dbf72278c84d44b
Repo schema: 6c99ee1f98340f6f3ef397a332583f0e843a627d


phpMyAdmin version 4.5.1.0

[jwatzman]

We're going to need more info than this. Does this happen on a stock phpmyadmin? Do you know what it's doing when it's stuck -- can you attach gdb and get a backtrace or something like that? Even an strace from the affected time would be useful.

[r3wt]

to update, as of the last 10 minutes this problem has randomly subsided. not sure why.

[r3wt]

Issue has returned again after rebooting the server. checking the logs i see
cap_set_proc() failed issue is probably identical to #4913 , only after a while it seems to go away. probably some regression in phpmyadmin.

[r3wt]

can reproduce on a fresh ubuntu 14.04 vm. setting up a new prod server and issue returns same as on dev machine.

[jaredjones]

Can confirm same issue on HHVM 3.11.1

[jaredjones]

(gdb) bt
#0 0x00000000013ee951 in std::Hashtable<HPHP::jit::Block*, std::pair<HPHP::jit::Block* const, HPHP::jit::FrameStateMgr::BlockState>, std::allocator<std::pair<HPHP::jit::Block* const, HPHP::jit::FrameStateMgr::BlockState> >, std::detail::Select1st, std::equal_toHPHP::jit::Block*, std::hashHPHP::jit::Block*, std::detail::Mod_range_hashing, std::detail::Default_ranged_hash, std::detail::Prime_rehash_policy, std::detail::Hashtable_traits<false, false, true> >::clear() ()
#1 0x00000000016f33a3 in HPHP::jit::selectTracelet(HPHP::jit::RegionContext const&, int, bool, bool) ()
#2 0x00000000013288a4 in HPHP::jit::selectRegion(HPHP::jit::RegionContext const&, HPHP::jit::TransKind) ()
#3 0x00000000018d7fd1 in HPHP::jit::MCGenerator::translateWork(HPHP::jit::TranslArgs const&) ()
#4 0x00000000018db785 in HPHP::jit::MCGenerator::retranslate(HPHP::jit::TranslArgs const&) ()
#5 0x00000000018dbebc in HPHP::jit::MCGenerator::createTranslation(HPHP::jit::TranslArgs const&) ()
#6 0x00000000018dcb07 in HPHP::jit::MCGenerator::bindJmp(unsigned char, HPHP::SrcKey, HPHP::jit::ServiceRequest, HPHP::jit::TransFlags, bool&) ()
#7 0x00000000018dfcfe in MCGenerator_handleServiceRequest ()
#8 0x000000000197a456 in handleSRHelper ()
#9 0x0000000002f02799 in enterTCHelper ()
#10 0x0000000001820e53 in HPHP::jit::x64::enterTCImpl(unsigned char, HPHP::ActRec) ()
#11 0x00000000018c1de9 in HPHP::jit::MCGenerator::enterTC(unsigned char, HPHP::ActRec) ()
#12 0x000000000110eb39 in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue, HPHP::Func const, HPHP::Variant const&, HPHP::ObjectData, HPHP::Class, HPHP::VarEnv, HPHP::StringData, HPHP::ExecutionContext::InvokeFlags, bool) ()
#13 0x0000000002f49c65 in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue, HPHP::Unit const) ()
#14 0x0000000000c96eef in HPHP::include_impl_invoke(HPHP::String const&, bool, char const) ()
#15 0x0000000000ddcb14 in HPHP::hphp_invoke(HPHP::ExecutionContext, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()
#16 0x0000000000f24b40 in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()
#17 0x0000000000f263ee in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport_) ()
#18 0x0000000000faecaa in HPHP::ServerWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptrHPHP::FastCGIJob, bool) ()
#19 0x0000000000faf21e in HPHP::ServerWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::FastCGITransportTraits>::doJob(std::shared_ptrHPHP::FastCGIJob)
()
#20 0x0000000000fa6596 in HPHP::JobQueueWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::Server_, true, false, HPHP::JobQueueDropVMStack>::start() ()
#21 0x000000000242c5e7 in HPHP::AsyncFuncImpl::ThreadFunc(void_) ()
#22 0x0000000000d19ea7 in HPHP::start_routine_wrapper(void_) ()
#23 0x00007f1de69b8182 in start_thread (arg=0x7f1dd41ff700) at pthread_create.c:312
#24 0x00007f1de5ec547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

[jaredjones]

@jwatzman
Anything else you need from me?

The CPU hangs at 100% and HHVM becomes unresponsive. I believe the issue is with PHP files that aren't yet compiled into the HHBC database, because after the file is executed at least once after I restart HHVM, that access in PHPMyAdmin no longer causes HHVM to hang.

[jaredjones]

I did another backtrace and got:

#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f93178 in HPHP::HttpServer::runOrExitProcess() ()
#2 0x0000000000df9d5c in HPHP::execute_program_impl(int, char**) ()
#3 0x0000000000dfb0bf in HPHP::execute_program(int, char**) ()
#4 0x0000000000ae6461 in main ()

[fredemmott]

• run thread apply all bt instead - otherwise you just get it for the main thread, which isn't actually executing your request
• profiling tools (eg 'perf') could also get you more useful data

[jaredjones]

Sure thing, here is your trace. I cannot use perf since linux-tools doesn't work on the virtualization kernel: 2.6.32-042stab108.8

Thread 4 (Thread 0x7f84f73ff700 (LWP 30172)):
#0 0x00007f8507f7db13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f850a633813 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2 0x00007f850a61f907 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3 0x0000000002ead62e in folly::EventBase::loopBody(int) ()
#4 0x0000000002eb044d in folly::EventBase::loopForever() ()
#5 0x00000000027a898a in proxygen::WorkerThread::runLoop() ()
#6 0x00000000027a8bcf in std::thread::_Impl<std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()> >::_M_run() ()
#7 0x00007f8508815a40 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f8508a70182 in start_thread (arg=0x7f84f73ff700) at pthread_create.c:312
#9 0x00007f8507f7d47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f84f65ff700 (LWP 30173)):
#0 0x0000000001435036 in HPHP::jit::FrameStateMgr::pauseBlock(HPHP::jit::Block_) ()
#1 0x0000000001525408 in HPHP::jit::IRBuilder::pushBlock(HPHP::jit::BCMarker, HPHP::jit::Block_) ()
#2 0x000000000130d970 in HPHP::jit::irgen::makeExit(HPHP::jit::IRGS&, int) ()
#3 0x00000000012e303b in HPHP::jit::irgen::checkTypeLocal(HPHP::jit::IRGS&, unsigned int, HPHP::jit::Type, int, bool) ()
#4 0x00000000016f005c in HPHP::jit::(anonymous namespace)::RegionFormer::go() ()
#5 0x00000000016f1f28 in HPHP::jit::selectTracelet(HPHP::jit::RegionContext const&, int, bool, bool) ()
#6 0x00000000013288a4 in HPHP::jit::selectRegion(HPHP::jit::RegionContext const&, HPHP::jit::TransKind) ()
#7 0x00000000018d7fd1 in HPHP::jit::MCGenerator::translateWork(HPHP::jit::TranslArgs const&) ()
#8 0x00000000018db785 in HPHP::jit::MCGenerator::retranslate(HPHP::jit::TranslArgs const&) ()
#9 0x00000000018dbebc in HPHP::jit::MCGenerator::createTranslation(HPHP::jit::TranslArgs const&) ()
#10 0x00000000018dcb07 in HPHP::jit::MCGenerator::bindJmp(unsigned char_, HPHP::SrcKey, HPHP::jit::ServiceRequest, HPHP::jit::TransFlags, bool&) ()
#11 0x00000000018dfcfe in MCGenerator_handleServiceRequest ()
#12 0x000000000197a456 in handleSRHelper ()
#13 0x0000000002f02799 in enterTCHelper ()
#14 0x0000000001820e53 in HPHP::jit::x64::enterTCImpl(unsigned char_, HPHP::ActRec_) ()
#15 0x00000000018c1de9 in HPHP::jit::MCGenerator::enterTC(unsigned char_, HPHP::ActRec_) ()
#16 0x000000000110eb39 in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue_, HPHP::Func const_, HPHP::Variant const&, HPHP::ObjectData_, HPHP::Class_, HPHP::VarEnv_, HPHP::StringData_, HPHP::ExecutionContext::InvokeFlags, bool) ()
#17 0x0000000002f49c65 in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue_, HPHP::Unit const_)
()
#18 0x0000000000c96eef in HPHP::include_impl_invoke(HPHP::String const&, bool, char const_) ()
#19 0x0000000000ddcb14 in HPHP::hphp_invoke(HPHP::ExecutionContext_, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()
#20 0x0000000000f24b40 in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport_, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()
#21 0x0000000000f263ee in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport_) ()
#22 0x0000000000faecaa in HPHP::ServerWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptrHPHP::FastCGIJob, bool) ()
---Type to continue, or q to quit---
#23 0x0000000000faf21e in HPHP::ServerWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::FastCGITransportTraits>::doJob(std::shared_ptrHPHP::FastCGIJob) ()
#24 0x0000000000fa6596 in HPHP::JobQueueWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::Server_, true, false, HPHP::JobQueueDropVMStack>::start() ()
#25 0x000000000242c5e7 in HPHP::AsyncFuncImpl::ThreadFunc(void_) ()
#26 0x0000000000d19ea7 in HPHP::start_routine_wrapper(void_) ()
#27 0x00007f8508a70182 in start_thread (arg=0x7f84f65ff700) at pthread_create.c:312
#28 0x00007f8507f7d47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f84f15ff700 (LWP 30174)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00000000023f2518 in HPHP::SynchronizableMulti::waitImpl(int, int, bool, timespec_) ()
#2 0x0000000000fa6433 in HPHP::JobQueueWorkerstd::shared_ptr<HPHP::FastCGIJob, HPHP::Server_, true, false, HPHP::JobQueueDropVMStack>::start() ()
#3 0x000000000242c5e7 in HPHP::AsyncFuncImpl::ThreadFunc(void_) ()
#4 0x0000000000d19ea7 in HPHP::start_routine_wrapper(void_) ()
#5 0x00007f8508a70182 in start_thread (arg=0x7f84f15ff700) at pthread_create.c:312
#6 0x00007f8507f7d47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f84f81ff700 (LWP 30170)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000000000f93178 in HPHP::HttpServer::runOrExitProcess() ()
#2 0x0000000000df9d5c in HPHP::execute_program_impl(int, char**) ()
#3 0x0000000000dfb0bf in HPHP::execute_program(int, char**) ()
#4 0x0000000000ae6461 in main ()

[Prochitaem]

Same issue:

HipHop VM 3.11.1 (rel)
Compiler: tags/HHVM-3.11.1-0-g64d37362bc0b6aee919492ad61cf65ce0a5d5e92
Repo schema: 8b80ba45250a6669cd610c189dbbb55b6218c2a3

phpMyAdmin 4.5.4.1

[JorgenPhi]

Same issue here... Stock Ubuntu 14.04 server w/ nginx and hhvm.

[kidager]

I can confirm the same issue still exists (HHVM version 3.13.1)

[Orvid]

We've been unable to reproduce this bug, and are unable to investigate further. We are closing the issue for now, if you identify a test case we can use to reproduce your bug feel free to re-open.

[TechnikEmpire]

So uh, you guys didn't try at all because I just had this repro exactly on
Ubuntu 16.04.3
nginx 1.13.8
MariaDB 10.0.33
phpmyadmin 4.7.7

Simply start up phpmyadmin for the first time after configuring this setup and start clicking the expansion nodes in the database tree view and watch hhvm turn your server into a deadlocked space header.