3.4.0 (official hhvm-3.4.0~trusty package) eats all memory+swap

[tat]

I upgraded my aws instances (c3.large) to 3.4.0 (official packages got from http://dl.hhvm.com/ubuntu) and all of them get killed by oom-killer after eating all RAM and swap in about 5 minutes (getting about 300 requests per minute).

Is there anything I can check to track down the issue?

My server.ini:
pid = /var/run/hhvm/pid
hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.resource_limit.max_socket = 10000
hhvm.log.header = true

Thanks,
stefano

[mklooss]

can confirm same senario here, but also on HHVM 3.3.
we have to restart the hhvm process every 6 hours to keep the server online
were are using an Dedicated Server

[tat]

In my case hhvm eats RAM+swap (about 5gigs in total) in about 5/7 minutes.

[mklooss]

yesterday we had the same on 64 GB RAM and 8 GB SWAP in about 6 hours :/

[jwatzman]

@tat, this is an increase from 3.3 to 3.4? That's interesting. Can you get a heap profile for us? The process is unfortunately somewhat involved.

• You can use these admin server commands to start heap profiling and then dump a heap profile: https://github.com/facebook/hhvm/blob/master/hphp/doc/command.admin_server#L70-L72
• This old blog post talks about how to enable the admin server and talk to it. (Some parts of it are outdated, talking about the old HPHPc cross-compiler and web server, but the rest is still good.) http://hhvm.com/blog/521/the-adminserver
• Then, use pprof to generate a PDF and post it here, with some command like pprof --show_bytes --pdfwhich hhvmhhvm.blah.heap > out.pdf-- this page has the full details https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Leak-Checking

cc @paulbiss

@mklooss, what you're experiencing is unfortunately somewhat expected, and is a long-term issue we've been slowly looking into. It's not indicative of server instability, we just haven't optimized for a super-long-running server very much, since FB pushes twice a day. (Though 6 hours is still quite short.)

[fredemmott]

The admin server speaks FastCGI now, not HTTP - you'll also need to configure your webserver to give you access to it.

[tat]

Thanks for the feedback, I've got the admin interface working but I'm getting an error from the activate command:
Error 2 in mallctl("prof.active", ...)

do you know what's the issue? where is the file supposed to be written to? /tmp ?

Here's the jmalloc-stats output I captured from the admin interface, http://pastebin.com/vBvfPiP5

btw @jwatzman 3.3 is working fine for me, RAM usage is stable at about 350MB; it has been running for days without restarts.

[mklooss]

jemalloc Stats: https://gist.github.com/mklooss/8091e48c4551f40d05c8
currently the HHVM Process eats ~10 GB RAM, process runs ~ 2 hours

[frankh]

I'm getting the same problem running a large wordpress site on HHVM. Memory usage starts at ~450mb and climbs to 1.2mb before restarting (not 100% if OOM killed or crashes yet) every ~2 hours

This is HHVM 3.4.0 on ubuntu/trusty

[jwatzman]

I just cherry-picked a memory leak fix into the 3.4 branch -- can someone who's experiencing this build that branch and report back? If it fixes it, we can roll a 3.4.1 release. The issue is that if you are passing invalid arguments to some builtin functions, such that the builtin raises a warning, we leak a small amount of memory each time -- and it looks new in 3.4. If your PHP app generates a lot of warnings from builtins, then this could easily be your bug :)

Thanks for the feedback, I've got the admin interface working but I'm getting an error from the activate command:
Error 2 in mallctl("prof.active", ...)

do you know what's the issue? where is the file supposed to be written to? /tmp ?

I don't, sorry -- @fredemmott, @paulbiss, can either of you advise better?

[jwatzman]

can someone who's experiencing this build that branch and report back? If it fixes it, we can roll a 3.4.1 release.

I went ahead and built a deb for trusty with this patch: http://dl.hhvm.com/ubuntu/hhvm_3.4.1-devtest~trusty_amd64.deb You can manually install that so you don't have to build HHVM yourself; let me know if it works better.

[denji]

configure:

./configure -DENABLE_SSP=ON -DDEBUG_MEMORY_LEAK=ON -DDEBUG_APC_LEAK=ON

  -DDEBUG_APC_LEAK=ON|OFF : Allow easier debugging of apc leaks : Default: OFF
  -DDEBUG_MEMORY_LEAK=ON|OFF : Allow easier debugging of memory leaks : Default: OFF
  -DENABLE_SSP=ON|OFF : Enabled GCC/LLVM stack-smashing protection : Default: OFF

[levixie]

@jwatzman which change you cherry-pick?
I only see some doc update
Thanks