Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passenger process crashing on 5.0.27 #1794

Closed
VvanGemert opened this issue Apr 11, 2016 · 30 comments

Comments

Projects
None yet
6 participants
@VvanGemert
Copy link

commented Apr 11, 2016

Since version 5.0.27 we're getting random crashes of a passenger process which results in 502 errors in Nginx for all the requests being handled by the crashing process. We've rolled back to version 5.0.26, which doesn't have this issue. We're running this version on two servers which both crash the same way since the update to 5.0.27. The webservers are loadbalanced by Haproxy.

We're also using 5.0.27 successfully on identical servers but without the 'direct' spawn method required for our application using Ruby 1.8.7. So my guess would be it's related to this option. I've attached the crash log and additional info below.

Passenger version: open source 5.0.27 with nginx (Phusion APT repo) (multi-app mode)
OS: Ubuntu 14.04 LTS
Ruby versions running: Ruby 1.8.7-p370, Ruby 1.9.3-p374, Ruby 2.1.1, Ruby 2.3.0

Passenger Nginx config:

server {
  listen   80 default_server; ## listen for ipv4
  listen   443 default_server ssl

  server_name  .piedpiper.com;

  root /srv/piedpiper/current/public;

  passenger_ruby /home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby;
  passenger_enabled on;
  passenger_spawn_method direct;
  passenger_min_instances 5;
  passenger_max_requests 50000;
}

Nginx log message:

Assertion failed! src/cxx_supportlib/MemoryKit/mbuf.cpp:218: void Passenger::MemoryKit::mbuf_block_put(Passenger::MemoryKit::mbuf_block*): ((mbuf_block)->ne
xt.stqe_next) == __null
[ pid=31259, timestamp=1460034815 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 31259 with UID 0, si_addr=0x7a1b, randomSeed=1460
034763

Passenger crash log:

[ pid=28580, timestamp=1460034509 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 28580 with UID 0, si_addr=0x6fa4, randomSeed=1460034161
[ pid=28580 ] Crash log dumped to /var/tmp/passenger-crash-log.1460034509
[ pid=28580 ] Date, uname and ulimits:
Thu Apr  7 13:08:29 UTC 2016
Linux 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              256132
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
[ pid=28580 ] Phusion Passenger version: 5.0.27
[ pid=28580 ] Last assertion failure: (((mbuf_block)->next.stqe_next) == __null), function void Passenger::MemoryKit::mbuf_block_put(Passenger::MemoryKit::mbuf_block*), file src/cxx_supportlib/MemoryKit/mbuf.cpp, line 218.
[ pid=28580 ] libc backtrace available!
--------------------------------------
[ pid=28580 ] Backtrace with 35 frames:
Using backtrace sanitizer.
Passenger core[0x481390]
Passenger core[0x480c3c]
Passenger core[0x481f40]
Passenger core[0x482bfa]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x10340) [0x7f00c36b5340]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; gsignal+0x39) [0x7f00c2676cc9]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; abort+0x148) [0x7f00c267a0d8]
Passenger core[0x482e8c]
Passenger core[0x47bedb]
Passenger core[0x4a0781]
Passenger core[0x5e6615]
Passenger core[0x5e62fe]
Passenger core[0x5f71cb]
Passenger core[0x5f0411]
Passenger core[0x5dcbb4]
Passenger core[0x5daf05]
Passenger core[0x4a20df]
Passenger core[0x4a357c]
Passenger core[0x4a4ced]
Passenger core[0x4a49e7]
Passenger core[0x4a49a1]
Passenger core[0x6d1d8e]
Passenger core[0x6d2c68]
Passenger core[0x4f3c5c]
Passenger core[0x559981]
Passenger core[0x553451]
Passenger core[0x548d08]
Passenger core[0x45e7be]
Passenger core[0x6c6cc9]
Passenger core[0x4f3a20]
Passenger core[0x4f35db]
Passenger core[0x4f019a]
Passenger core[0x6bbae9]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x8182) [0x7f00c36ad182]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; clone+0x6d) [0x7f00c273a47d]
--------------------------------------
[ pid=28580 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7f00c3cc97c0, LWP 28580):
     in 'void waitForExitEvent()' (CoreMain.cpp:875)
     in 'void mainLoop()' (CoreMain.cpp:768)
     in 'int runCore()' (CoreMain.cpp:995)

Thread 'Pool garbage collector' (0x7f00c3c86700, LWP 28580):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.cpp:62)

Thread 'Pool analytics collector' (0x7f00c3cc7700, LWP 28580):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.cpp:64)

Thread 'Main event loop: thread 1' (0x7f00bd3d5700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 1: libuv poller' (0x7f00bcbd4700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 2' (0x7f00bcb53700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 2: libuv poller' (0x7f00bc352700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 3' (0x7f00affff700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 3: libuv poller' (0x7f00bc2d1700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 4' (0x7f00af7fe700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 4: libuv poller' (0x7f00bc250700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 5' (0x7f00aeffd700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 5: libuv poller' (0x7f00bc1cf700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 6' (0x7f00ae7fc700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 6: libuv poller' (0x7f00bc14e700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 7' (0x7f00adffb700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 7: libuv poller' (0x7f00bc0cd700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 8' (0x7f00ad7fa700, LWP 28580):
     in 'Passenger::ServerKit::Channel::Result Passenger::Core::Controller::onAppSourceData(Passenger::Core::Client*, Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::Request*, const Passenger::MemoryKit::mbuf&, int)' (ForwardResponse.cpp:160) -- Client 8-1054

Thread 'Main event loop: thread 8: libuv poller' (0x7f00acff9700, LWP 28580):
     (empty)

Thread 'API event loop' (0x7f00acf78700, LWP 28580):
     (empty)

Thread 'API event loop: libuv poller' (0x7f00ac777700, LWP 28580):
     (empty)

Thread 'Load balancer' (0x7f0067fff700, LWP 28580):
     (empty)

Thread 'PipeWatcher: PID 28611 stdout, fd 77' (0x7f00ac06d700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28611 stderr, fd 78' (0x7f00675f2700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28616 stdout, fd 81' (0x7f00675b1700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28616 stderr, fd 82' (0x7f0067570700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28801 stdout, fd 178' (0x7f0065513700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28801 stderr, fd 179' (0x7f00654d2700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28882 stdout, fd 189' (0x7f0065479700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28882 stderr, fd 190' (0x7f0065438700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28982 stdout, fd 137' (0x7f00653df700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28982 stderr, fd 150' (0x7f006539e700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29163 stdout, fd 195' (0x7f0065345700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29163 stderr, fd 196' (0x7f0065304700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29080 stdout, fd 191' (0x7f00652ab700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29080 stderr, fd 192' (0x7f006526a700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29285 stdout, fd 168' (0x7f0065211700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29285 stderr, fd 170' (0x7f00651d0700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29380 stdout, fd 86' (0x7f006518f700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29380 stderr, fd 87' (0x7f006514e700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29649 stderr, fd 124' (0x7f006508b700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29794 stdout, fd 132' (0x7f0064fde700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29804 stdout, fd 93' (0x7f0064f9d700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29541 stdout, fd 159' (0x7f0064f5c700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29541 stderr, fd 161' (0x7f0064f1b700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29841 stdout, fd 121' (0x7f0064eda700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29841 stderr, fd 122' (0x7f0064e99700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30053 stdout, fd 96' (0x7f0064e40700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30053 stderr, fd 97' (0x7f0064dff700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30341 stdout, fd 101' (0x7f0064da6700, LWP 30340):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30341 stderr, fd 102' (0x7f0064d65700, LWP 30340):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30649 stdout, fd 104' (0x7f0064d0c700, LWP 30648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30649 stderr, fd 105' (0x7f0064ccb700, LWP 30648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30831 stdout, fd 90' (0x7f0064c72700, LWP 30830):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30831 stderr, fd 91' (0x7f0064c31700, LWP 30830):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31272 stdout, fd 100' (0x7f0064bd8700, LWP 31271):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31272 stderr, fd 103' (0x7f0064b97700, LWP 31271):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31740 stdout, fd 113' (0x7f0064b3e700, LWP 31739):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31740 stderr, fd 114' (0x7f0064afd700, LWP 31739):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 32491 stdout, fd 107' (0x7f0064aa4700, LWP 32490):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 32491 stderr, fd 108' (0x7f0064a63700, LWP 32490):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 626 stdout, fd 109' (0x7f0064a0a700, LWP 625):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 626 stderr, fd 110' (0x7f00649c9700, LWP 625):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 827 stdout, fd 129' (0x7f0064970700, LWP 826):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 827 stderr, fd 130' (0x7f006492f700, LWP 826):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 2042 stdout, fd 134' (0x7f00648ee700, LWP 2041):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 2042 stderr, fd 135' (0x7f00648ad700, LWP 2041):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

### Request handler state (thread 1)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "1-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0004122257232666016,
                "timestamp" : 1460034509.332909
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "content_length" : 35956,
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "www.piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "PARSING_BODY",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -6.914138793945312e-06,
                    "timestamp" : 1460034509.333058
                },
                "last_data_send_time" : null,
                "method" : "POST",
                "path" : "/projects/E3350212/update.xml",
                "refcount" : 1,
                "request_body_already_read" : 35956,
                "request_body_fully_read" : true,
                "request_body_type" : "CONTENT_LENGTH",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4f-WHCU6B27UH",
                    "pid" : 29380
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0003762245178222656,
                    "timestamp" : 1460034509.332804
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.480
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.790
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 13,
    "mbuf_pool" :
    {
        "active_blocks" : 3,
        "active_memory" :
        {
            "bytes" : 1536,
            "human_readable" : "1.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 313,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 160256,
            "human_readable" : "156.5 KB"
        }
    },
    "peak_active_client_count" : 14,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.480
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.790
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2648712,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 2)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "2-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0002169609069824219,
                "timestamp" : 1460034509.33530
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "COMPLETE",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001230239868164062,
                    "timestamp" : 1460034509.335277
                },
                "last_data_send_time" : null,
                "method" : "GET",
                "path" : "/dashboard",
                "refcount" : 1,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4e-sT7bi5Yd7P",
                    "pid" : 28616
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001230239868164062,
                    "timestamp" : 1460034509.335322
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.930
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 14,
    "mbuf_pool" :
    {
        "active_blocks" : 4,
        "active_memory" :
        {
            "bytes" : 2048,
            "human_readable" : "2.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 291,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 148992,
            "human_readable" : "145.5 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.930
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2741175,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 3)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.040
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.960
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 17,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 294,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 150528,
            "human_readable" : "147.0 KB"
        }
    },
    "peak_active_client_count" : 17,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.040
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.020
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2418239,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 1,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 1
    }
}

### Request handler state (thread 4)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.50
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.660
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 329,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 168448,
            "human_readable" : "164.5 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.50
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.660
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2578363,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 2,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 2
    }
}

### Request handler state (thread 5)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "5-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0002715587615966797,
                "timestamp" : 1460034509.339006
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "COMPLETE",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001716613769531250,
                    "timestamp" : 1460034509.338994
                },
                "last_data_send_time" : null,
                "method" : "GET",
                "path" : "/projects/22896014-javastraat-112-2/embed2",
                "refcount" : 1,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4e-o5wXNU2aWE",
                    "pid" : 28882
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001716613769531250,
                    "timestamp" : 1460034509.339036
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "5-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.730
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 4,
        "active_memory" :
        {
            "bytes" : 2048,
            "human_readable" : "2.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 305,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 156160,
            "human_readable" : "152.5 KB"
        }
    },
    "peak_active_client_count" : 16,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.730
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2614924,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 6)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.850
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 292,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 149504,
            "human_readable" : "146.0 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.850
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 3280567,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 5,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 5
    }
}

### Request handler state (thread 7)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 184.60
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 16,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 343,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 175616,
            "human_readable" : "171.5 KB"
        }
    },
    "peak_active_client_count" : 16,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 184.60
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2633528,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 2,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 2
    }
}

### Request handler state (thread 8)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "8-1054" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "1s ago",
                "relative_timestamp" : -0.2005658149719238,
                "timestamp" : 1460034509.142548
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "content_length" : 128,
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "www.piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "WAITING_FOR_REFERENCES",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "1s ago",
                    "relative_timestamp" : -0.2005050182342529,
                    "timestamp" : 1460034509.142490
                },
                "last_data_send_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : 0.0,
                    "timestamp" : 1460034509.342995
                },
                "method" : "POST",
                "path" : "/login",
                "refcount" : 2,
                "request_body_already_read" : 128,
                "request_body_fully_read" : true,
                "request_body_type" : "CONTENT_LENGTH",
                "response_begun" : true,
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "1s ago",
                    "relative_timestamp" : -0.2005050182342529,
                    "timestamp" : 1460034509.142545
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "8-1054",
            "number" : 1054,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.440
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.170
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 13,
    "mbuf_pool" :
    {
        "active_blocks" : 5,
        "active_memory" :
        {
            "bytes" : 2560,
            "human_readable" : "2.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 316,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 161792,
            "human_readable" : "158.0 KB"
        }
    },
    "peak_active_client_count" : 14,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.440
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.170
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2467229,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}

### Request handler config (thread 1)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 2)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 3)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 4)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 5)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 6)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 7)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 8)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Pool state (simple)
----------- General information -----------
Max pool size : 50
App groups    : 3
Processes     : 23
Requests in top-level queue : 0

----------- Application groups -----------
/srv/portals/current/public:
  App root: /srv/portals/current
  Requests in queue: 0
  * PID: 29794   Sessions: 0       Processed: 31      Uptime: 5m 17s
    CPU: 1%      Memory  : 140M    Last used: 6s ago
    URL     : http://127.0.0.1:47878
    Password: Ol8OGwXCU5TgwLbo
  * PID: 29804   Sessions: 0       Processed: 0       Uptime: 5m 17s
    CPU: 0%      Memory  : 72M     Last used: 5m 17s ago
    URL     : http://127.0.0.1:42470
    Password: Ol8OGwXCU5TgwLbo

/srv/piedpiper/current/public:
  App root: /srv/piedpiper/current
  Requests in queue: 0
  * PID: 28616   Sessions: 1       Processed: 1144    Uptime: 5m 41s
    CPU: 22%     Memory  : 241M    Last used: 0s ago
    URL     : http://127.0.0.1:33582
    Password: alzftuPyrEUTXgL2
  * PID: 28882   Sessions: 1       Processed: 1098    Uptime: 5m 34s
    CPU: 21%     Memory  : 55M     Last used: 0s ago
    URL     : http://127.0.0.1:51254
    Password: alzftuPyrEUTXgL2
  * PID: 29080   Sessions: 0       Processed: 863     Uptime: 5m 28s
    CPU: 16%     Memory  : 240M    Last used: 1s ago
    URL     : http://127.0.0.1:55297
    Password: alzftuPyrEUTXgL2
  * PID: 29380   Sessions: 1       Processed: 599     Uptime: 5m 22s
    CPU: 12%     Memory  : 242M    Last used: 1s ago
    URL     : http://127.0.0.1:36184
    Password: alzftuPyrEUTXgL2
  * PID: 29541   Sessions: 0       Processed: 389     Uptime: 5m 17s
    CPU: 9%      Memory  : 238M    Last used: 1s ago
    URL     : http://127.0.0.1:49947
    Password: alzftuPyrEUTXgL2
  * PID: 29841   Sessions: 0       Processed: 225     Uptime: 5m 10s
    CPU: 5%      Memory  : 241M    Last used: 1s ago
    URL     : http://127.0.0.1:51225
    Password: alzftuPyrEUTXgL2
  * PID: 30053   Sessions: 0       Processed: 86      Uptime: 5m 5s
    CPU: 3%      Memory  : 238M    Last used: 1s ago
    URL     : http://127.0.0.1:43115
    Password: alzftuPyrEUTXgL2
  * PID: 30341   Sessions: 0       Processed: 29      Uptime: 4m 45s
    CPU: 2%      Memory  : 238M    Last used: 2s ago
    URL     : http://127.0.0.1:42400
    Password: alzftuPyrEUTXgL2
  * PID: 30649   Sessions: 0       Processed: 10      Uptime: 4m 27s
    CPU: 2%      Memory  : 237M    Last used: 13s ago
    URL     : http://127.0.0.1:34719
    Password: alzftuPyrEUTXgL2
  * PID: 31740   Sessions: 0       Processed: 4       Uptime: 2m 56s
    CPU: 3%      Memory  : 237M    Last used: 13s ago
    URL     : http://127.0.0.1:57264
    Password: alzftuPyrEUTXgL2
  * PID: 827     Sessions: 0       Processed: 1       Uptime: 1m 43s
    CPU: 5%      Memory  : 240M    Last used: 13s ago
    URL     : http://127.0.0.1:38925
    Password: alzftuPyrEUTXgL2
  * PID: 2042    Sessions: 0       Processed: 0       Uptime: 7s
    CPU: 62%     Memory  : 236M    Last used: 7s ago
    URL     : http://127.0.0.1:56086
    Password: alzftuPyrEUTXgL2

/srv/assets/current/public:
  App root: /srv/assets/current
  Requests in queue: 0
  * PID: 28611   Sessions: 0       Processed: 2402    Uptime: 5m 42s
    CPU: 16%     Memory  : 97M     Last used: 0s ago
    URL     : http://127.0.0.1:53306
    Password: Dsim6dBNi9wdsdKN
  * PID: 28801   Sessions: 0       Processed: 1007    Uptime: 5m 38s
    CPU: 8%      Memory  : 98M     Last used: 1s ago
    URL     : http://127.0.0.1:56174
    Password: Dsim6dBNi9wdsdKN
  * PID: 28982   Sessions: 0       Processed: 377     Uptime: 5m 34s
    CPU: 3%      Memory  : 95M     Last used: 2s ago
    URL     : http://127.0.0.1:57738
    Password: Dsim6dBNi9wdsdKN
  * PID: 29163   Sessions: 0       Processed: 117     Uptime: 5m 30s
    CPU: 2%      Memory  : 94M     Last used: 7s ago
    URL     : http://127.0.0.1:46674
    Password: Dsim6dBNi9wdsdKN
  * PID: 29285   Sessions: 0       Processed: 36      Uptime: 5m 26s
    CPU: 1%      Memory  : 90M     Last used: 24s ago
    URL     : http://127.0.0.1:36250
    Password: Dsim6dBNi9wdsdKN
  * PID: 30831   Sessions: 0       Processed: 10      Uptime: 4m 20s
    CPU: 1%      Memory  : 84M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:60219
    Password: Dsim6dBNi9wdsdKN
  * PID: 31272   Sessions: 0       Processed: 4       Uptime: 3m 34s
    CPU: 2%      Memory  : 83M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:47879
    Password: Dsim6dBNi9wdsdKN
  * PID: 32491   Sessions: 0       Processed: 2       Uptime: 2m 33s
    CPU: 2%      Memory  : 84M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:50087
    Password: Dsim6dBNi9wdsdKN
  * PID: 626     Sessions: 0       Processed: 0       Uptime: 1m 54s
    CPU: 3%      Memory  : 77M     Last used: 1m 54s ago
    URL     : http://127.0.0.1:50511
    Password: Dsim6dBNi9wdsdKN


### mbuf stats

nfree_mbuf_blockq  : 313
nactive_mbuf_blockq: 3
mbuf_block_chunk_size: 512

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.27</passenger_version><group_count>3</group_count><process_count>23</process_count><max>50</max><capacity_used>23</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/srv/portals/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>2</capacity_used><secret>Ol8OGwXCU5TgwLbo</secret><group default="true"><name>/srv/portals/current/public</name><component_name>/srv/portals/current/public</component_name><app_root>/srv/portals/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>1Bg9SS9kFBt4QYySMtYt</uuid><enabled_process_count>2</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>2</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>Ol8OGwXCU5TgwLbo</secret><api_key>Ol8OGwXCU5TgwLbo</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/portals/current</app_root><app_group_name>/srv/portals/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-2.1.1/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/portals/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-2.1.1/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>Ol8OGwXCU5TgwLbo</api_key><min_processes>2</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>29794</pid><sticky_session_id>1907340957</sticky_session_id><gupid>1734e4f-wlt7AiL450</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>31</processed><spawner_creation_time>1460034189502621</spawner_creation_time><spawn_start_time>1460034192071354</spawn_start_time><spawn_end_time>1460034192090197</spawn_end_time><last_used>1460034503644880</last_used><last_used_desc>6s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>4cbd782</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>218296</rss><pss>180165</pss><private_dirty>143992</private_dirty><swap>0</swap><real_memory>143992</real_memory><vmsize>762872</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/portals/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.nqFZo5dwp3mjo6vakiXGhfAQQyab88sanPdvSAv3y3u3OdvfP2yjkkC6iCWKKuP</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:47878</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29804</pid><sticky_session_id>387385906</sticky_session_id><gupid>1734e4f-MyXJSHIac2</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034189502621</spawner_creation_time><spawn_start_time>1460034192148573</spawn_start_time><spawn_end_time>1460034192166233</spawn_end_time><last_used>1460034192166233</last_used><last_used_desc>5m 17s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>4cbd782</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>0</cpu><rss>147208</rss><pss>109623</pss><private_dirty>74460</private_dirty><swap>0</swap><real_memory>74460</real_memory><vmsize>627680</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/portals/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.U8aIYvLCK6wzwYj1AxehFk1dE4rnQFYKm2jvLmgK0z9WeohP0DBhHlvgfJFfFSi</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:42470</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup><supergroup><name>/srv/piedpiper/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>12</capacity_used><secret>alzftuPyrEUTXgL2</secret><group default="true"><name>/srv/piedpiper/current/public</name><component_name>/srv/piedpiper/current/public</component_name><app_root>/srv/piedpiper/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>boYegTE5EWAwu1ZEAkfm</uuid><enabled_process_count>12</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>12</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>alzftuPyrEUTXgL2</secret><api_key>alzftuPyrEUTXgL2</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/piedpiper/current</app_root><app_group_name>/srv/piedpiper/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/piedpiper/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>direct</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>alzftuPyrEUTXgL2</api_key><min_processes>5</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>28616</pid><sticky_session_id>1147769522</sticky_session_id><gupid>1734e4e-sT7bi5Yd7P</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>1144</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034161338182</spawn_start_time><spawn_end_time>1460034168987485</spawn_end_time><last_used>1460034509075818</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 41s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>22</cpu><rss>255544</rss><pss>248167</pss><private_dirty>247704</private_dirty><swap>0</swap><real_memory>247704</real_memory><vmsize>516120</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.VlIACqmLAzBr6iONPdeZRdnuqJjuR8SXDLt6DnslzwUHIH9pkc0X9SGOUQcnJL4</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:33582</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28882</pid><sticky_session_id>205145316</sticky_session_id><gupid>1734e4e-o5wXNU2aWE</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>1098</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034168988314</spawn_start_time><spawn_end_time>1460034175307485</spawn_end_time><last_used>1460034509137226</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 34s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>21</cpu><rss>255776</rss><pss>153099</pss><private_dirty>57316</private_dirty><swap>0</swap><real_memory>57316</real_memory><vmsize>516624</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.tBqrKQeaDuDflbo6xdxPjveLFCX43GAcJMfzxupSkS3DdU2QdtuXjoZLuvPsC9D</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:51254</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29080</pid><sticky_session_id>836435173</sticky_session_id><gupid>1734e4e-BOaR5jZbb8</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>863</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034175308397</spawn_start_time><spawn_end_time>1460034181226241</spawn_end_time><last_used>1460034508950817</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 28s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>16</cpu><rss>254472</rss><pss>247099</pss><private_dirty>246628</private_dirty><swap>0</swap><real_memory>246628</real_memory><vmsize>514372</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.UCJb7sjToo4PTlT87GqXtVREluejAs2dDsm2mIRGipzGSaKFSkqbBg1BupLubOO</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:55297</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29380</pid><sticky_session_id>1511737890</sticky_session_id><gupid>1734e4f-WHCU6B27UH</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>599</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034181227100</spawn_start_time><spawn_end_time>1460034187076457</spawn_end_time><last_used>1460034508984267</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 22s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>12</cpu><rss>255768</rss><pss>248470</pss><private_dirty>248048</private_dirty><swap>0</swap><real_memory>248048</real_memory><vmsize>515672</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.6xfFrPQhV58iiQ2ahncFuyYzQwscLtHjRzoOqGueK2GDQ1qa3tG76NT7asjQbMW</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:36184</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29541</pid><sticky_session_id>1699461801</sticky_session_id><gupid>1734e4f-hFmx64IYrS</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>389</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034187077109</spawn_start_time><spawn_end_time>1460034192997591</spawn_end_time><last_used>1460034508611582</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>252224</rss><pss>244920</pss><private_dirty>244496</private_dirty><swap>0</swap><real_memory>244496</real_memory><vmsize>512236</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.979yazxkCYFnBzklWCap9TLd1OTm2acokv53f1vJ35yChLUqIzPQ7SBMORYazvE</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:49947</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29841</pid><sticky_session_id>93839976</sticky_session_id><gupid>1734e4f-5WGeBqwh31</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>225</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034192998388</spawn_start_time><spawn_end_time>1460034199125300</spawn_end_time><last_used>1460034508638243</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 10s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>5</cpu><rss>255276</rss><pss>248154</pss><private_dirty>247768</private_dirty><swap>0</swap><real_memory>247768</real_memory><vmsize>515336</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.Zl5H8LyP3oe02CmRr8OxwzvhvuFqNqZu4wYh0Bd7JeRKBT6iUrx5bHQykikLqQ1</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:51225</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30053</pid><sticky_session_id>1195154836</sticky_session_id><gupid>1734e4f-WAu7t8RAGG</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>86</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034199126245</spawn_start_time><spawn_end_time>1460034204880009</spawn_end_time><last_used>1460034508649844</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 5s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>251456</rss><pss>244592</pss><private_dirty>244204</private_dirty><swap>0</swap><real_memory>244204</real_memory><vmsize>511876</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vJyVaVZvdaDD3wCKhEiimu2Q08GRRXQp6gnjKhH49dGRXaskkAsp1yas522uN1P</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:43115</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30341</pid><sticky_session_id>1542455619</sticky_session_id><gupid>1734e4f-hfNr8temR5</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>29</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034218292966</spawn_start_time><spawn_end_time>1460034224136214</spawn_end_time><last_used>1460034507702825</last_used><last_used_desc>2s ago</last_used_desc><uptime>4m 45s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>251028</rss><pss>244114</pss><private_dirty>243756</private_dirty><swap>0</swap><real_memory>243756</real_memory><vmsize>511512</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.dmpgmxpf0AsclxpztHSu1YFTqPc1hDtI038vB2AuFzSR8LcEmifj9KLmEuIshT8</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:42400</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30649</pid><sticky_session_id>1970717670</sticky_session_id><gupid>1734e4f-kwkRCzZDBM</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>10</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034236483732</spawn_start_time><spawn_end_time>1460034242905155</spawn_end_time><last_used>1460034496314730</last_used><last_used_desc>13s ago</last_used_desc><uptime>4m 27s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>250420</rss><pss>243778</pss><private_dirty>243432</private_dirty><swap>0</swap><real_memory>243432</real_memory><vmsize>508812</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.rsfSWOAhX745aaoInmLK1MzuYa78VkUq42qN3xjsAMerbI0dFVqQuxxR5RfFkEk</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:34719</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>31740</pid><sticky_session_id>319550609</sticky_session_id><gupid>1734e51-JfajNuzuwH</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>4</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034327140536</spawn_start_time><spawn_end_time>1460034333080673</spawn_end_time><last_used>1460034496324808</last_used><last_used_desc>13s ago</last_used_desc><uptime>2m 56s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>249476</rss><pss>243018</pss><private_dirty>242696</private_dirty><swap>0</swap><real_memory>242696</real_memory><vmsize>508088</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vOcXkYtmRMcbBEd0rT5pO1MltqNMovOaXBGrsJ1H3e4a0bKcinLAucv16ScbgqL</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:57264</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>827</pid><sticky_session_id>554078224</sticky_session_id><gupid>1734e52-1gkZfJ0Zu3</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>1</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034400055334</spawn_start_time><spawn_end_time>1460034406290173</spawn_end_time><last_used>1460034496368837</last_used><last_used_desc>13s ago</last_used_desc><uptime>1m 43s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>5</cpu><rss>253088</rss><pss>246906</pss><private_dirty>246612</private_dirty><swap>0</swap><real_memory>246612</real_memory><vmsize>512124</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.b9IDrQYx6DofvHH7Im6zNDO0JKIYgMNzNifqj1fWNmWkbEw8TJfICDkUT96Um6i</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:38925</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>2042</pid><sticky_session_id>2137707450</sticky_session_id><gupid>1734e54-HD27bHZrdu</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034496387206</spawn_start_time><spawn_end_time>1460034502258439</spawn_end_time><last_used>1460034502258439</last_used><last_used_desc>7s ago</last_used_desc><uptime>7s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>62</cpu><rss>248908</rss><pss>242784</pss><private_dirty>242496</private_dirty><swap>0</swap><real_memory>242496</real_memory><vmsize>507892</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.HRWUo7qw72cO8lgHyB9thG8NPhItlbP78ZFZwwRNSUlWRdUFe66oBMgBAlZ2Iqz</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:56086</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup><supergroup><name>/srv/assets/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>9</capacity_used><secret>Dsim6dBNi9wdsdKN</secret><group default="true"><name>/srv/assets/current/public</name><component_name>/srv/assets/current/public</component_name><app_root>/srv/assets/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>jPYGpGi4vaz1nTBaMPuR</uuid><enabled_process_count>9</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>9</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>Dsim6dBNi9wdsdKN</secret><api_key>Dsim6dBNi9wdsdKN</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/assets/current</app_root><app_group_name>/srv/assets/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-1.9.3-p551/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/assets/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>direct</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-1.9.3-p551/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>Dsim6dBNi9wdsdKN</api_key><min_processes>5</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>28611</pid><sticky_session_id>2074224666</sticky_session_id><gupid>1734e4e-QTafa2yv3Q</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>2402</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034161309905</spawn_start_time><spawn_end_time>1460034167194438</spawn_end_time><last_used>1460034509117154</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 42s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>16</cpu><rss>108848</rss><pss>100652</pss><private_dirty>100032</private_dirty><swap>0</swap><real_memory>100032</real_memory><vmsize>668844</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.Ezcbpj598kBYY1Nx23d6YbTu9DYbNb0ggSkP6PnZOkPCbC1eopEaWG3GL8CIzSV</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:53306</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28801</pid><sticky_session_id>805413509</sticky_session_id><gupid>1734e4e-UX4AfGVK3G</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>1007</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034167195192</spawn_start_time><spawn_end_time>1460034171473155</spawn_end_time><last_used>1460034508833315</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 38s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>8</cpu><rss>109172</rss><pss>101108</pss><private_dirty>100488</private_dirty><swap>0</swap><real_memory>100488</real_memory><vmsize>668596</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.P3yDxLQa3FPOgtnT6DdkddSyIATn74vty7jbA79iO3hFGZohrOqNGolw1PRkiX6</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:56174</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28982</pid><sticky_session_id>1886360986</sticky_session_id><gupid>1734e4e-8fVvFNqm1K</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>377</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034171474013</spawn_start_time><spawn_end_time>1460034175529790</spawn_end_time><last_used>1460034507518051</last_used><last_used_desc>2s ago</last_used_desc><uptime>5m 34s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>106564</rss><pss>98382</pss><private_dirty>97768</private_dirty><swap>0</swap><real_memory>97768</real_memory><vmsize>668596</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vHm6lYRZbaNLC5T9028I4E40pxzTiGBUrwTtXFVVHMW2KScSbCzlTO9P93qxZyX</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:57738</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29163</pid><sticky_session_id>887155047</sticky_session_id><gupid>1734e4e-qwyqMZyWDX</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>117</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034175530525</spawn_start_time><spawn_end_time>1460034179709612</spawn_end_time><last_used>1460034502552717</last_used><last_used_desc>7s ago</last_used_desc><uptime>5m 30s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>105116</rss><pss>96870</pss><private_dirty>96256</private_dirty><swap>0</swap><real_memory>96256</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.3iKloSxHg3o1zsXxHDNmfO7R4zsiJIW5ofJqh2WxL3atRqAZXMqeqY7X1s5QxD6</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:46674</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29285</pid><sticky_session_id>1408309555</sticky_session_id><gupid>1734e4e-zZzmQ2Yto6</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>36</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034179710273</spawn_start_time><spawn_end_time>1460034183913598</spawn_end_time><last_used>1460034485535565</last_used><last_used_desc>24s ago</last_used_desc><uptime>5m 26s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>101504</rss><pss>93503</pss><private_dirty>92892</private_dirty><swap>0</swap><real_memory>92892</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.srSFVjllvEcAqCt0azwfy5WHszhZW5hFt58HMyY1X2Fxwt9VGjEl64b7XZ4JgVx</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:36250</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30831</pid><sticky_session_id>174120964</sticky_session_id><gupid>1734e50-GI0n5QzHcj</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>10</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034245237412</spawn_start_time><spawn_end_time>1460034249406292</spawn_end_time><last_used>1460034436293771</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>4m 20s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>95348</rss><pss>87415</pss><private_dirty>86804</private_dirty><swap>0</swap><real_memory>86804</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.3LYoSNTEa00I9urLaSdkRBPfbkcPjcRKbcOMqTABwvzc6iZwfzMC5QuBGkex1S4</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:60219</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>31272</pid><sticky_session_id>85928402</sticky_session_id><gupid>1734e50-vwc0v78uqi</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>4</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034290917686</spawn_start_time><spawn_end_time>1460034295205240</spawn_end_time><last_used>1460034436338969</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>3m 34s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>94564</rss><pss>86506</pss><private_dirty>85896</private_dirty><swap>0</swap><real_memory>85896</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.dsl7lI9t1BsD690Dx75XRmhL59RdlJd1SoFUroEI7lFm1aNfy642f9SnXuIVUVG</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:47879</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>32491</pid><sticky_session_id>1699195757</sticky_session_id><gupid>1734e51-Jmn8diebz9</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>2</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034351156946</spawn_start_time><spawn_end_time>1460034356056661</spawn_end_time><last_used>1460034436340955</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>2m 33s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>94872</rss><pss>86667</pss><private_dirty>86052</private_dirty><swap>0</swap><real_memory>86052</real_memory><vmsize>668608</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.qOzNO5gAUoj362CkPmVZxSE2hTOIW7a7RQpyeMMQi0FuoIkn5bf2oTr0n7srjfN</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:50087</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>626</pid><sticky_session_id>943456820</sticky_session_id><gupid>1734e52-1tbfWs7zwi</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034391026230</spawn_start_time><spawn_end_time>1460034395124936</spawn_end_time><last_used>1460034395124936</last_used><last_used_desc>1m 54s ago</last_used_desc><uptime>1m 54s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>88492</rss><pss>80350</pss><private_dirty>79740</private_dirty><swap>0</swap><real_memory>79740</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.gQE0TfjVlWu1ixS7ZzlKiNb5mC6KuH5MiA78V7Cc4j2APW5CyDSFgplUeSb4jjO</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:50511</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>

--------------------------------------
[ pid=28580 ] Open files and file descriptors:
COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME
Passenger 28580 root  cwd    DIR                9,2     4096          2 /
Passenger 28580 root  rtd    DIR                9,2     4096          2 /
Passenger 28580 root  txt    REG                9,2  4358856  103024733 /usr/lib/passenger/support-binaries/PassengerAgent (deleted)
Passenger 28580 root  mem    REG                9,2    47712   89657060 /lib/x86_64-linux-gnu/libnss_files-2.19.so
Passenger 28580 root  mem    REG                9,2    47760   89656978 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
Passenger 28580 root  mem    REG                9,2    97296   89656837 /lib/x86_64-linux-gnu/libnsl-2.19.so
Passenger 28580 root  mem    REG                9,2    39824   89656836 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
Passenger 28580 root  mem    REG                9,2    43368   89656834 /lib/x86_64-linux-gnu/libcrypt-2.19.so
Passenger 28580 root  mem    REG                9,2   754880  101588317 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
Passenger 28580 root  mem    REG                9,2   295816  101581226 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
Passenger 28580 root  mem    REG                9,2    56768  101581339 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
Passenger 28580 root  mem    REG                9,2   166040  101581247 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
Passenger 28580 root  mem    REG                9,2    30944  101581413 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
Passenger 28580 root  mem    REG                9,2    85680  101581370 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
Passenger 28580 root  mem    REG                9,2   204104  101581316 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
Passenger 28580 root  mem    REG                9,2   657712  101581350 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
Passenger 28580 root  mem    REG                9,2   552584  101581262 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
Passenger 28580 root  mem    REG                9,2    35856  101581256 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
Passenger 28580 root  mem    REG                9,2    14256   89656478 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
Passenger 28580 root  mem    REG                9,2    18416   89656530 /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
Passenger 28580 root  mem    REG                9,2   268368  101581328 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
Passenger 28580 root  mem    REG                9,2    80256  101581381 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
Passenger 28580 root  mem    REG                9,2   251872  101581223 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
Passenger 28580 root  mem    REG                9,2   109296  101581202 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
Passenger 28580 root  mem    REG                9,2   101240   89656610 /lib/x86_64-linux-gnu/libresolv-2.19.so
Passenger 28580 root  mem    REG                9,2    43672  101581248 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
Passenger 28580 root  mem    REG                9,2    14592   89656723 /lib/x86_64-linux-gnu/libcom_err.so.2.1
Passenger 28580 root  mem    REG                9,2   186824  101584726 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
Passenger 28580 root  mem    REG                9,2   831616  101581346 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
Passenger 28580 root  mem    REG                9,2    14664   89656831 /lib/x86_64-linux-gnu/libdl-2.19.so
Passenger 28580 root  mem    REG                9,2   524704   89656514 /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
Passenger 28580 root  mem    REG                9,2   775112  101580953 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
Passenger 28580 root  mem    REG                9,2   323056  101581024 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
Passenger 28580 root  mem    REG                9,2    59640  101581026 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
Passenger 28580 root  mem    REG                9,2   290520  101581220 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
Passenger 28580 root  mem    REG                9,2  1934624   89657205 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Passenger 28580 root  mem    REG                9,2   387256   89657204 /lib/x86_64-linux-gnu/libssl.so.1.0.0
Passenger 28580 root  mem    REG                9,2   105696  101581349 /usr/lib/x86_64-linux-gnu/librtmp.so.0
Passenger 28580 root  mem    REG                9,2   207128  101581246 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
Passenger 28580 root  mem    REG                9,2  1840928   89657054 /lib/x86_64-linux-gnu/libc-2.19.so
Passenger 28580 root  mem    REG                9,2    90160   89656458 /lib/x86_64-linux-gnu/libgcc_s.so.1
Passenger 28580 root  mem    REG                9,2  1071552   89656829 /lib/x86_64-linux-gnu/libm-2.19.so
Passenger 28580 root  mem    REG                9,2   979056  101585650 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
Passenger 28580 root  mem    REG                9,2   100728   89656575 /lib/x86_64-linux-gnu/libz.so.1.2.8
Passenger 28580 root  mem    REG                9,2   422696  101584867 /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
Passenger 28580 root  mem    REG                9,2   141574   89656975 /lib/x86_64-linux-gnu/libpthread-2.19.so
Passenger 28580 root  mem    REG                9,2    31792   89656984 /lib/x86_64-linux-gnu/librt-2.19.so
Passenger 28580 root  mem    REG                9,2   149120   89656976 /lib/x86_64-linux-gnu/ld-2.19.so
Passenger 28580 root    0u   CHR              136,2      0t0          5 /dev/pts/2 (deleted)
Passenger 28580 root    1w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    2w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    3u  unix 0xffff8807ee4d3800      0t0 3456622232 socket
Passenger 28580 root    5w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    8r  FIFO                0,8      0t0 3456625344 pipe
Passenger 28580 root    9w  FIFO                0,8      0t0 3456625344 pipe
Passenger 28580 root   10r  FIFO                0,8      0t0 3456625345 pipe
Passenger 28580 root   11w  FIFO                0,8      0t0 3456625345 pipe
Passenger 28580 root   12r  FIFO                0,8      0t0 3456625346 pipe
Passenger 28580 root   13w  FIFO                0,8      0t0 3456625346 pipe
Passenger 28580 root   14u  unix 0xffff8800d309ad80      0t0 3456625347 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   15u  unix 0xffff8800d309fa80      0t0 3456625348 /tmp/passenger.1uP1fgj/agents.s/core_api
Passenger 28580 root   16r   CHR                1,9      0t0       1034 /dev/urandom
Passenger 28580 root   17u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   18u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   19r  FIFO                0,8      0t0 3456618077 pipe
Passenger 28580 root   20w  FIFO                0,8      0t0 3456618077 pipe
Passenger 28580 root   21u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   22r  FIFO                0,8      0t0 3456618078 pipe
Passenger 28580 root   23w  FIFO                0,8      0t0 3456618078 pipe
Passenger 28580 root   24u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   25u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   26u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   27u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   28u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   29u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   30r  FIFO                0,8      0t0 3456618079 pipe
Passenger 28580 root   31w  FIFO                0,8      0t0 3456618079 pipe
Passenger 28580 root   32u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   33u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   34u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   35u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   36r  FIFO                0,8      0t0 3456618080 pipe
Passenger 28580 root   37w  FIFO                0,8      0t0 3456618080 pipe
Passenger 28580 root   38u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   39u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   40u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   41u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   42r  FIFO                0,8      0t0 3456618081 pipe
Passenger 28580 root   43w  FIFO                0,8      0t0 3456618081 pipe
Passenger 28580 root   44u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   45u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   46u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   47u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   48r  FIFO                0,8      0t0 3456618082 pipe
Passenger 28580 root   49w  FIFO                0,8      0t0 3456618082 pipe
Passenger 28580 root   50u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   51u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   52u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   53u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   54r  FIFO                0,8      0t0 3456618083 pipe
Passenger 28580 root   55w  FIFO                0,8      0t0 3456618083 pipe
Passenger 28580 root   56u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   57u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   58u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   59u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   60r  FIFO                0,8      0t0 3456618084 pipe
Passenger 28580 root   61w  FIFO                0,8      0t0 3456618084 pipe
Passenger 28580 root   62u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   63u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   64u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   65u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   66r  FIFO                0,8      0t0 3456618085 pipe
Passenger 28580 root   67w  FIFO                0,8      0t0 3456618085 pipe
Passenger 28580 root   68u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   69u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   70u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   71u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   72r  FIFO                0,8      0t0 3456618086 pipe
Passenger 28580 root   73w  FIFO                0,8      0t0 3456618086 pipe
Passenger 28580 root   74u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   75u  unix 0xffff880306b0e900      0t0 3456742629 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   76u  unix 0xffff880306b0bb80      0t0 3456743665 socket
Passenger 28580 root   77u  unix 0xffff8800d309d780      0t0 3456625356 socket
Passenger 28580 root   78r  FIFO                0,8      0t0 3456625357 pipe
Passenger 28580 root   79u  unix 0xffff8801dda2c280      0t0 3456742625 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   81u  unix 0xffff880789b8fa80      0t0 3456625733 socket
Passenger 28580 root   82r  FIFO                0,8      0t0 3456625734 pipe
Passenger 28580 root   84u  unix 0xffff8800d3098380      0t0 3456672132 socket
Passenger 28580 root   85u  unix 0xffff8801dda2f700      0t0 3456741578 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   86u  unix 0xffff8800d309f380      0t0 3456629440 socket
Passenger 28580 root   87r  FIFO                0,8      0t0 3456629441 pipe
Passenger 28580 root   88u  unix 0xffff88077dcb4980      0t0 3456682879 socket
Passenger 28580 root   89u  unix 0xffff8802efc0d780      0t0 3456742627 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   90u  unix 0xffff88077dcb2300      0t0 3456653127 socket
Passenger 28580 root   91r  FIFO                0,8      0t0 3456653128 pipe
Passenger 28580 root   92u  unix 0xffff8807ee4d3f00      0t0 3456741571 socket
Passenger 28580 root   93u  unix 0xffff88077c7bb480      0t0 3456631060 socket
Passenger 28580 root   94u  unix 0xffff8807f0bd8700      0t0 3456740782 socket
Passenger 28580 root   96u  unix 0xffff88077dcb6580      0t0 3456642019 socket
Passenger 28580 root   97r  FIFO                0,8      0t0 3456642020 pipe
Passenger 28580 root   98u  unix 0xffff8801dda2ad80      0t0 3456704674 socket
Passenger 28580 root  100u  unix 0xffff880789841180      0t0 3456668328 socket
Passenger 28580 root  101u  unix 0xffff8800d3099f80      0t0 3456646476 socket
Passenger 28580 root  102r  FIFO                0,8      0t0 3456646477 pipe
Passenger 28580 root  103r  FIFO                0,8      0t0 3456668329 pipe
Passenger 28580 root  104u  unix 0xffff8800d3098e00      0t0 3456656667 socket
Passenger 28580 root  105r  FIFO                0,8      0t0 3456656668 pipe
Passenger 28580 root  107u  unix 0xffff8807f0bdd780      0t0 3456687062 socket
Passenger 28580 root  108r  FIFO                0,8      0t0 3456687063 pipe
Passenger 28580 root  109u  unix 0xffff8807ed458a80      0t0 3456703668 socket
Passenger 28580 root  110r  FIFO                0,8      0t0 3456703669 pipe
Passenger 28580 root  113u  unix 0xffff8800d309e580      0t0 3456681461 socket
Passenger 28580 root  114r  FIFO                0,8      0t0 3456681462 pipe
Passenger 28580 root  118u  unix 0xffff88077dcb3f00      0t0 3456641498 socket
Passenger 28580 root  121u  unix 0xffff8801dda2cd00      0t0 3456636240 socket
Passenger 28580 root  122r  FIFO                0,8      0t0 3456636241 pipe
Passenger 28580 root  123u  unix 0xffff8807f0bdd080      0t0 3456640044 socket
Passenger 28580 root  124r  FIFO                0,8      0t0 3456640045 pipe
Passenger 28580 root  129u  unix 0xffff88077c7b9880      0t0 3456706106 socket
Passenger 28580 root  130r  FIFO                0,8      0t0 3456706107 pipe
Passenger 28580 root  132u  unix 0xffff8802efc09c00      0t0 3456639198 socket
Passenger 28580 root  134u  unix 0xffff8807f0bd8000      0t0 3456737094 socket
Passenger 28580 root  135r  FIFO                0,8      0t0 3456737095 pipe
Passenger 28580 root  136u  unix 0xffff8800d3303480      0t0 3456627825 socket
Passenger 28580 root  137u  unix 0xffff8801dda2de80      0t0 3456618329 socket
Passenger 28580 root  150r  FIFO                0,8      0t0 3456618330 pipe
Passenger 28580 root  152u  unix 0xffff88077dcb3800      0t0 3456633285 socket
Passenger 28580 root  159u  unix 0xffff8800d309c980      0t0 3456629737 socket
Passenger 28580 root  161r  FIFO                0,8      0t0 3456629738 pipe
Passenger 28580 root  168u  unix 0xffff8801dda2f000      0t0 3456618369 socket
Passenger 28580 root  170r  FIFO                0,8      0t0 3456618370 pipe
Passenger 28580 root  176u  unix 0xffff8807899c1f80      0t0 3456626199 socket
Passenger 28580 root  178u  unix 0xffff880789078a80      0t0 3456618109 socket
Passenger 28580 root  179r  FIFO                0,8      0t0 3456618110 pipe
Passenger 28580 root  184u  unix 0xffff88077dcb7a80      0t0 3456633126 socket
Passenger 28580 root  189u  unix 0xffff8807899c4980      0t0 3456626201 socket
Passenger 28580 root  190r  FIFO                0,8      0t0 3456626202 pipe
Passenger 28580 root  191u  unix 0xffff8807899c6200      0t0 3456626489 socket
Passenger 28580 root  192r  FIFO                0,8      0t0 3456626490 pipe
Passenger 28580 root  195u  unix 0xffff8802efc0ec80      0t0 3456628108 socket
Passenger 28580 root  196r  FIFO                0,8      0t0 3456628109 pipe
Passenger 28580 root  197u  unix 0xffff8807899c4d00      0t0 3456632050 socket
--------------------------------------
[ pid=28580 ] Dumping a more detailed backtrace with crash-watch...
*** ERROR ***: 'gdb' isn't installed. Please install it first.
       Debian/Ubuntu: sudo apt-get install gdb
RedHat/CentOS/Fedora: sudo yum install gdb
            Mac OS X: please install the Developer Tools or XCode
             FreeBSD: use the devel/gdb port

or
passenger.crash.log.zip

Please let me know if additional information is required.

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2016

Thanks for reporting. It would be very helpful if we could get a crash log with gdb installed (see note at the bottom of the trace), and the debug symbols package installed (apt-get install passenger-dbg, doesn't affect production use, just more info in the trace).

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Apr 11, 2016

I've installed gdb and passenger-dbg packages but I still get the same message in the crash log:

*** ERROR ***: 'gdb' isn't installed. Please install it first.

Even restarted nginx..

Here's the new crash log, maybe there is something new in there:

passenger.crash.log.zip

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2016

Thanks; the second log shows it occurred both on a POST as wel as a GET request, so it's a bit more mysterious.

Regarding gdb, if it can be found when you enter gdb on the command line then it might be that Nginx filtered out the PATH to it so that Passenger can't find it. You could solve that by adding env PATH=<PATH>; in your Nginx config, whereby <PATH> is for example what you get out ofecho $PATH.

@remomueller

This comment has been minimized.

Copy link

commented Apr 11, 2016

I've started getting crashes (502) as well after the upgrade to 5.0.27, here's an attached gist: https://gist.github.com/remomueller/6e6a97364c755b231f2cded76e6054b4

The error message is similar to the one closed a while back in #1193, so it may be related?

Mac OS X 10.11.4
Nginx 1.8.1
RVM 1.27.0
Ruby 2.3.0
Rails 4.2.6 and 5.0.0.beta3

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2016

@remomueller I think your issue is completely different and not related to Passenger, but instead to Xcode, see: #1780.

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Apr 12, 2016

Thanks OnixGH for the tip. I've managed to enabled gdb. Here's the new crash log:

passenger.crash.log.zip

@remomueller

This comment has been minimized.

Copy link

commented Apr 12, 2016

Hi OnixGH, the 7.2.1 workaround from #1780 worked for me, thanks for pointing me in the right direction!

@QuinnyPig

This comment has been minimized.

Copy link

commented Apr 12, 2016

I can replicate this on a Ubuntu 14.04 environment. Note that OS X was not involved here-- I'm using the phusion-passenger apt repo.

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2016

@QuinnyPig is there any way you can provide a sample app that demonstrates the issue (for example using a VM or a DigitalOcean Droplet?). You can also mail it to us if it's not suitable for public sharing.

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Apr 28, 2016

We haven't been able to reproduce this yet, so we've added extra diagnostic code (1d7a315) and a few robustness improvements (c148d61) in the 5.0.28 release. Can @VvanGemert or @QuinnyPig give that version a try and provide a crashlog?

@VvanGemert

This comment has been minimized.

Copy link
Author

commented May 2, 2016

Hi OnixGH, thanks for the heads up. I've tried version 5.0.28 and still having the same issue. Here's the crash log:

crash.log.zip

Before the crash log it logged these lines also in nginx/error.log:

[ 2016-05-02 08:31:08.8130 467/7f3cad7fa700 Mem/mbuf.cpp:236 ]: [BUG] Assertion failed:
STAILQ_NEXT(mbuf_block, next) == NULL
mbuf_block: 0x7f3c8403f0b0
mbuf_block.magic: 3735928559
mbuf_block.next: 0x7f3c8403c780
mbuf_block.start: 0x7f3c8403eee0
mbuf_block.end: 0x7f3c8403f0b0
mbuf_block.refcount: 0
mbuf_block.offset: 0
mbuf_block.pool: 0x278f018
mbuf_block.pool.nfree_mbuf_blockq: 286
mbuf_block.pool.nactive_mbuf_blockq: 5
mbuf_block.pool.mbuf_block_chunk_size: 512
mbuf_block.pool.mbuf_block_offset: 464

@FooBarWidget

This comment has been minimized.

Copy link
Member

commented May 13, 2016

Here is my analysis so far:

  • The signature of the mbuf_block is intact. This implies that the problem is probably not memory corruption.
  • STAILQ(mbuf_block, next) is not NULL. If the cause is not memory corruption then it's probably because an mbuf_block, which was already put on the freelist, was being dereferenced.
  • Every crash log shows that the origin of the crash is psg_lstr_deinit(&resp->bodyCacheBuffer). I have performed a code review on how the Core Controller uses bodyCacheBuffer. I have found no issues that would explain why a deinit would crash.
  • I have also performed a code review of the mbuf and LString subsystems. I have found no issues.

I suspect that the crash is a symptom of another problem: maybe the mbufs which are added to the bodyCacheBuffer are prematurely put back to the freelist. But this should not be possible because the mbuf API should prevent such mistakes.

So I have not been able to solve the problem yet, but I have added extra guards and debugging code (in the improve_mbuf_error_detection branch) for spotting usage of prematurely freed mbufs. Maybe that will take us closer to the problem.

@twigbranch

This comment has been minimized.

Copy link

commented May 18, 2016

We are having the same issue.

On centos 7, passenger 5.0.28, apache 2.4, rvm and running four virtual hosts: two with rails 3.09 / ruby 2.1.1 and two with rails 2.3.18 / ree-1.8.7.

One of the 2.3.18 / ree-1.8.7 apps seems to be causing the error to be reported in the main error log:
Mem/mbuf.cpp:236 ]: [BUG] Assertion failed: STAILQ_NEXT(mbuf_block, next) == NULL

Also, in spite of this error, the requests that generate it result in a page that's properly rendered by the browser - though on occasion, maybe 5-10% of the time there is a 500 error received by the browser and reported in the virtual host log file, for ANY of the virtual hosts. The error message is:
[Tue May 17 05:07:15.459332 2016] [core:error] [pid 17607] [client xx.xx.xx.xx:55400] End of script output before headers:

More about the 2.3.18 / ree-1.8.7 virtual host that is causing the assertion failed error:
It had been upgraded from rails 2.1 some time ago, and as a result had a residual application.rb (which had been copied to the proper application_controller.rb). That residual application.rb had never appeared to be an issue before. With both the application.rb and the application_controller.rb in the app/controllers folder, the "Assertion failed" error would occur on EVERY request (except for requests for static files under public/). When we removed the residual application.rb, the "Assertion failed" error occured only on certain requests: it seems to appear whenever the locale is specified in the url i.e. /en/, /fr/, etc. (and processed accordingly), and on POSTs. Note that the other 2.3.18 / ree-1.8.7 virtual host also had the residual application.rb, but it did not generate the error on any request (there was no locale routing, detection (in application.rb), switching (setting cookie hash and redirecting) though and request processing might be simpler there). There may be other cases where the "Assertion failed" error occurs that I have not seen yet.

Incidentally, the passenger error_log reports that a "Crash log" is saved to /var/tmp, yet there are no crash logs there. I tried change the ownership of /var/tmp to apache:apache but that didn't have a difference.

@FooBarWidget

This comment has been minimized.

Copy link
Member

commented May 22, 2016

@twigbranch Are you able to reproduce the assertion error semi-reliably? We are having trouble reproducing the problem so it's hard for us to fix. Can you help us reproduce it, for example by supplying a staging server or VM with instructions to reproduce the issue?

@twigbranch

This comment has been minimized.

Copy link

commented May 22, 2016

@FooBarWidget FWIW, downgrading to 5.0.26, the original poster reported, resolved the issue. I'll look into a VM, but it could take some time.

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2016

@VvanGemert patch file: https://gist.githubusercontent.com/OnixGH/e33d7ee4b7f3792a99556f2ee5c705d0/raw/12536aa7a96ffb358827c5080a9322757c8fbbec/mbuf_5028.patch

You can apply with patch -p1 < mbuf_5028.patch onto a Passenger (Open Source) 5.0.28 installation root (passenger-config --root). Then recompile the necessary parts by running rake nginx.

If you have any trouble, we would need the logs at log level 6 (or 7, but we haven't seen it reproduced at that level yet). You can revert the patch with -R in the patch line (and rake again).

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 13, 2016

@OnixGH Thanks. It will take me some time to apply the patch and test it. I'm currently using the APT repo version of Passenger and need to install the gem version or build the source I think. So I will need to reconfigure nginx to test the patch. I will try tomorrow.

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2016

Any luck yet, @VvanGemert?

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

@OnixGH I've checked out the repo and switched branch. Build the gem and installed it, configured Nginx and got this error when it tries to spawn passenger processes:

Message from application: no such file to load -- /home/piedpiper/.rvm/gems/ruby-2.3.0/gems/passenger-5.0.28/src/ruby_supportlib/phusion_passenger/vendor/union_station_hooks_core/lib/union_station_hooks_core (LoadError)

[ 2016-06-15 12:51:58.4068 2280/7fc7027fc700 Ser/FdSourceChannel.h:76 ]: FdSourceChannel: checkout new mbuf (thread signature = 0x2784620)

[ 2016-06-15 12:51:58.4069 2280/7fc7027fc700 Ser/FdSourceChannel.h:76 ]: FdSourceChannel: checkout new mbuf (thread signature = 0x2784620)

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2016

@VvanGemert passenger has submodules, you need to do a recursive checkout to get those too.

@FooBarWidget

This comment has been minimized.

Copy link
Member

commented Jun 15, 2016

In other words: git submodule update --init --recursive

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

@OnixGH @FooBarWidget That makes sense. Rebuilding the gem and installing it..

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

@OnixGH @FooBarWidget Ok I'm running the version on one server in production now. The nginx error log is spammed with these messages, is that ok?

[ 2016-06-15 13:29:12.6443 4637/7f65177fe700 Ser/FdSourceChannel.h:76 ]: FdSourceChannel: checkout new mbuf (thread signature = 0x2934620)

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2016

@VvanGemert yep, the patch contains some vital debugging additions to help pinpoint any potential remaining issues (just in case).

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

It usually crashes within a 5 minute range. I haven't seen any crashes yet. So far so good.

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

@OnixGH It's not crashing anymore! Thanks for the support. I leave it running through the night and it will do about a million requests before the morning. If it doesn't crash, we can consider it fixed.

@VvanGemert

This comment has been minimized.

Copy link
Author

commented Jun 16, 2016

@OnixGH Everything is ok. There were no crashes last night. I believe we can consider this fixed. I'm reverting back to 5.0.26 in production and wait for a new release. Thanks guys / girls for helping me out. Passenger is an awesome product and has been fueling our site since 2008! 👍 👍

@OnixGH OnixGH added this to the 5.0.29 milestone Jun 16, 2016

@OnixGH

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2016

@VvanGemert: Super duper, thanks for testing! 5.0.29 is on the way, nearly done :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.