Debugging applications that are killed with killjava.sh #79

Closed
trastle opened this Issue Aug 6, 2014 · 13 comments

Comments

Projects
None yet
3 participants
@trastle

trastle commented Aug 6, 2014

I have an application which is repeatedly failing when we push it to CF using the current master of the buildpack. The events are similar to the following:

2014-08-06T11:37:51.00+0100   audit.app.update   USER   state: STOPPED
2014-08-06T11:32:34.00+0100   audit.app.update   USER   instances: 1, memory: 2048
2014-08-06T11:02:47.00+0100   audit.app.update   admin                            state: STARTED
2014-08-06T11:02:45.00+0100   audit.app.update   admin                            state: STOPPED
2014-08-06T10:52:50.00+0100   app.crash          APP                         index: 0, reason: CRASHED, exit_description: app instance exited, exit_status: 137
2014-08-06T10:51:37.00+0100   audit.app.update   USER   state: STARTED
2014-08-06T10:46:38.00+0100   audit.app.update   USER   instances: 1, memory: 2048

I can see from the applogs the kill script is being called due to an OOM event:

2014-08-06T12:04:46.40+0100 [App/0]   OUT # java.lang.OutOfMemoryError: Java heap space
2014-08-06T12:04:46.40+0100 [App/0]   OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2014-08-06T12:04:46.40+0100 [App/0]   OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2014-08-06T12:04:46.46+0100 [App/0]   ERR /bin/bash: line 35:    33 Killed                  ( $PWD/.java-buildpack/open_jdk_jre/bin/java -cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx1536M -Xms1536M -XX:MaxPermSize=209715K -XX:PermSize=209715K -Xss1M -javaagent:$PWD/.java-buildpack/app_dynamics_agent/javaagent.jar -Dappdynamics.agent.applicationName='APP' -Dappdynamics.agent.tierName='CloudFoundry' -Dappdynamics.agent.nodeName=$(expr "$VCAP_APPLICATION" : '.*instance_id[": ]*"\([a-z0-9]\+\)".*') -Dappdynamics.agent.accountAccessKey=xx -Dappdynamics.agent.accountName=xx -Dappdynamics.controller.hostName=xx -Dappdynamics.controller.port=443 -Dappdynamics.controller.ssl.enabled=true xx ) > >(tee $DROPLET_BASE_DIR/logs/stdout.log) 2> >(tee $DROPLET_BASE_DIR/logs/stderr.log >&2)
https://github.com/cloudfoundry/java-buildpack/blob/6f502e7b2355c16d45b0d5e2c7e572ea01e6df39/resources/open_jdk_jre/bin/killjava.sh#L23-L25

Within that kill script you guys have added some diagnostics 👍 however I don't know how to read them as they are not pumped through loggregator and the container is then removed.

function log {
echo "$(date +%FT%T.%2N%z) FATAL $1" >> $LOG_FILE
}

How do I get at these logs to find out more about what is going on?

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Aug 6, 2014

Member

Yup, this is a known problem. The killjava.sh was created when crashlogs still existed and hasn't been handled since the move to loggergator. The problem we have is that the loggergator only listens to stdout/stderr of the "application" process. As of recently, we've had no way of getting that information to loggergator but I know a lot of new work has gone into it, so maybe we can now. We've had an issue outstanding for nearly a year but haven't been doing much with it. I'll move it back up in priority now and we'll get it onto the backlog.

Member

nebhale commented Aug 6, 2014

Yup, this is a known problem. The killjava.sh was created when crashlogs still existed and hasn't been handled since the move to loggergator. The problem we have is that the loggergator only listens to stdout/stderr of the "application" process. As of recently, we've had no way of getting that information to loggergator but I know a lot of new work has gone into it, so maybe we can now. We've had an issue outstanding for nearly a year but haven't been doing much with it. I'll move it back up in priority now and we'll get it onto the backlog.

@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Aug 6, 2014

Thanks @nebhale would be great to know if this is possible. I am stumped by this one at the moment, can't even pastebin the data as the URL returned is random!

trastle commented Aug 6, 2014

Thanks @nebhale would be great to know if this is possible. I am stumped by this one at the moment, can't even pastebin the data as the URL returned is random!

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Aug 6, 2014

Member

I think (based on a recent discussion in vcap-dev) that the crashlogs functionality is still exposed via the REST API. Let me do some digging on the list to see if I can find that reference which might give you what you want, right now.

Member

nebhale commented Aug 6, 2014

I think (based on a recent discussion in vcap-dev) that the crashlogs functionality is still exposed via the REST API. Let me do some digging on the list to see if I can find that reference which might give you what you want, right now.

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Sep 30, 2014

Member

@trastle I've gotten some movement on this today and wanted to know what metrics you'd like to see come out of this? At the moment, I've got this:

2014-09-30T16:48:41.93+0100 [App/0]   OUT INFO  Exhausting 1.4 GiB of memory
2014-09-30T16:48:42.07+0100 [App/0]   OUT INFO  101.4 MiB of 1.4 GiB used
2014-09-30T16:48:42.37+0100 [App/0]   OUT INFO  477.9 MiB of 1.4 GiB used
2014-09-30T16:48:42.84+0100 [App/0]   OUT INFO  841.0 MiB of 1.4 GiB used
2014-09-30T16:48:43.21+0100 [App/0]   OUT INFO  1.2 GiB of 1.4 GiB used
2014-09-30T16:48:43.44+0100 [App/0]   OUT INFO  1.3 GiB of 1.4 GiB used
2014-09-30T16:48:44.26+0100 [App/0]   OUT #
2014-09-30T16:48:44.26+0100 [App/0]   OUT # java.lang.OutOfMemoryError: Java heap space
2014-09-30T16:48:44.26+0100 [App/0]   OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2014-09-30T16:48:44.26+0100 [App/0]   OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2014-09-30T16:48:44.93+0100 [App/0]   OUT Processes Before:
2014-09-30T16:48:44.93+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-09-30T16:48:44.93+0100 [App/0]   OUT root         1     0  0 15:48 ?        00:00:00 wshd: 18476n61l3k    
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        31     1  0 15:48 ?        00:00:00 /bin/bash
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        32    31 89 15:48 ?        00:00:26 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx1536M -Xms1536M -XX:MaxMetaspaceSize=209715K -XX:MetaspaceSize=209715K -Xss1M -Daccess.logging.enabled=false -Dhttp.port=62084 -Djava.endorsed.dirs=/home/vcap/app/.java-buildpack/tomcat/endorsed -classpath /home/vcap/app/.java-buildpack/tomcat/bin/bootstrap.jar:/home/vcap/app/.java-buildpack/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/vcap/app/.java-buildpack/tomcat -Dcatalina.home=/home/vcap/app/.java-buildpack/tomcat -Djava.io.tmpdir=/home/vcap/app/.java-buildpack/tomcat/temp org.apache.catalina.startup.Bootstrap start
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        62    32  0 15:48 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        63    62  0 15:48 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        64    63  0 15:48 ?        00:00:00 ps -ef
2014-09-30T16:48:44.94+0100 [App/0]   OUT Processes After:
2014-09-30T16:48:44.94+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-09-30T16:48:44.94+0100 [App/0]   OUT root         1     0  0 15:48 ?        00:00:00 wshd: 18476n61l3k    
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        31     1  0 15:48 ?        00:00:00 /bin/bash
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        32    31 89 15:48 ?        00:00:26 [java] <defunct>
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        62    32  0 15:48 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        63    62  0 15:48 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        66    63  0 15:48 ?        00:00:00 ps -ef

Is that sufficient?

Member

nebhale commented Sep 30, 2014

@trastle I've gotten some movement on this today and wanted to know what metrics you'd like to see come out of this? At the moment, I've got this:

2014-09-30T16:48:41.93+0100 [App/0]   OUT INFO  Exhausting 1.4 GiB of memory
2014-09-30T16:48:42.07+0100 [App/0]   OUT INFO  101.4 MiB of 1.4 GiB used
2014-09-30T16:48:42.37+0100 [App/0]   OUT INFO  477.9 MiB of 1.4 GiB used
2014-09-30T16:48:42.84+0100 [App/0]   OUT INFO  841.0 MiB of 1.4 GiB used
2014-09-30T16:48:43.21+0100 [App/0]   OUT INFO  1.2 GiB of 1.4 GiB used
2014-09-30T16:48:43.44+0100 [App/0]   OUT INFO  1.3 GiB of 1.4 GiB used
2014-09-30T16:48:44.26+0100 [App/0]   OUT #
2014-09-30T16:48:44.26+0100 [App/0]   OUT # java.lang.OutOfMemoryError: Java heap space
2014-09-30T16:48:44.26+0100 [App/0]   OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2014-09-30T16:48:44.26+0100 [App/0]   OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2014-09-30T16:48:44.93+0100 [App/0]   OUT Processes Before:
2014-09-30T16:48:44.93+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-09-30T16:48:44.93+0100 [App/0]   OUT root         1     0  0 15:48 ?        00:00:00 wshd: 18476n61l3k    
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        31     1  0 15:48 ?        00:00:00 /bin/bash
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        32    31 89 15:48 ?        00:00:26 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx1536M -Xms1536M -XX:MaxMetaspaceSize=209715K -XX:MetaspaceSize=209715K -Xss1M -Daccess.logging.enabled=false -Dhttp.port=62084 -Djava.endorsed.dirs=/home/vcap/app/.java-buildpack/tomcat/endorsed -classpath /home/vcap/app/.java-buildpack/tomcat/bin/bootstrap.jar:/home/vcap/app/.java-buildpack/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/vcap/app/.java-buildpack/tomcat -Dcatalina.home=/home/vcap/app/.java-buildpack/tomcat -Djava.io.tmpdir=/home/vcap/app/.java-buildpack/tomcat/temp org.apache.catalina.startup.Bootstrap start
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        62    32  0 15:48 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        63    62  0 15:48 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.93+0100 [App/0]   OUT vcap        64    63  0 15:48 ?        00:00:00 ps -ef
2014-09-30T16:48:44.94+0100 [App/0]   OUT Processes After:
2014-09-30T16:48:44.94+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-09-30T16:48:44.94+0100 [App/0]   OUT root         1     0  0 15:48 ?        00:00:00 wshd: 18476n61l3k    
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        31     1  0 15:48 ?        00:00:00 /bin/bash
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        32    31 89 15:48 ?        00:00:26 [java] <defunct>
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        62    32  0 15:48 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        63    62  0 15:48 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-09-30T16:48:44.94+0100 [App/0]   OUT vcap        66    63  0 15:48 ?        00:00:00 ps -ef

Is that sufficient?

@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Oct 1, 2014

thanks @nebhale this is a HUGE improvement :-D

The only other thing I can think of off hand would be to include the number of file handles available & disk space before & after killing the app.

We have seen a few (maybe 5) instances where an app is being restarted after an OOM Exception which was actually caused by no more file handles being available within the warden container.

This is would be especially handy as it's quite tricky as a new CF dev to see how many file handles (and hence threads) are available to you in CF.

trastle commented Oct 1, 2014

thanks @nebhale this is a HUGE improvement :-D

The only other thing I can think of off hand would be to include the number of file handles available & disk space before & after killing the app.

We have seen a few (maybe 5) instances where an app is being restarted after an OOM Exception which was actually caused by no more file handles being available within the warden container.

This is would be especially handy as it's quite tricky as a new CF dev to see how many file handles (and hence threads) are available to you in CF.

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Oct 1, 2014

Member

Any suggestions on how to get that informatIon? I'll admit to being a novice with respect to those diagnostics.

Member

nebhale commented Oct 1, 2014

Any suggestions on how to get that informatIon? I'll admit to being a novice with respect to those diagnostics.

@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Oct 1, 2014

Sorry for the delayed response. I had to dig up my notes from debugging this issue a couple of months ago.

I have converted my notes into a blog post: http://blog.troyastle.com/2014/10/debugging-javalangoutofmemoryerror.html

This is probably a more detailed explanation that I can chuck in a github comment.

trastle commented Oct 1, 2014

Sorry for the delayed response. I had to dig up my notes from debugging this issue a couple of months ago.

I have converted my notes into a blog post: http://blog.troyastle.com/2014/10/debugging-javalangoutofmemoryerror.html

This is probably a more detailed explanation that I can chuck in a github comment.

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Oct 1, 2014

Member

No worries. The post looks great, are you OK with me adding just a ulimit -a and a df -h output into what's already there? I just want to make sure that you don't get more noise that you really want.

Member

nebhale commented Oct 1, 2014

No worries. The post looks great, are you OK with me adding just a ulimit -a and a df -h output into what's already there? I just want to make sure that you don't get more noise that you really want.

@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Oct 1, 2014

Good point On the noise side.

I believe that if an app is out of disk we will get a specific event in $cf events...? so there may be minimal value in doing the df -h. The ulimit -a would give us information that we have no visibility on at the moment.

trastle commented Oct 1, 2014

Good point On the noise side.

I believe that if an app is out of disk we will get a specific event in $cf events...? so there may be minimal value in doing the df -h. The ulimit -a would give us information that we have no visibility on at the moment.

@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Oct 1, 2014

That said I can see the df -h in now so I'd be very happy if it stayed.
More information is always helpful.

T.

trastle commented Oct 1, 2014

That said I can see the df -h in now so I'd be very happy if it stayed.
More information is always helpful.

T.

@nebhale

This comment has been minimized.

Show comment
Hide comment
@nebhale

nebhale Oct 1, 2014

Member

@trastle That look OK for you?

2014-10-01T14:08:33.79+0100 [App/0]   OUT INFO  Exhausting 1.4 GiB of memory
2014-10-01T14:08:33.90+0100 [App/0]   OUT INFO  105.3 MiB of 1.4 GiB used
2014-10-01T14:08:34.22+0100 [App/0]   OUT INFO  486.6 MiB of 1.4 GiB used
2014-10-01T14:08:34.81+0100 [App/0]   OUT INFO  860.6 MiB of 1.4 GiB used
2014-10-01T14:08:35.17+0100 [App/0]   OUT INFO  1.2 GiB of 1.4 GiB used
2014-10-01T14:08:36.31+0100 [App/0]   OUT #
2014-10-01T14:08:36.31+0100 [App/0]   OUT # java.lang.OutOfMemoryError: Java heap space
2014-10-01T14:08:36.31+0100 [App/0]   OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2014-10-01T14:08:36.31+0100 [App/0]   OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2014-10-01T14:08:37.01+0100 [App/0]   OUT Process Status (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT =======================
2014-10-01T14:08:37.01+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-10-01T14:08:37.01+0100 [App/0]   OUT root         1     0  0 13:08 ?        00:00:00 wshd: 1847628cl9b    
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        31     1  0 13:08 ?        00:00:00 /bin/bash
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        32    31 95 13:08 ?        00:00:27 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx1536M -Xms1536M -XX:MaxMetaspaceSize=209715K -XX:MetaspaceSize=209715K -Xss1M -Daccess.logging.enabled=false -Dhttp.port=62118 -Djava.endorsed.dirs=/home/vcap/app/.java-buildpack/tomcat/endorsed -classpath /home/vcap/app/.java-buildpack/tomcat/bin/bootstrap.jar:/home/vcap/app/.java-buildpack/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/vcap/app/.java-buildpack/tomcat -Dcatalina.home=/home/vcap/app/.java-buildpack/tomcat -Djava.io.tmpdir=/home/vcap/app/.java-buildpack/tomcat/temp org.apache.catalina.startup.Bootstrap start
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        62    32  9 13:08 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        63    62  0 13:08 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        64    63  0 13:08 ?        00:00:00 ps -ef
2014-10-01T14:08:37.01+0100 [App/0]   OUT ulimit (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT ===============
2014-10-01T14:08:37.01+0100 [App/0]   OUT core file size          (blocks, -c) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT data seg size           (kbytes, -d) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT scheduling priority             (-e) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT file size               (blocks, -f) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT pending signals                 (-i) 1024
2014-10-01T14:08:37.01+0100 [App/0]   OUT max locked memory       (kbytes, -l) 64
2014-10-01T14:08:37.01+0100 [App/0]   OUT max memory size         (kbytes, -m) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT open files                      (-n) 16384
2014-10-01T14:08:37.01+0100 [App/0]   OUT pipe size            (512 bytes, -p) 8
2014-10-01T14:08:37.01+0100 [App/0]   OUT POSIX message queues     (bytes, -q) 819200
2014-10-01T14:08:37.01+0100 [App/0]   OUT real-time priority              (-r) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT stack size              (kbytes, -s) 8192
2014-10-01T14:08:37.01+0100 [App/0]   OUT cpu time               (seconds, -t) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT max user processes              (-u) 512
2014-10-01T14:08:37.01+0100 [App/0]   OUT virtual memory          (kbytes, -v) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT file locks                      (-x) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT Free Disk Space (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT ========================
2014-10-01T14:08:37.01+0100 [App/0]   OUT Filesystem            Size  Used Avail Use% Mounted on
2014-10-01T14:08:37.01+0100 [App/0]   OUT tmpfs                  17G     0   17G   0% /dev/shm
2014-10-01T14:08:37.04+0100 [App/0]   OUT Process Status (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT ======================
2014-10-01T14:08:37.04+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-10-01T14:08:37.04+0100 [App/0]   OUT root         1     0  0 13:08 ?        00:00:00 wshd: 1847628cl9b    
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        31     1  0 13:08 ?        00:00:00 /bin/bash
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        32    31 95 13:08 ?        00:00:27 [java] <defunct>
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        62    32  9 13:08 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        63    62  0 13:08 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        68    63  0 13:08 ?        00:00:00 ps -ef
2014-10-01T14:08:37.04+0100 [App/0]   OUT ulimit (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT ==============
2014-10-01T14:08:37.04+0100 [App/0]   OUT core file size          (blocks, -c) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT data seg size           (kbytes, -d) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT scheduling priority             (-e) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT file size               (blocks, -f) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT pending signals                 (-i) 1024
2014-10-01T14:08:37.04+0100 [App/0]   OUT max locked memory       (kbytes, -l) 64
2014-10-01T14:08:37.04+0100 [App/0]   OUT max memory size         (kbytes, -m) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT open files                      (-n) 16384
2014-10-01T14:08:37.04+0100 [App/0]   OUT pipe size            (512 bytes, -p) 8
2014-10-01T14:08:37.04+0100 [App/0]   OUT POSIX message queues     (bytes, -q) 819200
2014-10-01T14:08:37.04+0100 [App/0]   OUT real-time priority              (-r) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT stack size              (kbytes, -s) 8192
2014-10-01T14:08:37.04+0100 [App/0]   OUT cpu time               (seconds, -t) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT max user processes              (-u) 512
2014-10-01T14:08:37.04+0100 [App/0]   OUT virtual memory          (kbytes, -v) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT file locks                      (-x) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT Free Disk Space (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT =======================
2014-10-01T14:08:37.04+0100 [App/0]   OUT Filesystem            Size  Used Avail Use% Mounted on
2014-10-01T14:08:37.04+0100 [App/0]   OUT tmpfs                  17G     0   17G   0% /dev/shm
Member

nebhale commented Oct 1, 2014

@trastle That look OK for you?

2014-10-01T14:08:33.79+0100 [App/0]   OUT INFO  Exhausting 1.4 GiB of memory
2014-10-01T14:08:33.90+0100 [App/0]   OUT INFO  105.3 MiB of 1.4 GiB used
2014-10-01T14:08:34.22+0100 [App/0]   OUT INFO  486.6 MiB of 1.4 GiB used
2014-10-01T14:08:34.81+0100 [App/0]   OUT INFO  860.6 MiB of 1.4 GiB used
2014-10-01T14:08:35.17+0100 [App/0]   OUT INFO  1.2 GiB of 1.4 GiB used
2014-10-01T14:08:36.31+0100 [App/0]   OUT #
2014-10-01T14:08:36.31+0100 [App/0]   OUT # java.lang.OutOfMemoryError: Java heap space
2014-10-01T14:08:36.31+0100 [App/0]   OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2014-10-01T14:08:36.31+0100 [App/0]   OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2014-10-01T14:08:37.01+0100 [App/0]   OUT Process Status (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT =======================
2014-10-01T14:08:37.01+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-10-01T14:08:37.01+0100 [App/0]   OUT root         1     0  0 13:08 ?        00:00:00 wshd: 1847628cl9b    
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        31     1  0 13:08 ?        00:00:00 /bin/bash
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        32    31 95 13:08 ?        00:00:27 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx1536M -Xms1536M -XX:MaxMetaspaceSize=209715K -XX:MetaspaceSize=209715K -Xss1M -Daccess.logging.enabled=false -Dhttp.port=62118 -Djava.endorsed.dirs=/home/vcap/app/.java-buildpack/tomcat/endorsed -classpath /home/vcap/app/.java-buildpack/tomcat/bin/bootstrap.jar:/home/vcap/app/.java-buildpack/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/vcap/app/.java-buildpack/tomcat -Dcatalina.home=/home/vcap/app/.java-buildpack/tomcat -Djava.io.tmpdir=/home/vcap/app/.java-buildpack/tomcat/temp org.apache.catalina.startup.Bootstrap start
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        62    32  9 13:08 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        63    62  0 13:08 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.01+0100 [App/0]   OUT vcap        64    63  0 13:08 ?        00:00:00 ps -ef
2014-10-01T14:08:37.01+0100 [App/0]   OUT ulimit (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT ===============
2014-10-01T14:08:37.01+0100 [App/0]   OUT core file size          (blocks, -c) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT data seg size           (kbytes, -d) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT scheduling priority             (-e) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT file size               (blocks, -f) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT pending signals                 (-i) 1024
2014-10-01T14:08:37.01+0100 [App/0]   OUT max locked memory       (kbytes, -l) 64
2014-10-01T14:08:37.01+0100 [App/0]   OUT max memory size         (kbytes, -m) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT open files                      (-n) 16384
2014-10-01T14:08:37.01+0100 [App/0]   OUT pipe size            (512 bytes, -p) 8
2014-10-01T14:08:37.01+0100 [App/0]   OUT POSIX message queues     (bytes, -q) 819200
2014-10-01T14:08:37.01+0100 [App/0]   OUT real-time priority              (-r) 0
2014-10-01T14:08:37.01+0100 [App/0]   OUT stack size              (kbytes, -s) 8192
2014-10-01T14:08:37.01+0100 [App/0]   OUT cpu time               (seconds, -t) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT max user processes              (-u) 512
2014-10-01T14:08:37.01+0100 [App/0]   OUT virtual memory          (kbytes, -v) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT file locks                      (-x) unlimited
2014-10-01T14:08:37.01+0100 [App/0]   OUT Free Disk Space (Before)
2014-10-01T14:08:37.01+0100 [App/0]   OUT ========================
2014-10-01T14:08:37.01+0100 [App/0]   OUT Filesystem            Size  Used Avail Use% Mounted on
2014-10-01T14:08:37.01+0100 [App/0]   OUT tmpfs                  17G     0   17G   0% /dev/shm
2014-10-01T14:08:37.04+0100 [App/0]   OUT Process Status (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT ======================
2014-10-01T14:08:37.04+0100 [App/0]   OUT UID        PID  PPID  C STIME TTY          TIME CMD
2014-10-01T14:08:37.04+0100 [App/0]   OUT root         1     0  0 13:08 ?        00:00:00 wshd: 1847628cl9b    
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        31     1  0 13:08 ?        00:00:00 /bin/bash
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        32    31 95 13:08 ?        00:00:27 [java] <defunct>
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        62    32  9 13:08 ?        00:00:00 sh -c /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        63    62  0 13:08 ?        00:00:00 bash /home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh
2014-10-01T14:08:37.04+0100 [App/0]   OUT vcap        68    63  0 13:08 ?        00:00:00 ps -ef
2014-10-01T14:08:37.04+0100 [App/0]   OUT ulimit (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT ==============
2014-10-01T14:08:37.04+0100 [App/0]   OUT core file size          (blocks, -c) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT data seg size           (kbytes, -d) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT scheduling priority             (-e) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT file size               (blocks, -f) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT pending signals                 (-i) 1024
2014-10-01T14:08:37.04+0100 [App/0]   OUT max locked memory       (kbytes, -l) 64
2014-10-01T14:08:37.04+0100 [App/0]   OUT max memory size         (kbytes, -m) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT open files                      (-n) 16384
2014-10-01T14:08:37.04+0100 [App/0]   OUT pipe size            (512 bytes, -p) 8
2014-10-01T14:08:37.04+0100 [App/0]   OUT POSIX message queues     (bytes, -q) 819200
2014-10-01T14:08:37.04+0100 [App/0]   OUT real-time priority              (-r) 0
2014-10-01T14:08:37.04+0100 [App/0]   OUT stack size              (kbytes, -s) 8192
2014-10-01T14:08:37.04+0100 [App/0]   OUT cpu time               (seconds, -t) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT max user processes              (-u) 512
2014-10-01T14:08:37.04+0100 [App/0]   OUT virtual memory          (kbytes, -v) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT file locks                      (-x) unlimited
2014-10-01T14:08:37.04+0100 [App/0]   OUT Free Disk Space (After)
2014-10-01T14:08:37.04+0100 [App/0]   OUT =======================
2014-10-01T14:08:37.04+0100 [App/0]   OUT Filesystem            Size  Used Avail Use% Mounted on
2014-10-01T14:08:37.04+0100 [App/0]   OUT tmpfs                  17G     0   17G   0% /dev/shm
@trastle

This comment has been minimized.

Show comment
Hide comment
@trastle

trastle Oct 1, 2014

Looks great!
Thanks @nebhale

trastle commented Oct 1, 2014

Looks great!
Thanks @nebhale

@nebhale nebhale closed this in c94424d Oct 1, 2014

@glyn

This comment has been minimized.

Show comment
Hide comment
@glyn

glyn Oct 1, 2014

Member

Note that warden applies various resource limits to applications, although most if not all of them are not set by default. See the setrlimit man page for definitions.

Then there are disk limits which are fed into setquota.

Member

glyn commented Oct 1, 2014

Note that warden applies various resource limits to applications, although most if not all of them are not set by default. See the setrlimit man page for definitions.

Then there are disk limits which are fed into setquota.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment