Unexpected crashes due to memory exhaustion

[jblt-sidn]

Since moving to version 2.25, we've been having unexpected crashes of the java validator process. Increased VM memory but that didn't help.

Log:

2018-09-06 13:37:57,681 WARN [qtp686428408-24] QueuedThreadPool Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@2c6cea33 in qtp686428408{STARTED,8<=9<=200,i=0,q=0}
2018-09-06 13:37:59,264 WARN [qtp686428408-24] LoggingOutputStream Exception in thread "qtp686428408-24"
2018-09-06 13:38:21,915 WARN [qtp686428408-24] LoggingOutputStream java.lang.OutOfMemoryError: Java heap space
2018-09-06 13:39:42,048 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream Uncaught error from thread [
2018-09-06 13:39:42,048 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream default-akka.actor.default-dispatcher-15
2018-09-06 13:39:42,049 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream ] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[
2018-09-06 13:39:42,049 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream default
2018-09-06 13:39:42,049 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream ]
2018-09-06 13:39:42,049 WARN [default-akka.actor.default-dispatcher-15] LoggingOutputStream java.lang.OutOfMemoryError: Java heap space
2018-09-06 13:39:42,107 INFO [shutdownHook1] RTRServerHandler Client disconnected : /10.20.28.43:49536
2018-09-06 13:39:42,113 INFO [shutdownHook2] ServerConnector Stopped ServerConnector@1c99d1df{HTTP/1.1}{0.0.0.0:80}
2018-09-06 13:39:42,116 INFO [shutdownHook1] RTRServerHandler Client disconnected : /10.20.28.23:60841
2018-09-06 13:39:42,119 INFO [shutdownHook1] RTRServer RTR server stopped
2018-09-06 13:39:42,121 INFO [shutdownHook2] ContextHandler Stopped o.e.j.s.ServletContextHandler@203ddacf{/,jar:file:/sidn/rpki-validator-app-955/lib/rpki-validator-app-2.25-SNAPSHOT.jar!/public,UNAVAILABLE}
2018-09-06 13:39:42,142 INFO [shutdownHook2] Main Terminating...

[lolepezy]

Hi,

Could you please mention which version have you moved from?

[jblt-sidn]

Hi,

We moved from 2.20 to 2.25.

[jblt-sidn]

Java environment could be relevant?

java version "1.7.0_181"
OpenJDK Runtime Environment (IcedTea 2.6.14) (7u181-2.6.14-0ubuntu0.2)
OpenJDK 64-Bit Server VM (build 24.181-b01, mixed mode)

[lolepezy]

Hi,

1. It is hard to say which changes between 2.20 and 2.25 made the application consume more memory, but it there any chance you can give it more? Normally 3gb should be more than enough, at least our test instance runs for months with this setting.

2. Generally we recommend people to switch to RPKI validator 3, which consumes less memory and can also be faster in some cases. You can try it
   https://github.com/RIPE-NCC/rpki-validator-3/wiki/RIPE-NCC-RPKI-Validator-3-Production

[jblt-sidn]

Hi,

We feel that there might be a simple solution if we did a little bit more research together.

The application doest not seem to consume more memory, yet it still crashes after running for a few hours. This may be due to a memory leak, or something entirely different. We were wondering if the log message "LoggingOutputStream java.lang.OutOfMemoryError: Java heap space" could provide any clues? Maybe some log buffer is getting full?

The java application is not running containerized or in a chroot or SE linux enabled environment. Could there be another reason the application cannot empty some log buffer? There is plenty disk space by the way.

[footplus]

Hello,

I can confirm this (moved from 2.24 to 2.25).

I pushed the JVM size up to 5.5GB, it seems to maximize the time between crashes, but it does not prevent them.

On Ubuntu 16.04.

java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

Rolled back to 2.24; will try version 3.