Cortex crashed after a OutOfMemoryError

[srilumpa]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
Cortex version / git hash	1.1.4-1
Package Type	DEB

Problem Description

Our Cortex instance crashed last night with a Java OutOfMemoryError. We installed Cortex with the DEB package and we haven't touched the basic configuration on how the jvm is started (so I guess the default MaxHeapSize is 500MB).

Our instance was running since January the 12th. I don't know how many jobs were run but I assume a lot: I counted about 1.200 analysis that crashed in TheHive tonight because of Cortex not being available, so I guess there are about 3000+ jobs a day.

Not sure how to help on this, so feel free to ask for any additional information.

Steps to Reproduce

1. Let it run for a long time with a lot of jobs (?)

Possible Solutions

• allow configuration of the JVM HeapSize
• Keep in memory only a limited (configurable) set of jobs (?)

Complementary information

Here are the logs generated by the crash:

Jan 24 19:07:35 my-cortex-server cortex[30357]: info s.ExternalAnalyzerSrv - Execute sh -c "./robtex.py"  in Robtex
Jan 24 19:16:38 my-cortex-server cortex[30357]: info s.ExternalAnalyzerSrv - Execute sh -c "./passivetotal_analyzer.py"  in PassiveTotal
Jan 24 19:16:55 my-cortex-server cortex[30357]: Exception in thread "Thread-128136" java.lang.OutOfMemoryError: Java heap space
Jan 24 19:16:58 my-cortex-server cortex[30357]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-128136"
Jan 24 19:17:17 my-cortex-server cortex[30357]: error p.c.s.n.PlayRequestHandler - Exception caught in Netty
Jan 24 19:17:17 my-cortex-server cortex[30357]: java.lang.OutOfMemoryError: Java heap space
Jan 24 19:17:39 my-cortex-server cortex[30357]: Exception in thread "AsyncAppender-Worker-ASYNCFILE" java.lang.OutOfMemoryError: Java heap space
Jan 24 19:18:34 my-cortex-server cortex[30357]: Exception in thread "process reaper" java.lang.OutOfMemoryError: Java heap space
Jan 24 19:18:35 my-cortex-server cortex[30357]:   at java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:519)
Jan 24 19:18:35 my-cortex-server cortex[30357]:   at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:532)
Jan 24 19:18:35 my-cortex-server cortex[30357]:   at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:301)
Jan 24 19:18:35 my-cortex-server cortex[30357]:   at java.lang.UNIXProcess$$Lambda$15/1220975671.run(Unknown Source)
Jan 24 19:18:36 my-cortex-server cortex[30357]:   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jan 24 19:18:36 my-cortex-server cortex[30357]:   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jan 24 19:18:36 my-cortex-server cortex[30357]:   at java.lang.Thread.run(Thread.java:748)
Jan 24 19:20:59 my-cortex-server cortex[30357]: Exception in thread "Thread-128137" java.lang.OutOfMemoryError: Java heap space
Jan 24 19:21:18 my-cortex-server cortex[30357]: Exception in thread "Thread-128138" java.lang.OutOfMemoryError: Java heap space
Jan 24 19:28:27 my-cortex-server cortex[30357]: warn i.n.u.HashedWheelTimer - An exception was thrown by TimerTask.
Jan 24 19:28:27 my-cortex-server cortex[30357]: java.lang.OutOfMemoryError: Java heap space
Jan 24 19:28:31 my-cortex-server systemd[1]: cortex.service: Main process exited, code=exited, status=255/n/a
Jan 24 19:28:31 my-cortex-server systemd[1]: cortex.service: Unit entered failed state.
Jan 24 19:28:31 my-cortex-server systemd[1]: cortex.service: Failed with result 'exit-code'.
Jan 25 10:10:31 my-cortex-server systemd[1]: Started cortex.
Jan 25 10:10:32 my-cortex-server cortex[25686]: info a.e.s.Slf4jLogger - Slf4jLogger started
Jan 25 10:10:33 my-cortex-server cortex[25686]: info s.MispSrv - MISP modules is disabled, loader is /opt/cortex/contrib/misp-modules-loader.py
Jan 25 10:10:33 my-cortex-server cortex[25686]: info play.api.Play - Application started (Prod)
Jan 25 10:10:34 my-cortex-server cortex[25686]: info p.c.s.NettyServer - Listening for HTTPS on /0.0.0.0:443

[nadouani]

Hello @srilumpa, this will no longer be an issue on Cortex 2 since it will not keep the data in memory but will persist it.

So closing this issue, as we are working on Cortex 2 that will be release in February.