Investigate Heap Requirements for Smarti

[westei]

When running with -Xmx4g a java.lang.OutOfMemoryError: Java heap space was encountered.

As this is unexpected we need to further investigate memory consumption of Smarti. This include

• implementation of a stress test utility
• testing with high loads of conversation with different lengths and random messages
• testing memory footprint of different analysis configurations (especially NLP processing components and models)
• investigate for possible memory leaks

NOTE: marking this as enhancement with the intention to create additional issues based on investigation results

[westei]

The investigation concluded that their are no memory leaks present. Even long processing runs of >1000 conversations with >5000 messages showed no increase in base memory.

The OOM errors could be traced down to messages containing long sentences (or other Strings e.g. ASKII graphics) that cause the Stanford NLP Parser to require huge amounts of memory.

Several solutions where tested with the following results:

1. first and foremost ist is necessary to limit the maximum length of tokens a sentence can have for the parser to process it. This can now be done by using the nlp.stanfordnlp.de.parseMaxlen property. The default 30 is good for 4g java heap and the current analysis configuration.
2. to NOT use the Factored Parser. While this is the default of Stanford NLP it is by far the slowest and need the most Memory. For Smarti the default was set to the PCFG Parser. The Shift Reduce Parser is an alternative that is even faster and uses less memory (configure nlp.stanfordnlp.de.parseModel=edu/stanford/nlp/models/lexparser/germanPCFG.ser.gz to use this parser)
3. the number of processing threads is now configureable (as requested by Smarti should run with only 2 processing threads #145) and the default is now set to 2 (was 8). For every thread one should reserve about 500m additional heap (so on an 8 core machine it is recommended to run Smarti with 8g heap if 8 processing threads are configured)

With this configuration no OOM errors where encountered. Only when processing ASKII Graphics the PCFG Parser was driving the System to its limits while causes a lot of GC overhead. After processing the System recovered without problems and continued normal.

NOTE: Those changes require the newest SNAPSHOT version of redlink-nlp. As this also update to Stanford NLP 3.8.0 (was 3.6.0) Snarti users will need to update the Stanford NLP jars in the ext folder accordingly (see also according changes to dist/src/main/resources/plugin-info.txt)

[ruKurz]

at least very cool that we have knowledge in that depth.

[ruKurz]

@westei Can you please write a short doc on how you run the stress test for the ressource consumption behavior of Smarti?

[westei]

created #179 for the documentation