OutOfMemoryError in LogStash

[mmguero]

lately I've seen some of these depending on the worklaod:

logstash_1           | warning: thread "[malcolm-output]>worker4" terminated with exception (report_on_exception is true):warning: thread "[malcolm-output]>worker0" terminated with exception (report_on_exception is true):
logstash_1           | java.lang.OutOfMemoryError: Java heap space
...

I think that as the size of the events has grown larger (due to adding more normalized fields such as ECS) we need to make adjustments to some logstash runtime parameters:

pipeline.workers, pipeline.batch.size and pipeline.batch.delay - these settings are used to tune the performance and resource utilization of the the logstash container; see Tuning and Profiling Logstash Performance, logstash.yml and Multiple Pipelines

I've added code into install.py to reduce the number of default pipeline workers. I think probably the big one is we need to reduce pipeline.batch.size so that we don't need quite as much memory available to hold a batch in memory. I'm going to try reducing it from 125 (default) to 75. These will be configurable in the docker-compose.yml files.