FfDL logging issues with Elastic Search

[Tomcli]

The FfDL Elastic Search sometimes has an overhead issue when creating the emetrics/logline mapping.

[2018-02-15T18:11:21,334][INFO ][o.e.c.m.MetaDataMappingService] [VPF0eed] [dlaas_learner_data/Of58W91xS-6OlsuQEGByzw] update_mapping [logline]
[2018-02-15T18:11:36,852][INFO ][o.e.m.j.JvmGcMonitorService] [VPF0eed] [gc][556] overhead, spent [258ms] collecting in the last [1s]

When the Elastic Search works properly, it should have the following logs for mapping update/create.

[2018-02-15T17:52:52,598][INFO ][o.e.c.m.MetaDataMappingService] [R7H6R6o] [dlaas_learner_data/d-NMzvwRT_CXgMwHOBinTg] update_mapping [logline]
[2018-02-15T17:54:51,289][INFO ][o.e.c.m.MetaDataMappingService] [R7H6R6o] [dlaas_learner_data/d-NMzvwRT_CXgMwHOBinTg] create_mapping [emetrics]

[sboagibm]

Probably first thing to try is to see if we can monkey with templates/infrastructure/storage.yml to give it a bit more memory.

[animeshsingh]

@sboagibm dont see any memory field here
https://github.com/IBM/FfDL/blob/master/templates/infrastructure/storage.yml

[sboagibm]

@whummer do you know off the top of your head how to allocate more memory there?

[k0105]

Shouldn't allocating more memory work the same way as with containers, i.e. something like the following?

      requests:
        memory: "500Mi"
      limits:
        memory: "1000Mi"

What are your specs? How much RAM do you have and how much space does /var/lib/elasticsearch have left?

You could also try increasing the Java heap size with -Xmx (maximum heap size) and -Xms (initial heap size) in jvm.options. jmap -histo <pid> should provide a heap histogram, cmp. https://docs.oracle.com/javase/10/tools/jmap.htm

Finally: What are bootstrap.memory_lock, indices.memory.index_buffer_size and MAX_OPEN_FILES?

[If all of those check out, this might just be normal garbage collection, if I'm not mistaken. 25% does not seem too critical if it does not occur too frequently. The threshold for hard warnings is 50%, iirc.]

[animeshsingh]

Thanks @k0105 - We will take a look @Tomcli @whummer

[Tomcli]

Thanks @k0105, this issue was caused by some helper pods that has insufficient memory. I did a quick PR that will fix this issue.