Failed to build store metadata causes OutOfMemory exception

[edalford11]

Elasticsearch version: 2.3.2

JVM version: 1.7.0_91

OS version: CentOS 6.7

We received the following error message in the logs

[2016-06-20 19:33:24,465][WARN ][index.store              ] [search1_node1] [v4_messages_12_2013][3]failed to build store metadata. checking segment info integrity (with commit [no])
java.nio.file.NoSuchFileException: /var/lib/elasticsearch/stocktwits/nodes/0/indices/v4_messages_12_2013/3/index/segments_27

which immediately caused the node to reach an OutOfMemory exception, even though it was only using 70% of the heap. After doing some digging it appears that index it is referring to could be corrupted. Is this a cause of running an older version of java?

[bleskes]

@edalford11 can you post the rest of the error message? I'm curious to see the stack trace.

After doing some digging it appears that index it is referring to could be corrupted

Can you share what you found?

Is this a cause of running an older version of java?

This should not be the case, but if you can upgrade (which is recommended and will be required as of 5.0) it will tell us more.

[edalford11]

Turns out that the original log message posted came after the OutOfMemory exception happened so that makes a little more sense.

I seem to be missing log files for when the OutOfMemory exception occurred. I see the logs on all of the other nodes when the event occurred but for some reason the node that actually went out of memory is missing logs, which makes this problem hard to solve.

All of the nodes were 70% and under on the heap usage which makes me confused as to why this happened but without logs I suppose it makes this hard to diagnose. I'll keep digging but this can probably be closed until I get more information.

[bleskes]

yeah, OOMs are nasty. I'm closing this for now. Please reopen (or open a new ticket) if you find something interesting.