Memory leak

[ThePixelDeveloper]

I had a long discussion about this on the mailing list. The solutions tried have had no effect so I'm considering this a bug.

System Information

• Total memory = 469M
• vm.swappiness = 10

Java

Vendor:       Sun Microsystems Inc.
Name:         Java HotSpot(TM) Client VM
Version:      19.1-b02
Java version: 1.6.0_24

Elastic

Version: 0.16.2

Indicies

Number of documents: 3888, Store size: 1.8mb (1925915 B)
Field cache evictions: 0, Field cache size: 125.9kb, Filter cache size: 18.3kb
Merges: Current: 0, Total: 0, Took: 0s

Graphs

Solutions tried

Locking memory

No effect. The following option was set for the elastisearch user:

ulimit -l unlimited

Swapped from OpenJDK to Oracle's Java

No effect

Next action

I'll be trying to monitor the JVM at a lower level using a tool like visualgc to see what I can gain from it.

[AdrianRossouw]

I have been experiencing the same, and it was confirmed by kimchy that this is a a problem he is debugging.

Description

The JVM process steadily consuming more memory until the system becomes starved of resources and falls over. This additional memory consumption is not tied to the reported heap/non-heap memory consumption from BigDesk or the nodes/stats API.

While initially considered ubuntu specific, there have been reports of it happening on other linux distributions, but the low memory configuration of the 'problem systems' makes the issue more pronounced, and these systems have been running ubuntu so far.

Solutions tried

switch to oracle jre

no effect.

stop using the servicewrapper

no effect.

determine if open files/sockets cause the problem

Debugging with lsof and netstat return reasonable results, showing that the pipes are being closed correctly according to the operating system.

remove native libraries to check for leaks in them

Upon kimchy's instructions i removed the sigar, jna and jline libraries from /lib, and restarted the server. These are the results after 4 days

[AdrianRossouw]

I have cloned one of the environments and made different changes to each clone.

1. upgraded the clone 'spin1' to ubuntu 10.10 from 10.04.
2. upgraded the clone 'spin2' to the latest jre (update 26).

Initial states: spin1 and spin2
12 hours later: spin1 and spin2

This is too early to tell what is going wrong, but on telling difference has shown up.
Namely, the tcp connection count on spin1 is incredibly high, much higher than
spin2 with the newer JRE. These figures are provided by sigar, and a connection
reference leak is a possible cause for a memory leak of this nature.

  # from spin2 

    "tcp" : {
      "active_opens" : 72,
      "passive_opens" : 2980,
      "curr_estab" : 29,
      "in_segs" : 130486,
      "out_segs" : 98255,
      "retrans_segs" : 69,
      "estab_resets" : 6,
      "attempt_fails" : 7,
      "in_errs" : 0,
      "out_rsts" : 6
    }

    # from spin1
    "tcp" : {
      "active_opens" : 10256,
      "passive_opens" : 14670,
      "curr_estab" : 29,
      "in_segs" : 205680,
      "out_segs" : 204016,
      "retrans_segs" : 108,
      "estab_resets" : 59,
      "attempt_fails" : 7,
      "in_errs" : 0,
      "out_rsts" : 44
    }

[ThePixelDeveloper]

Since my last post I've updated to the latest JRE (update 26) and still experience the same issues.

Here are node stats after it's been running for a few days:

https://gist.github.com/754639add60d4196e8f5

Stats after an elasticsearch restart:

https://gist.github.com/17587c560f58a363e5ad

It should also be noted that my network stats are mostly the same with regard to active opens.

[kimchy]

Heya, I think I found the issue, which is a leak in the JVM itself when using GarbageCollectorMXBean#getLastGcInfo. It get called periodically in elasticsearch (every 1 second), and also on node stats API.

I have pushed a change to disable it into both 0.16 branch and master.

[AdrianRossouw]

Adding -Des.monitor.jvm.enabled=false to the command line solved the issue.

After 1 day, the test environment is using 160mb of memory, while the control is using 200mb.

[aloiscochard]

kimchy.karma += 100k

[ThePixelDeveloper]

I've been running off 0.16 head for the past day and can confirm the memory leak has been fixed

[kimchy]

cool, I will release 0.16.4 today that disables that by default, and also does not have the leak when calling node stats.