Node ignoring --max-old-space-size

[mnhan3]

top output (kibana started with max-old-space-sze=200 along with gc tracing)

60742 elastics  20   0 3607m 1.8g 114m S  0.0 47.2  15:06.82 /usr/bin/java -Xms1500M -Xmx1500M -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSw 
17468 kibana    20   0 1622m 751m 9268 S  0.7 19.7   0:45.33 /opt/kibana/bin/../node/bin/node --expose-gc --trace-gc --trace-gc-verbose --max-old-space-siz 

Nothing is accessing Kibana. There's only 1 index in elasticsearch. Kibana node process continues to eat memory until it gets killed by OOM.

stdout of gc tracing shows:

[17468]  4999994 ms: Scavenge 180.9 (223.5) -> 171.9 (226.5) MB, 14.7 ms (+ 55.2 ms in 188 steps since last GC) [allocation failure].
[17468] Memory allocator,   used: 231908 KB, available:  38428 KB
[17468] New space,          used:   3477 KB, available:  12906 KB, committed:  32768 KB
[17468] Old pointers,       used: 123574 KB, available:     35 KB, committed: 123841 KB
[17468] Old data space,     used:  37628 KB, available:    815 KB, committed:  38486 KB
[17468] Code space,         used:   9254 KB, available:  12442 KB, committed:  21912 KB
[17468] Map space,          used:   1971 KB, available:   6015 KB, committed:   8190 KB
[17468] Cell space,         used:     83 KB, available:     20 KB, committed:    128 KB
[17468] PropertyCell space, used:     56 KB, available:      7 KB, committed:     64 KB
[17468] Large object space, used:      0 KB, available:  37387 KB, committed:      0 KB
[17468] All spaces,         used: 176045 KB, available:  32243 KB, committed: 225390 KB
[17468] External memory reported:   -385 KB
[17468] Total time spent in GC  : 757.7 ms
[17468]  5142419 ms: Scavenge 184.6 (227.5) -> 175.6 (229.5) MB, 14.7 ms (+ 53.7 ms in 188 steps since last GC) [allocation failure].
[17468] Memory allocator,   used: 234980 KB, available:  35356 KB
[17468] New space,          used:   3489 KB, available:  12894 KB, committed:  32768 KB
[17468] Old pointers,       used: 126653 KB, available:      0 KB, committed: 126864 KB
[17468] Old data space,     used:  38213 KB, available:    229 KB, committed:  38486 KB
[17468] Code space,         used:   9289 KB, available:  12404 KB, committed:  21912 KB
[17468] Map space,          used:   2039 KB, available:   5947 KB, committed:   8190 KB
[17468] Cell space,         used:     83 KB, available:     20 KB, committed:    128 KB
[17468] PropertyCell space, used:     56 KB, available:      7 KB, committed:     64 KB
[17468] Large object space, used:      0 KB, available:  34315 KB, committed:      0 KB
[17468] All spaces,         used: 179825 KB, available:  31503 KB, committed: 228413 KB
[17468] External memory reported:   -398 KB
[17468] Total time spent in GC  : 772.4 ms
[17468]  5284865 ms: Scavenge 188.3 (230.5) -> 179.3 (234.5) MB, 14.1 ms (+ 55.1 ms in 188 steps since last GC) [allocation failure].
[17468] Memory allocator,   used: 240100 KB, available:  30236 KB
[17468] New space,          used:   3487 KB, available:  12896 KB, committed:  32768 KB
[17468] Old pointers,       used: 129744 KB, available:    943 KB, committed: 130895 KB
[17468] Old data space,     used:  38799 KB, available:    613 KB, committed:  39494 KB
[17468] Code space,         used:   9329 KB, available:  12362 KB, committed:  21912 KB
[17468] Map space,          used:   2109 KB, available:   5872 KB, committed:   8190 KB
[17468] Cell space,         used:     83 KB, available:     20 KB, committed:    128 KB
[17468] PropertyCell space, used:     56 KB, available:      7 KB, committed:     64 KB
[17468] Large object space, used:      0 KB, available:  29195 KB, committed:      0 KB
[17468] All spaces,         used: 183610 KB, available:  32715 KB, committed: 233452 KB
[17468] External memory reported:   -412 KB
[17468] Total time spent in GC  : 786.5 ms
[17468] Speed up marking after 1024 steps

This is a RHEL 6 instance on vmware. It has 4gigs of ram allocated. Its running ES 2.2 and kibana 4.4 behind a web ssl proxy. How can I get Kibana to stop running the box out of ram. Already using lastest kibana release and using the old data flag.

Mike

[rashidkpc]

Just re-read your message, can you post the entirety of the top output? It cuts off at --max-old-space-siz

[mnhan3]

Here's the full command via ps aux.

ps aux|grep node
kibana   17468  0.8 21.7 1766484 852536 pts/0  Sl   13:20   0:52 /opt/kibana/bin/../node/bin/node --expose-gc --trace-gc --trace-gc-verbose --max-old-space-size=200 /opt/kibana/bin/../src/cli

[rashidkpc]

Changed the title, haven't seen this happen anywhere, will try to replicate

[rashidkpc]

Is this happening when Kibana is just sitting? Are you installing any plugins? Optimize running?

[mnhan3]

Kibana is just sitting. No Dashboard created. No plugins installed. I haven't optimize anything in kibana. I just added the ES index pattern to kibana at the settings when it first came up. After restarting kibana, not even accessing it on browser and memory is growing. The only optimization i remember doing was with curator, i told it to optimize indexes on ES. Would that cause kibana to memory leak?

[rashidkpc]

Kibana has a built in code optimizing that is separate from the elasticsearch _optimize operation. It can possible cause memory to increase temporarily. We're looking into this.

[mnhan3]

do you need any logs from my installation? Anything you want me to test/try? Currently kibana hasn't killed my vm with setting old-space-size=200M. It using 1gig of memory but haven't killed it like when I had it running default with nothing set. It looks like it stabilized at 1gig, but that's still huge amount of memory to be holding idle.

60742 elastics  20   0 3607m 1.8g 114m S  0.3 47.3  20:22.77 /usr/bin/java -Xms1500M -Xmx1500M -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSw
17468 kibana    20   0 1900m 1.0g 6916 S  0.7 27.7  14:18.74 /opt/kibana/bin/../node/bin/node --expose-gc --trace-gc --trace-gc-verbose --max-old-space-siz

from my messages log, oom killed kibana with default setting at around 1.8+g ram used when i originally ran it with no node_options.

Feb  6 04:35:27 elks01 kernel: [60742]   498 60742   921715   454038   0       0             0 java 
Feb  6 04:35:27 elks01 kernel: [60857] 19538 60857  1898775   424612   0       0             0 node 
--snip-
Feb  6 04:35:27 elks01 kernel: Out of memory: Kill process 60857 (node) score 710 or sacrifice child
Feb  6 04:35:27 elks01 kernel: Killed process 60857, UID 19538, (node) total-vm:7595100kB, anon-rss:1696804kB, file-rss:1644kB

Update:
nevermind i spoke too soon, started to slowly grow again. Sort of oscillating between 1.0 and 1.1 gigs of ram. Access kibana a few times, doesn't seem to make it use any additional memory.

[jdmchone]

Any update on this issue? We're experiencing the same thing

[rashidkpc]

We've been trying to reproduce this, we've tried with a variety of settings and aren't seeing any memory increase over time even when setting --max-old-space-sze=200. Here's a chart after applying that on our internal dashboarding system

$ ps aux|grep node
rashidk+  2042  2.1  7.5 1222400 304876 ?      Ssl  Feb11  31:21 /opt/kibana/bin/../node/bin/node --max-old-space-size=200 /opt/kibana/bin/../src/cli

[mnhan3]

So what now. Since I'm not the only one seeing this, there must be a common setup that triggers this issue. My kibana test server hasn't run out of memory yet, but using 1g of ram when idle with a setting of 200M is kind of bad.

On a new 8gig vm, i untared the kiban tar.gz into /opt/kibana, move it out of the kibana-4.4-linux-x86_64 directory. stuck my kibana.yml into the config dir with the apropriate certs for ssl to es. add the --max-old-space-size=200 parameter into kibana in the bin and fired it up as root. cd bin;./kibana& In 11minutes, the result:
top:

36024 root      20   0 1129m 223m 9248 S  1.3  2.8   0:11.27 ./../node/bin/node --max-old-space-size=200 ./../src/cli

A ps:

 ps auxf|grep node
root     36024  1.2  2.8 1160632 232736 pts/0  Sl   11:23   0:11                      \_ ./../node/bin/node --max-old-space-size=200 ./../src/cli
root     36370  0.0  0.0 103304   876 pts/0    S+   11:38   0:00                      \_ grep node

Not even accessing this kibana instance. Its only listening to localhost. It continues to grow at 1M/3s.

so it takes only ~11 minutes to go beyond the 200M limit. Could something in the kibana.index on the ES server be "corrupt"/messed up to cause this. Or is it something to do with ssl?