New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES_HEAP_SIZE or LS_HEAP_SIZE has no effect #129

Closed
heikkis opened this Issue Apr 9, 2017 · 16 comments

Comments

Projects
None yet
3 participants
@heikkis

heikkis commented Apr 9, 2017

I'm trying to do small ELK instance for CI-pipeline but I was not able to lower the memory requirements via parameters in documentation.

docker run --rm -e LS_HEAP_SIZE=100m -e ES_HEAP_SIZE=100m -it --name elk -p 6601:5601 -p 3200:9200 -p 9044:5044 sebp/elk:530

[root@x ~]# docker status elk
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
elk                 301.52%             2.737 GiB / 125.7 GiB   2.18%               70 B / 648 B        0 B / 71.17 kB      0
^C
[root@x ~]# docker top elk
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                19317               19297               0                   09:32               pts/2               00:00:00            /bin/bash /usr/local/bin/start.sh
root                19372               19317               0                   09:32               ?                   00:00:00            /usr/sbin/cron
991                 19453               19317               99                  09:32               pts/2               00:00:23            /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
992                 19746               19317               99                  09:32               pts/2               00:00:47            /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/logstash -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.config /etc/logstash/conf.d --path.logs /var/log/logstash
993                 19782               19317               42                  09:32               ?                   00:00:04            /opt/kibana/bin/../node/bin/node --max-old-space-size=250 --no-warnings /opt/kibana/bin/../src/cli -l /var/log/kibana/kibana5.log
root                19785               19317               0                   09:32               pts/2               00:00:00            tail -f /var/log/elasticsearch/elasticsearch.log /var/log/logstash/logstash-plain.log /var/log/kibana/kibana5.log
[root@x ~]#
@kenwdelong

This comment has been minimized.

Show comment
Hide comment
@kenwdelong

kenwdelong Apr 9, 2017

Contributor

ES_HEAP_SIZE is processed in lines 60-66 of start.sh. It modifies the file /etc/elasticsearch/jvm.options. First step would be to look in that file in your running docker container to make sure the file was updated properly.

Also, I think ES 5 cannot run in less than 2GB of RAM; it could be that it's ignoring your 100m and going with its minimum.

Contributor

kenwdelong commented Apr 9, 2017

ES_HEAP_SIZE is processed in lines 60-66 of start.sh. It modifies the file /etc/elasticsearch/jvm.options. First step would be to look in that file in your running docker container to make sure the file was updated properly.

Also, I think ES 5 cannot run in less than 2GB of RAM; it could be that it's ignoring your 100m and going with its minimum.

@heikkis

This comment has been minimized.

Show comment
Hide comment
@heikkis

heikkis Apr 9, 2017

Lines are added to jvm.options but still no affect. Same with 3g setting.

docker run -d -e LS_HEAP_SIZE=100m -e ES_HEAP_SIZE=3g -it --name elk -p 6601:5601 -p 3200:9200 -p 9044:5044 sebp/elk:530

/usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

heikkis commented Apr 9, 2017

Lines are added to jvm.options but still no affect. Same with 3g setting.

docker run -d -e LS_HEAP_SIZE=100m -e ES_HEAP_SIZE=3g -it --name elk -p 6601:5601 -p 3200:9200 -p 9044:5044 sebp/elk:530

/usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

@heikkis

This comment has been minimized.

Show comment
Hide comment
@heikkis

heikkis Apr 9, 2017

It seems that correct file to modify would be /opt/elasticsearch/config/jvm.options for ES.

Not /etc/elasticsearch/jvm.options

heikkis commented Apr 9, 2017

It seems that correct file to modify would be /opt/elasticsearch/config/jvm.options for ES.

Not /etc/elasticsearch/jvm.options

@kenwdelong

This comment has been minimized.

Show comment
Hide comment
@kenwdelong

kenwdelong Apr 11, 2017

Contributor

Arg, you are correct about ES_HEAP_SIZE. Noted here: https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#jvm-options

This image uses the tar, so it should be the /opt/elasticsearch/config/jvm.options file.

This is my bad, as I submitted that pull request. I'll update.

Contributor

kenwdelong commented Apr 11, 2017

Arg, you are correct about ES_HEAP_SIZE. Noted here: https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#jvm-options

This image uses the tar, so it should be the /opt/elasticsearch/config/jvm.options file.

This is my bad, as I submitted that pull request. I'll update.

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 11, 2017

Owner

Thanks for the analysis and for the PR @kenwdelong. I'm also responsible for this bug as I moved from deb to tar at one point and didn't notice that the path for jvm.options had to be changed, which means that the images with tags from 502 to 530 have the issue with ES_HEAP_SIZE.

Owner

spujadas commented Apr 11, 2017

Thanks for the analysis and for the PR @kenwdelong. I'm also responsible for this bug as I moved from deb to tar at one point and didn't notice that the path for jvm.options had to be changed, which means that the images with tags from 502 to 530 have the issue with ES_HEAP_SIZE.

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 11, 2017

Owner

Still need to look into LS_HEAP_SIZE (probably won't be until this weekend for me).

Owner

spujadas commented Apr 11, 2017

Still need to look into LS_HEAP_SIZE (probably won't be until this weekend for me).

@heikkis

This comment has been minimized.

Show comment
Hide comment
@heikkis

heikkis Apr 11, 2017

Same for logstash. /opt/logstash/config/jvm.options file should be used.

heikkis commented Apr 11, 2017

Same for logstash. /opt/logstash/config/jvm.options file should be used.

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 11, 2017

Owner

OK thanks, will take care of it this weekend (unless a PR comes in before then 😄).

Owner

spujadas commented Apr 11, 2017

OK thanks, will take care of it this weekend (unless a PR comes in before then 😄).

@heikkis

This comment has been minimized.

Show comment
Hide comment
@heikkis

heikkis Apr 11, 2017

Thanks 👍

heikkis commented Apr 11, 2017

Thanks 👍

@kenwdelong

This comment has been minimized.

Show comment
Hide comment
@kenwdelong

kenwdelong Apr 11, 2017

Contributor

Phew. I thought I was "that guy" who forgot to test!

Contributor

kenwdelong commented Apr 11, 2017

Phew. I thought I was "that guy" who forgot to test!

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 12, 2017

Owner

Call out to the Java experts out there (@kenwdelong 😉): despite what the documentation says (https://www.elastic.co/guide/en/logstash/5.3/tuning-logstash.html) I think that Logstash actually no longer supports the LS_HEAP_SIZE environment variable (if only for the fact that it no longer works as it did before!).

The /opt/logstash/config/jvm.options file reads as follows (no mention of LS_HEAP_SIZE anywhere):

## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx1g

################################################################
## Expert settings

Now I would be tempted to do the same as for Elasticsearch and set both Xms and Xmx to whatever the LS_HEAP_SIZE env var contains, but I just want to check with you if this is how heap sizes work (or if e.g. I just need to set Xmx… which I suppose might be a problem if it's less than Xms).

Owner

spujadas commented Apr 12, 2017

Call out to the Java experts out there (@kenwdelong 😉): despite what the documentation says (https://www.elastic.co/guide/en/logstash/5.3/tuning-logstash.html) I think that Logstash actually no longer supports the LS_HEAP_SIZE environment variable (if only for the fact that it no longer works as it did before!).

The /opt/logstash/config/jvm.options file reads as follows (no mention of LS_HEAP_SIZE anywhere):

## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx1g

################################################################
## Expert settings

Now I would be tempted to do the same as for Elasticsearch and set both Xms and Xmx to whatever the LS_HEAP_SIZE env var contains, but I just want to check with you if this is how heap sizes work (or if e.g. I just need to set Xmx… which I suppose might be a problem if it's less than Xms).

@kenwdelong

This comment has been minimized.

Show comment
Hide comment
@kenwdelong

kenwdelong Apr 12, 2017

Contributor

Xms will set the minimum (starting) heap size. If more memory is required, java will increase the heap until it gets to Xmx, the maximum heap size. This act of increasing the heap size is kinda intensive (at least it used to be, things might be better now) and degrades performance, so a common practice is to set Xmx and Xms to the same value. That way the heap starts out at your desired size and you don't need to wait for the resizing events. And if Xms is larger than Xmx, the JVM will not start (at least most of them).

Given that Logstash's jvm.options has Xmx and Xms values in it, I think it's a safe bet that LS_HEAP_SIZE is deprecated.

My suggestion would be to do the same thing we did for ES_HEAP_SIZE, and awk the jvm.options file. Then in order to test, run it with, say, LS_HEAP_SIZE=700m and use top or ps to see that the command that ran actually had the correct options set!

Contributor

kenwdelong commented Apr 12, 2017

Xms will set the minimum (starting) heap size. If more memory is required, java will increase the heap until it gets to Xmx, the maximum heap size. This act of increasing the heap size is kinda intensive (at least it used to be, things might be better now) and degrades performance, so a common practice is to set Xmx and Xms to the same value. That way the heap starts out at your desired size and you don't need to wait for the resizing events. And if Xms is larger than Xmx, the JVM will not start (at least most of them).

Given that Logstash's jvm.options has Xmx and Xms values in it, I think it's a safe bet that LS_HEAP_SIZE is deprecated.

My suggestion would be to do the same thing we did for ES_HEAP_SIZE, and awk the jvm.options file. Then in order to test, run it with, say, LS_HEAP_SIZE=700m and use top or ps to see that the command that ran actually had the correct options set!

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 13, 2017

Owner

Thanks very much for the comprehensive explanation! Will do as you suggested.

Owner

spujadas commented Apr 13, 2017

Thanks very much for the comprehensive explanation! Will do as you suggested.

@spujadas spujadas closed this in 95461c1 Apr 15, 2017

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 15, 2017

Owner

Fixed (recreated a 530 image).

docker-compose.yml.129-heap_size

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ES_HEAP_SIZE=2g
    - LS_HEAP_SIZE=1g

Then:

$ docker-compose -f docker-compose.yml.129-heap_size up
$ docker exec -it elkdocker_elk_1 bash
# ps aux | less -+S
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
elastic+    63 27.5 53.6 4736372 1099640 ?     Sl   08:58   0:14 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -X
X:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -serv
er -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Di
o.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownH
ookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/e
lasticsearch -cp /opt/elasticsearch/lib/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.El
asticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.da
ta=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
...
logstash   197 84.4 21.9 3588696 450368 ?      SNl  08:58   0:29 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=tr
ue -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/logstash -Xmx1g -Xms1g -Xss2048k -Djffi.
boot.library.path=/opt/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classp
ath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.s
hell=/bin/sh org.jruby.Main /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.config /etc/logstash/c
onf.d --path.logs /var/log/logstash
...

Many thanks to both of you for your help in resolving this issue! 👍

Owner

spujadas commented Apr 15, 2017

Fixed (recreated a 530 image).

docker-compose.yml.129-heap_size

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ES_HEAP_SIZE=2g
    - LS_HEAP_SIZE=1g

Then:

$ docker-compose -f docker-compose.yml.129-heap_size up
$ docker exec -it elkdocker_elk_1 bash
# ps aux | less -+S
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
elastic+    63 27.5 53.6 4736372 1099640 ?     Sl   08:58   0:14 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -X
X:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -serv
er -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Di
o.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownH
ookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/e
lasticsearch -cp /opt/elasticsearch/lib/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.El
asticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.da
ta=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
...
logstash   197 84.4 21.9 3588696 450368 ?      SNl  08:58   0:29 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=tr
ue -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/logstash -Xmx1g -Xms1g -Xss2048k -Djffi.
boot.library.path=/opt/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classp
ath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.s
hell=/bin/sh org.jruby.Main /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.config /etc/logstash/c
onf.d --path.logs /var/log/logstash
...

Many thanks to both of you for your help in resolving this issue! 👍

@kenwdelong

This comment has been minimized.

Show comment
Hide comment
@kenwdelong

kenwdelong Apr 16, 2017

Contributor

Just be careful - I think those are the default heap sizes. You should try something oddball like 1100m for logstash and 2100m for elasticsearch just to be sure.

Contributor

kenwdelong commented Apr 16, 2017

Just be careful - I think those are the default heap sizes. You should try something oddball like 1100m for logstash and 2100m for elasticsearch just to be sure.

@spujadas

This comment has been minimized.

Show comment
Hide comment
@spujadas

spujadas Apr 16, 2017

Owner

Yep, good point!
I had checked the defaults for Logstash (-Xms256m -Xss2048k) but not for Elasticsearch (just checked, turns out they're -Xms2g -Xmx2g, so indeed the same values I'd used in my test 🙄).

Anyway just retested with:

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ES_HEAP_SIZE=2100m
    - LS_HEAP_SIZE=1100m

Which (fortunately) gives us:

# ps aux | less -+S
...
elastic+    63 55.8 56.2 4790160 1153296 ?     Sl   04:57   0:15 /usr/bin/java -
Xms2100m -Xmx2100m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -
server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -D
jdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKe
ySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdow
nHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDum
pOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib
/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.El
asticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/
var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.
conf=/etc/elasticsearch
...
logstash   194  130 15.8 3516944 325476 ?      SNl  04:57   0:13 /usr/bin/java -
XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X
X:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true
 -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/log
stash -Xmx1100m -Xms1100m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendo
r/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -clas
spath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor
/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /opt/logsta
sh/lib/bootstrap/environment.rb logstash/runner.rb --path.config /etc/logstash/c
onf.d --path.logs /var/log/logstash
...

Thanks for the extra pair of eyes!

Owner

spujadas commented Apr 16, 2017

Yep, good point!
I had checked the defaults for Logstash (-Xms256m -Xss2048k) but not for Elasticsearch (just checked, turns out they're -Xms2g -Xmx2g, so indeed the same values I'd used in my test 🙄).

Anyway just retested with:

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ES_HEAP_SIZE=2100m
    - LS_HEAP_SIZE=1100m

Which (fortunately) gives us:

# ps aux | less -+S
...
elastic+    63 55.8 56.2 4790160 1153296 ?     Sl   04:57   0:15 /usr/bin/java -
Xms2100m -Xmx2100m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -
server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -D
jdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKe
ySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdow
nHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDum
pOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib
/elasticsearch-5.3.0.jar:/opt/elasticsearch/lib/* org.elasticsearch.bootstrap.El
asticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/
var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.
conf=/etc/elasticsearch
...
logstash   194  130 15.8 3516944 325476 ?      SNl  04:57   0:13 /usr/bin/java -
XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X
X:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true
 -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/log
stash -Xmx1100m -Xms1100m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendo
r/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -clas
spath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor
/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /opt/logsta
sh/lib/bootstrap/environment.rb logstash/runner.rb --path.config /etc/logstash/c
onf.d --path.logs /var/log/logstash
...

Thanks for the extra pair of eyes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment