Documentation precision about heap size and docker

[soullivaneuh]

This is not really a bug report nor a feature request, but a possible documentation issue.

On the docker-compose guideline, you recommend to use ES_JAVA_OPTS, but not on the jvm.options configuration guide.

Well, the env way looks better with docker. However, I saw a notice on this guide:

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS

But the options are present on the config file from the official docker image. So, to make it working, I have to build a custom one removing those options:

FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.2 as search
RUN sed --in-place '/^-Xm[s,x]/d' config/jvm.options

Otherwise, the env usage does not work at all.

I'm maybe missing something, but the options should not be configured by default on the docker image if you recommend to use ENV vars, right?

I think the Dockerfile and/or the docs need some update/clarification.

Thoughts?

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[elasticmachine]

Pinging @elastic/es-docs (>docs)

[jasontedor]

Thanks for opening this issue @soullivaneuh. A few comments:

• jvm.options is fine for Docker usage, we should document that (bind mount it into the container, over /usr/share/elasticsearch/config/jvm.options)
• it's okay that -Xms1g and -Xmx1g are present in the default jvm.options file in the container, as ES_JAVA_OPTS will override jvm.options, by design; you'll see both flags on the command line, but with the JVM, the last flag wins (so -Xms1g -Xms2g would end up with a 2g heap)
• this means that you don't need a custom image

We'll get the docs updated to clarify these points.

[soullivaneuh]

Maybe I didn't well understand how it works, but here is a ps:

gallart  19254  0.4  9.5 4021760 779140 ?      SLsl Jan29   4:22 /opt/jdk-13.0.1+9/bin/java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-12316401923526476962 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Des.cgroups.hierarchy.override=/ -Xms256m -Xmx256m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=fusionauth -Ebootstrap.memory_lock=true

Here i removed the default option and put the env vars.

As you can see, the container consumes ~780Mo, this is far more than the limit I set. Also, if I setup multiple instances (staging server with auto-deploy) and the host RAM is full, the server is overloading consuming all the CPU resources for no reasons.

jvm.options is fine for Docker usage, we should document that (bind mount it into the container, over /usr/share/elasticsearch/config/jvm.options)

Well, the issue is jvm.options from default image is quite full of options. I don't want to get rid of them, and it's quite ugly to copy a vendor config on our git project just to remove/change 2 settings. Don't you have something like jvm.options.d or something similar for override?

[pugnascotia]

@soullivaneuh can I confirm which image you are using - I pulled and ran docker.elastic.co/elasticsearch/elasticsearch:6.3.2, and the ps output contains:

/opt/jdk-10.0.2/bin/java

which is different from your sample.

[pugnascotia]

Note that using ps doesn't report the Java heap size, because Elasticsearch and the JVM need memory for purposes other than the JVM heap. You can verify what heap size Elasticsearch is using with the /_nodes API e.g.

curl localhost:9200/_nodes?human | jq '.nodes | map_values(.jvm.mem)'

In all my experiments, configuring the heap settings via an environment variable e.g. ES_JAVA_OPTS='-Xms256m -Xmx256m' did what I expected, but please let me know if you try this and the /_nodes API doesn't give the correct number. You can also constrain total memory usage with Docker options.

I'll see about what we can do to improve the documentation.

[jasontedor]

Don't you have something like jvm.options.d or something similar for override?

When we initially designed jvm.options (#17675), we had considered this. At the time, designing for Docker use-cases wasn't something that we considered, although now it is. We've seen some other problems over the years (e.g., package users modifying their jvm.options and not getting new options when they upgrade because their package manager will preserve their modified jvm.options instead of taking the package-provided jvm.options). We've partially addressed this with the introduction of jvm.options (#48252) but something like jvm.options.d would indeed make that experience even better.

I opened #51882.