Startup in Docker fails with v8.19.3

[mrimann]

Elasticsearch Version

8.19.3

Installed Plugins

No response

Java Version

bundled

OS Version

Docker

Problem Description

We noticed a problem when starting Elasticsearch in a docker container, with a config-volume bind-mounted since v8.19.3 while prior 8.19.x versions worked fine.

(the same also applied to v9.x where the same issue appeared with v9.1.0 while v9.0.x worked fine. There we could just remove the bind-mounted config-volume as a workaround.)

Steps to Reproduce

• start v8.19.2 with the following minimalistic docker-compose.yml-file:

services:
  app:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.19.2
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2000m -Xmx2000m"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data/
      - elasticsearch-config:/usr/share/elasticsearch/config/
    ports:
      - "127.0.0.1:9200:9200"
    restart: always
    healthcheck:
      test: ["CMD", "curl", "-s", "http://localhost:9200"]
      interval: 30s
      timeout: 10s
      retries: 5
volumes:
  elasticsearch:
    driver: local
  elasticsearch-config:
    driver: local

• run docker compose up
• output shows that the container starts without errors, and after a few seconds, http://localhost:9200 starts to become available
• stop the container (Ctrl-C)
• now change version in the docker-compose.yml file (3rd line) to 8.19.3
• run docker compose up
• instead of a starting container, I see the following output / error message and the container does not start:

[+] Running 1/1
 ✔ Container elasticsearch  Recreated                                                                                                                                                             0.1s
Attaching to elasticsearch
elasticsearch  | java.lang.RuntimeException: starting java failed with [1]
elasticsearch  | output:
elasticsearch  | [0.000s][error][logging] Error opening log file 'logs/gc.log': No such file or directory
elasticsearch  | [0.000s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
elasticsearch  | error:
elasticsearch  | Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m', see error log for details.
elasticsearch  | Error: Could not create the Java Virtual Machine.
elasticsearch  | Error: A fatal exception has occurred. Program will exit.
elasticsearch  | 	at org.elasticsearch.server.cli.JvmOption.flagsFinal(JvmOption.java:125)
elasticsearch  | 	at org.elasticsearch.server.cli.JvmOption.findFinalOptions(JvmOption.java:90)
elasticsearch  | 	at org.elasticsearch.server.cli.MachineDependentHeap.determineHeapSettings(MachineDependentHeap.java:59)
elasticsearch  | 	at org.elasticsearch.server.cli.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:146)
elasticsearch  | 	at org.elasticsearch.server.cli.JvmOptionsParser.determineJvmOptions(JvmOptionsParser.java:98)
elasticsearch  | 	at org.elasticsearch.server.cli.ServerCli.startServer(ServerCli.java:269)
elasticsearch  | 	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:112)
elasticsearch  | 	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
elasticsearch  | 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
elasticsearch  | 	at org.elasticsearch.cli.Command.main(Command.java:54)
elasticsearch  | 	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65)
elasticsearch exited with code 70
elasticsearch exited with code 70
elasticsearch exited with code 70
Gracefully Stopping... press Ctrl+C again to force
 Container elasticsearch  Stopping
 Container elasticsearch  Stopped

---

If I delete both the container + the created volume to start from scratch, then run docker compose up again with v8.19.3, it starts properly and becomes available (and creates a fresh config-volume and files in it).

But since we have productive installations of Elasticsearch running in Docker, with config and/or keystore files within those bind-mounted config-volumes, it's not a real solution to just trash those volumes + start with a fresh volume to work around this issue.

---

I'm not sure, but I see the possibility that it could be related to #124966 which was introduced in v8.19.3 and changes the working directory to the logs directory. Maybe that's related and causes problems when that (mounted) directory is not empty.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[crispygoth]

Not sure if it's related, but recent Docker images seem to have switched /bin/sh from bash to dash, and it looks like dash is more strict on environment variable names. In particular, it does not allow dots in variable names, which means that trying to set ES config via the environment can fail if dash is involved in the container start-up.

I think the default config of the Docker image does explicitly use /bin/bash throughout, but there are going to be situations (e.g. customised entrypoints) where people might be using /bin/sh which will may been broken by this change.

[rjernst]

@mrimann Your issue is indeed related to #124966.

In your config directory do you have changes to jvm.options? If not, when you upgrade Elasticsearch do you copy in the new default jvm.options? The issue you are seeing is due to not taking newer changes from that file.

[mrimann]

@rjernst In the meantime, we've decided to remove the whole config-volume that is mounted to avoid this and potential future issues.

Regarding your question: I guess this could haven been the issue, but I cannot check it for now, as I don't have the setup at hand with the mounted config volume. But it seems to be plausible as we probably just mounted the configuration volume (empty or with an initial copy of what was there within the running ES container). And then never touched those configuration volumes again. So if that jvm.options file was in there, it did not get any changes from us (manually) and if that broke the whole thing, that could be it.

[rjernst]

Given the above I will close this issue. If you encounter this again, with the latest jvm.options, please let us know.