Elasticsearch won't start when path.data or path.logs env var are defined

[LudovicTOURMAN]

Bug Description

When defining a custom data/logs path through environment variables (As described here), elasticsearch failed to start as it try to write as elasticsearch [1000] user in a folder owned by root [0] user.
Using default Dockerfile configuration, the entrypoint allow to chown data/logs folders through the TAKE_FILE_OWNERSHIP env var.

This block part should be changed to also chown paths defined through env vars path.data and path.logs.

Bug Information

• Docker image used : docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4
• Operating System : Ubuntu 17.10
• docker info output :

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 101
Server Version: 18.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 154
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-43-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.678GiB
Name: tamarindo
ID: KAFE:5AIR:6S5S:QNDV:XVRM:E6UZ:Z547:6X3L:6EQ3:UG2T:Y5FG:LBFJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

• docker inspect NAME|ID output :

[
    {
        "Id": "6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f",
        "Created": "2018-06-06T15:39:15.967389523Z",
        "Path": "/usr/local/bin/docker-entrypoint.sh",
        "Args": [
            "eswrapper"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2018-06-06T15:39:16.484400625Z",
            "FinishedAt": "2018-06-06T15:39:18.597261868Z"
        },
        "Image": "sha256:3822ba554fe95f9ef68baa75cae97974135eb6aa8f8f37cadf11f6a59bde0139",
        "ResolvConfPath": "/var/lib/docker/containers/6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f/hostname",
        "HostsPath": "/var/lib/docker/containers/6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f/hosts",
        "LogPath": "/var/lib/docker/containers/6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f/6c60f7e4a67f322a6b1cbe97f4703710c9215bb8dd80868a777bf1d1b871422f-json.log",
        "Name": "/es",
        "RestartCount": 0,
        "Driver": "aufs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": null,
            "Name": "aufs"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "6c60f7e4a67f",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "9200/tcp": {},
                "9300/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "path.logs=/home/logs",
                "discovery.type=single-node",
                "path.data=/home/data",
                "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ELASTIC_CONTAINER=true",
                "JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk"
            ],
            "Cmd": [
                "eswrapper"
            ],
            "ArgsEscaped": true,
            "Image": "docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4",
            "Volumes": null,
            "WorkingDir": "/usr/share/elasticsearch",
            "Entrypoint": [
                "/usr/local/bin/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "Elastic Docker Team <docker@elastic.co>",
                "org.label-schema.schema-version": "= 1.0     org.label-schema.name=CentOS Base Image     org.label-schema.vendor=CentOS     org.label-schema.license=GPLv2     org.label-schema.build-date=20180402"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "78435898aa74284efdbfd4e95eca824e1bd62fcb3120349c8b0749a1454f7c39",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/78435898aa74",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "697916e489e10e8342f1c17b0aa0af50038de7bc5345c250667f3a1c3b6feced",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

• Steps to reproduce :
  • Run docker run -it --rm --name es -e "discovery.type=single-node" -e "path.data=/home/data" -e "path.logs=/home/logs" docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4

[jarpy]

That's nasty, thanks for the report.

It will be interesting to fix completely too, since environment variables are only one way to set those values. If the user sets them that way, the startup script can see it easily and act accordingly. If, however, they set path.data in elasticsearch.yml, it's harder for the wrapper to notice.

[jarpy]

So, in practice, is there any need to mount the data dir to somewhere other than /usr/share/elasticsearch/data inside the container?

Can you help me understand the scenario where a user would need to mount it elsewhere?

[dliappis]

@LudovicTOURMAN one thing to note is that at least in your reproduction example you are assuming the directories exist in the container, but you don't seem to be using a bind mount or a volume.

If instead you create the dirs locally and ensure they have GID 0 access (item 1 in notes), things start working again:

mkdir -p es/{data,logs} && sudo chgrp -R 0 es/ && docker run -it --rm --name es -v $PWD/es/data:/home/data -v $PWD/es/logs:/home/logs -e "discovery.type=single-node" -e "path.data=/home/data" -e "path.logs=/home/logs" docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4

You are correct that the TAKE_FILE_OWNERSHIP env var right now doesn't know about changes in paths (bug/hard assumption), however, ideally this flag should really only be used as a last resort, as mentioned in the docs, as it has the potential to mutate the underlying file system from within the container.

[LudovicTOURMAN]

1. For your first point (elasticsearch.yml), I don't knon what is your internal thought about it, but theoric way should be to look if env var is empty, if yes, "grep" (or someway else) inside the file.
   This will raise other point like :

• Where is the elasticearch.yml file (As it's location could also rely on ES_PATH_CONF var)
• How to get the value as there is 2 valid format inside the file :

  path.data:

  and

  path:
    data:

Personnaly I use the yq library to properly get values from yaml files. It relies on jq which both are doing great jobs.
But I understand that you might not wan't to add an extra third library in your docker image just to manage a yaml value.

2. My use case is that I'm using your docker image with Kubernetes.
   As I want to persist those datas and logs on a same persistent volume, I'm binding those elsewhere as Kubernetes will mount the entire folder and I don't want to mess/delete original files and folder located in /usr/share/elasticsearch
   Long story short, my Kubernetes pod descriptor is part of an Helm Chart that I want to keep as much as generic possible. Avoiding it's configuration to be reliable on a specific docker image.
   I'm just suprised that it was working on a non-official elasticsearch docker image (Pires one).
   So now, trying to use your official image required me to do some specific changes in my Kubernetes descriptor to mount those volumes through the mountpath/subpath K8s options.

[kphunter]

@LudovicTOURMAN did you develop a workaround?

I'm trying to change a static configuration setting (max_content_length) when deploying to k8s via helm (using the official Elasticsearch docker container), but I'm having a hard time understanding how to pass the env variable via the helm values.yaml

[rjernst]

This issue seemed to be split into several related threads. As noted before, bind mounts should be used for paths. Regarding kubernetes, helm charts have been added for elasticsearch, as well as work on [kubernetes https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond).

Maintenance of the docker files for elasticsearch has moved to the elasticsearch repo. Please open any futher issues there. As we will be archiving this repository, I am going to close this issue.