Skip to content
This repository has been archived by the owner. It is now read-only.

Permission denied on bin/es-docker if ran with user remapping #49

Closed
jensdebruijn opened this issue Apr 10, 2017 · 21 comments
Closed

Permission denied on bin/es-docker if ran with user remapping #49

jensdebruijn opened this issue Apr 10, 2017 · 21 comments
Assignees

Comments

@jensdebruijn
Copy link

@jensdebruijn jensdebruijn commented Apr 10, 2017

  • docker info output:
    Containers: 85 Running: 0 Paused: 0 Stopped: 85 Images: 336 Server Version: 17.04.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/1000.1000/aufs Backing Filesystem: extfs Dirs: 372 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73 runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228 init version: 949e6fa Security Options: apparmor seccomp Profile: default userns Kernel Version: 4.8.0-46-generic Operating System: Ubuntu 16.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 31.33GiB Name: workBuntu ID: 5N25:YF6I:3CKB:YMIE:G4QQ:QYDD:PTVZ:TXP4:7YMH:IKR2:Q4XX:K4FO Docker Root Dir: /var/lib/docker/1000.1000 Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support

  • docker-compose version:
    docker-compose version 1.12.0, build b31ff33

  • Host OS and version:
    Ubuntu 16,04

Bug description

I am running elasticsearch with user remapping for docker as such:

https://blog.yadutaf.fr/2016/04/14/docker-for-your-users-introducing-user-namespace/

Running the image with docker-compose gives me the following error:

/bin/bash: bin/es-docker: Permission denied
elasticsearch exited with code 126

@dliappis dliappis self-assigned this Apr 10, 2017
@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Apr 10, 2017

@jens1245 Could you please provide your exact docker-compose file (or docker run command?)?

@jensdebruijn

This comment has been minimized.

Copy link
Author

@jensdebruijn jensdebruijn commented Apr 10, 2017

Certainly!:

version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet

volumes:
esdata:
driver: local

networks:
esnet:
driver: bridge

@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Apr 10, 2017

@jens1245 I reformatted your docker-compose.yml below for readability, but other than that this will work fine.

You are using a named volume called esdata.
If you haven't used that volume before this should work out of the box.
If you have used the same esdata: storage volume before, say with a different image and hence having dirs with different ownerships, perhaps ES won't be able to write there.
If you do not mind losing the data currently stored in esdata:, can you try wiping it away with:

docker-compose down -v

(-v will delete the volume)
and then start it again with:

docker-compose up

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.3.0
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    mem_limit: 1g
    cap_add:
      - IPC_LOCK
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

volumes:
  esdata:
    driver: local

networks:
  esnet:
    driver: bridge
@jensdebruijn

This comment has been minimized.

Copy link
Author

@jensdebruijn jensdebruijn commented Apr 10, 2017

I used:

docker-compose down -v

and rebuilt the image, but I still get the same error.

Did you set the user mapping?

{
        "userns-remap": "yadutaf"
}

in /etc/docker/daemon.json?

@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Apr 10, 2017

@jens1245 No I didn't set the userns-remap option. Would you care describing what are you trying to achieve with this option, for your use case? The docker ES image, defaults to a specific non-privileged user (elasticsearch uid:1000, gid: 1000) inside the container, with no sudo access. The ES wrapper script /bin/es-docker starts using this user, so this limits the attack surface, if it's security that has you looking at the userns-remap option.

@jensdebruijn

This comment has been minimized.

Copy link
Author

@jensdebruijn jensdebruijn commented Apr 10, 2017

I set the user mapping for another container. This way it is easy to share data with the host and at the same time limits the attack surface. Thus, in this case, it is not so much for Elasticsearch but for for overall security / usability. Unfortunately, I do not know of a way to use user remapping only for some containers.

Considering the feature is quite new, but received enthusiastically, it would be nice if the image works for this user case.

@jegade

This comment has been minimized.

Copy link

@jegade jegade commented Apr 15, 2017

I'm running in the same problem. I modified the uid/gid because i remap uid 1000 to root, which is great for other containers, but failed for elasticsearch. 1000 because i own the uid on the development maschine and can share code into the container.

I build my own Dockerfile and change the uid/gid to some other and everything works for now

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
# Switch to root for install
USER root
#Hack
RUN echo http://dl-2.alpinelinux.org/alpine/edge/community/ >> /etc/apk/repositories
RUN apk add --no-cache shadow && \ 
 usermod -u 555  elasticsearch && \
 groupmod -g 555 elasticsearch && \
	find . -user  1000 -exec chown -h 555 {} \; && \
	find . -group 1000 -exec chgrp -h 555 {} \; && \
	usermod -g 555 elasticsearch

# Switch back 
USER elasticsearch 

The other way, was to change my user id on the host, then 1000 will map not to 0

@jegade

This comment has been minimized.

Copy link

@jegade jegade commented Apr 15, 2017

Another temporary workaround is to write your own Dockerfile, based on the history from elasticsearch

FROM alpine

RUN apk --no-cache  add bash curl openjdk8 openssl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
RUN adduser -D -u 555 -h /usr/share/elasticsearch elasticsearch                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             

ARG ELASTIC_VERSION
ARG ES_DOWNLOAD_URL
ARG ES_JAVA_OPTS

ENV ELASTIC_VERSION=5.3.0 
ENV ES_DOWNLOAD_URL=https://artifacts.elastic.co/downloads/elasticsearch 
ENV PATH /usr/share/elasticsearch/bin:$PATH
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk

WORKDIR /usr/share/elasticsearch

# Download/extract defined ES version. busybox tar can't strip leading dir.
RUN wget ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    EXPECTED_SHA=$(wget -O - ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz.sha1) && \
    test $EXPECTED_SHA == $(sha1sum elasticsearch-${ELASTIC_VERSION}.tar.gz | awk '{print $1}') && \
    tar zxf elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    chown -R elasticsearch:elasticsearch elasticsearch-${ELASTIC_VERSION} && \
    mv elasticsearch-${ELASTIC_VERSION}/* . && \
    rmdir elasticsearch-${ELASTIC_VERSION} && \
    rm elasticsearch-${ELASTIC_VERSION}.tar.gz

RUN set -ex && for esdirs in config data logs; do \
        mkdir -p "$esdirs"; \
        chown -R elasticsearch:elasticsearch "$esdirs"; \
    done

# Install Plugin
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin list
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

# Remove x-pack 'https://github.com/elastic/elasticsearch-docker/issues/35#issuecomment-285912424
# RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove  x-pack
RUN rm -rf plugins/x-pack


USER elasticsearch

COPY elasticsearch.yml config/
COPY log4j2.properties config/
COPY bin/es-docker bin/es-docker

USER root
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml config/log4j2.properties bin/es-docker && \
    chmod 0750 bin/es-docker

USER elasticsearch
CMD ["/bin/bash", "bin/es-docker"]
@jensdebruijn

This comment has been minimized.

Copy link
Author

@jensdebruijn jensdebruijn commented Apr 16, 2017

Thank you so much!

I've slightly changed the Docker file you provided for my purpose (with X-pack) and it all works now:

FROM alpine

RUN apk --no-cache  add bash curl openjdk8 openssl
RUN adduser -D -u 555 -h /usr/share/elasticsearch elasticsearch

ARG ELASTIC_VERSION
ARG ES_DOWNLOAD_URL
ARG ES_JAVA_OPTS

ENV PATH /usr/share/elasticsearch/bin:$PATH
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk

WORKDIR /usr/share/elasticsearch

# Download/extract defined ES version. busybox tar can't strip leading dir.
RUN wget ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    EXPECTED_SHA=$(wget -O - ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz.sha1) && \
    test $EXPECTED_SHA == $(sha1sum elasticsearch-${ELASTIC_VERSION}.tar.gz | awk '{print $1}') && \
    tar zxf elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    chown -R elasticsearch:elasticsearch elasticsearch-${ELASTIC_VERSION} && \
    mv elasticsearch-${ELASTIC_VERSION}/* . && \
    rmdir elasticsearch-${ELASTIC_VERSION} && \
    rm elasticsearch-${ELASTIC_VERSION}.tar.gz

RUN set -ex && for esdirs in config data logs; do \
        mkdir -p "$esdirs"; \
        chown -R elasticsearch:elasticsearch "$esdirs"; \
    done

USER elasticsearch

# Install xpack
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch x-pack

COPY elasticsearch.yml config/
COPY log4j2.properties config/
COPY bin/es-docker bin/es-docker

USER root
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml config/log4j2.properties bin/es-docker && \
    chmod 0750 bin/es-docker

USER elasticsearch
CMD ["/bin/bash", "bin/es-docker"]

@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Apr 20, 2017

Another temporary workaround is to write your own Dockerfile, based on the history from elasticsearch

@jegade You do not need to use the docker history command to see how the image has been built, you can always refer to the Dockerfile used in the repo.

Additionally, while creating your customized image, you may also use the existing image as a base and add your own customizations to reduce repetition. One example is provided in the documentation here.

@jegade

This comment has been minimized.

Copy link

@jegade jegade commented Apr 20, 2017

@dliappis sorry, the refered Dockerfile does not contains the whole history. This is based on FROM docker.elastic.co/elasticsearch/elasticsearch-alpine-base:latest - where the USER is applied.

Changing the uid for the existing image does not work for me. If you know a solution, this would be great.

@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Apr 20, 2017

This is based on FROM docker.elastic.co/elasticsearch/elasticsearch-alpine-base:latest - where the USER is applied.

That is correct. This image is also in a public repo and you can find the Dockerfile here.

One strategy for changing the uid/gid in a dynamic way, when you build the image, is to use build args.
Here is an example:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN sed -r -i "s/^elasticsearch:x:\d+:\d+:(.*)\$/elasticsearch:x:$CUSTOM_UID:$CUSTOM_GID:\1/" /etc/passwd && \
    sed -r -i "s/^elasticsearch:x:\d+:(.*)\$/elasticsearch:x:$CUSTOM_GID:\1/" /etc/group && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch

For example, in a fresh VM where I have a user with uid=1001, gid=1001, I built a customized image with:

$ id
uid=1001(elastic) gid=1001(elastic) groups=1001(elastic),1002(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ docker build --build-arg CUSTOM_UID=$(id -u) --build-arg CUSTOM_GID=$(id -g) -t myes .

This results in a working container:

[elastic@fedora25vbox ~]$ docker run --rm myes
[2017-04-20T14:28:11,704][INFO ][o.e.n.Node               ] [] initializing ...
[2017-04-20T14:28:12,835][INFO ][o.e.e.NodeEnvironment    ] [pRZ41B3] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:0-8792783-ab2ae70e9d0911287fa77759bf0d62dcd73e613a03303d38f9e09327bc52ef9c)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [unknown], types [xfs]
@anshitabharti

This comment has been minimized.

Copy link

@anshitabharti anshitabharti commented Jun 16, 2017

@dliappis if i try the same with 5.4.1 version i still face Permission denied on bin/es-docker
but in 5.3.0 this works. Any changes to be done in the Dockerfile ?

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.1
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN sed -r -i "s/^elasticsearch:x:\d+:\d+:(.*)\$/elasticsearch:x:$CUSTOM_UID:$CUSTOM_GID:\1/" /etc/passwd && \
    sed -r -i "s/^elasticsearch:x:\d+:(.*)\$/elasticsearch:x:$CUSTOM_GID:\1/" /etc/group && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch
@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Jun 16, 2017

@anshitabharti Sure; starting with 5.4.0, the images are based on centos:7, so the /etc/{passwd,group} format is a bit different, but on the plus side it's easier to change the uid/gid like so:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.1
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN groupmod -g $CUSTOM_UID elasticsearch && \
    usermod -u $CUSTOM_GID elasticsearch && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch
@anshitabharti

This comment has been minimized.

Copy link

@anshitabharti anshitabharti commented Jun 16, 2017

@dliappis This worked. Thank you!

@perttuta

This comment has been minimized.

Copy link

@perttuta perttuta commented Jun 22, 2017

The above did not work for me. The file system fails to return the right file ownership for directories under /usr/share/elasticsearch and thus ElasticSearch fails to start. I had to use the following workaround to get chown'd files to be visible correctly to elasticsearch user:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.1
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN cp -r /usr/share/elasticsearch /tmp/silly_aufs_fix && \
    rm -rf /usr/share/elasticsearch && \
    cp -r /tmp/silly_aufs_fix /usr/share/elasticsearch && \
    groupmod -g $CUSTOM_UID elasticsearch && \
    usermod -u $CUSTOM_GID elasticsearch && \
    chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
USER elasticsearch

I guess this workaround forces AUFS to make another copy of the files and chown them so that changes are visible in my extended Docker image.

Another workaround I found is to run ls /usr/share/elasticsearch command as root on live container before running elasticsearch as elasticsearch user.

@dliappis

This comment has been minimized.

Copy link
Contributor

@dliappis dliappis commented Jun 22, 2017

@perttuta Thanks for sharing, what seems to be, an aufs specific problem! I believe your are hitting this aufs issue[1].

I was also able to reproduce it on a ubuntu:16.04 virtualbox vm using aufs.
My modinfo shows:

root@ubuntu1604vbox:~# modinfo aufs
filename:       /lib/modules/4.4.0-81-generic/kernel/fs/aufs/aufs.ko
alias:          fs-aufs
version:        4.x-rcN-20160111
description:    aufs -- Advanced multi layered unification filesystem
author:         Junjiro R. Okajima <aufs-users@lists.sourceforge.net>
license:        GPL
srcversion:     BD1C47D04FE5C2D88F63AAB
depends:        
intree:         Y
vermagic:       4.4.0-81-generic SMP mod_unload modversions 
parm:           brs:use <sysfs>/fs/aufs/si_*/brN (int)
parm:           allow_userns:allow unprivileged to mount under userns (bool)

And I assume the patch hasn't made it to the aufs version shipped with the latest Ubuntu 16.04 (aufs >=4.(x>=1)-20160905)

If it is possible in your environment, you may want to consider the following recommendation from Docker:

If your Linux kernel is version 4.0 or higher, and you use Docker CE, consider using the newer overlay2, which has potential performance advantages over the aufs storage driver.

Using the devicemapper, overlay or overlay2 storage driver, I couldn't reproduce the problem on Ubuntu 16.04.

As this issue has slightly become a sink for all issues generally related to permissions, I'll take the opportunity to layout some of the general options:

  1. The best practice for data dir persistence for ES in Docker is using a named docker data volume as mentioned in the production mode example. This can be achieved, on the cli, as easily as docker run -v esdata:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.4.2 and will be automatically re-created or reused if it already exists.

  2. When mounting a host dir as a data volume, it needs to be readable/writable by 1000:1000

  3. A number of volume plugins are available to utilize enterprise block storage backends.

Notes

[1] See also modifying files or directories on aufs:

Renaming directories: Calling rename(2) for a directory is not fully supported on AUFS. It returns EXDEV (“cross-device link not permitted”), even when both of the source and the destination path are on a same AUFS layer, unless the directory has no children. Your application needs to be designed to handle EXDEV and fall back to a “copy and unlink” strategy.

Related to: moby/moby#1295 (comment)

@treemore

This comment has been minimized.

Copy link

@treemore treemore commented Aug 9, 2017

my remote test server is root@ubuntu1604 . and i use docker.elastic.co/elasticsearch/elasticsearch:5.5.1 this image

USER root
RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
USER elasticsearch

the code above can not change any ownership at all in the container.
it is make me crazy because the same code in my mac can work perflect.
and i change the docker storage drive to overlay2. the problem is still

last time , i found the below post
https://stackoverflow.com/questions/29245216/write-in-shared-volumes-docker/29251160#29251160
they told me ,the linux is something different.

so i just in the docker host cmd run chown 1000:1000 the data dir and then i can docker-compose up the elastic service.

but i think maybe there is more perflect solve solution. i use an other image https://hub.docker.com/_/mongo/ . just set an volumes in the docker-compose.yml and all nothing is ok .Heavy did not encounter the problem of Permission

@ballinette

This comment has been minimized.

Copy link

@ballinette ballinette commented Sep 22, 2017

Hi.
I reproduced the same issue even with overly2 storage driver.

Server Version: 17.06.2-ce
Storage Driver: overlay2
Kernel Version: 4.10.0-33-generic
Operating System: Ubuntu 16.04.3 LTS

If I run the docker daemon without the user remapping option, the folder /usr/share/elasticsearch within the container belongs to elasticsearch user, as expected:

$ docker run docker.elastic.co/elasticsearch/elasticsearch:5.4.1 ls -ln /usr/share |grep elastic
drwx------  1 1000 1000 4096 Jun 14 08:00 elasticsearch

but if I run it with the user remapping option, while mapping to 1000 uid (my host user's uid), the same folder now belongs to root:

$ docker run docker.elastic.co/elasticsearch/elasticsearch:5.4.1 ls -ln /usr/share |grep elastic
drwx------  1 0 0 4096 Jun 14 08:00 elasticsearch

Now, if I run the daemon while remapping user to another uid (for example 1981), the foledr's ownership is OK again.

I really do not know how user remapping is implemented by docker, but what I understand is that there is a bug when a user into the container has exactly the same uid as the user remapping uid.

So, as a workaround, I changed my uid/gid in my host and now it works like a charm ;)

@treemore

This comment has been minimized.

Copy link

@treemore treemore commented Sep 30, 2017

@ballinette
I am not sure, I later compared with the mongo docker different, I think it is likely to need to add the data directory to. Dockerignore file. You can try it. I was also replaced by overlay2 did not work

@rjernst

This comment has been minimized.

Copy link
Member

@rjernst rjernst commented Jun 26, 2019

This issue is quite old. Maintenance of the docker files for elasticsearch has moved to the elasticsearch repo. Any future issues can also be opened in the forums, where there is an active community ready to assist. As we will be archiving this repository, I am going to close this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.