Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalog is created with wrong user and group root:root inside Docker #2904

Open
afiskon opened this issue Apr 19, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@afiskon
Copy link

commented Apr 19, 2019

We use Docker in our Jepsen tests and observe a very weird behavior. Sometimes catalogs inside the container are created with wrong user and group root:root with no obvious reason. For instance:

go: gopkg.in/yaml.v2@v2.0.0: mkdir /home/gopher/go/pkg/mod/cache: permission denied
go: error loading module requirements
Makefile:60: recipe for target 'install-godep' failed
make: *** [install-godep] Error 1
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ ls -la /home/gopher/go/pkg/
total 20
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 16:01 .
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:38 ..
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:39 dep
drwxr-xr-x. 2 root   root   4096 Apr 19 16:01 mod
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ rm -r /home/gopher/go/pkg/mod
rm: remove write-protected directory '/home/gopher/go/pkg/mod'? y
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ mkdir /home/gopher/go/pkg/mod
gopher@c8e8aeac7f36:~/go/src/github.com/insolar/insolar$ ls -la /home/gopher/go/pkg/
total 20
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 16:02 .
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:38 ..
drwxr-xr-x. 1 gopher gopher 4096 Apr 19 15:39 dep
drwxr-xr-x. 2 root   root   4096 Apr 19 16:02 mod

Steps to reproduce - follow the steps according to README.md in this repository https://github.com/insolar/insolar-jepsen

Environment:

$ docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:13:40 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ sudo kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux fake-host-name 4.4.178-1.el7.elrepo.x86_64 #1 SMP Wed Apr 3 05:46:30 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
@RomanSteinberg

This comment has been minimized.

Copy link

commented Apr 22, 2019

Hi,

there is a hypothesis that the problem occurred because of your Dockerfile line 27 and it is not a docker issue. You switched into root to get ssh access and after that all actions would take place with root privileges in the container.

Consider, another workflow. Run services as a root and then switch into gopher user. Another recommendations:

  • don't use sudo docker commands, use gosu or su-exec,
  • use entrypoints.

PS: #devzen calls for collaboration better then open source 😄

@afiskon

This comment has been minimized.

Copy link
Author

commented Apr 22, 2019

@RomanSteinberg

Thank you for the reply. This is an interesting hypothesis however I'm afraid it's most likely not right.

The problem appeared on the last line of https://github.com/insolar/insolar-jepsen/blob/master/base-image/Dockerfile:

RUN make install-deps pre-build

By like 27 you probably meant the last line of https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile

CMD ["/usr/bin/sudo", "/usr/sbin/sshd", "-D"]

However this Dockerfile uses the first one as a base image. Thus this line couldn't cause the problem because it wasn't executed yet.

Also it worth noticing that the problem doesn't occur on Docker Desktop Community for MacOS 2.0.0.3 (31259), nor does it occur on Ubuntu 18.04 LTS with Docker 18.09.2 and Kubernetes 1.14. Which suggests that Dockerfiles are most likely OK.

We observed the issue only in the environment described in the first message.

@RomanSteinberg

This comment has been minimized.

Copy link

commented Apr 22, 2019

By line 27 you probably meant the last line of https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile

Right. I forgot to specify the file.

@RomanSteinberg

This comment has been minimized.

Copy link

commented Apr 23, 2019

@afiskon
It looks like my hypothesis was wrong according to your last answer. But the example you provided is not minimal as soon as you have files which are not even executed (https://github.com/insolar/insolar-jepsen/blob/master/Dockerfile). So, I would like to suggest you to make a minimal example as much as it possible, because it will help people to reproduce the problem faster (obvious I think). Good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.