Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't run KIND on ChromeOS Linux VM #763

Open
adrianludwin opened this issue Aug 9, 2019 · 15 comments
Open

Can't run KIND on ChromeOS Linux VM #763

adrianludwin opened this issue Aug 9, 2019 · 15 comments

Comments

@adrianludwin
Copy link

@adrianludwin adrianludwin commented Aug 9, 2019

What happened: I tried to run kind (v0.4.0) on the Linux VM in Chrome OS 75.0.3770.144:

aludwin@penguin:~$ kind create cluster
Creating cluster "kind" ...
 βœ“ Ensuring node image (kindest/node:v1.15.0) πŸ–Ό
 βœ“ Preparing nodes πŸ“¦ 
 βœ— Creating kubeadm config πŸ“œ 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

Based on the error and a quick look at the source code, I decided to try to run kindest/node manually, and got the following result:

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

I suspect (but have not tried to confirm) that this comes from the following lines in images/base/entrypoint:

fix_mount() {
  echo 'INFO: ensuring we can execute /bin/mount even with userns-remap'
  # necessary only when userns-remap is enabled on the host, but harmless
  # The binary /bin/mount should be owned by root and have the setuid bit
  chown root:root /bin/mount
  chmod -s /bin/mount

  # This is a workaround to an AUFS bug that might cause `Text file
  # busy` on `mount` command below. See more details in
  # https://github.com/moby/moby/issues/9547
  sync

  echo 'INFO: remounting /sys read-only'
  # systemd-in-a-container should have read only /sys
  # https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
  # however, we need other things from `docker run --privileged` ...
  # and this flag also happens to make /sys rw, amongst other things
  mount -o remount,ro /sys

What you expected to happen: Cluster gets created.

How to reproduce it (as minimally and precisely as possible):

If my suspicion that the problems is in the entrypoint is correct, docker run kindest/node:v1.15.0 -it /bin/bash should be sufficient. If that's wrong, kind create cluster reliably reproduces.

Anything else we need to know?:

Environment:

  • kind version: v0.4.0
  • Kubernetes version: n/a
  • Docker version: 19.03.1
  • OS: Chrome OS 75.0.3770.144. VM is running Debian 9.
@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 9, 2019

We need to do that remount.

There is also a previous bug for this. I don't think kind will work inside the chromeOS sandbox at the moment.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 9, 2019

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

Permission is denied because this command line is missing flags related to that (which kind will include).

If you run kind with --loglevel=debug you can see more detail including this command.

@BenTheElder BenTheElder changed the title Can't run KIND on ChromeOS Linux VM, likely due to being unable to mount `sys` Can't run KIND on ChromeOS Linux VM Aug 9, 2019
@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 9, 2019

@adrianludwin

This comment has been minimized.

Copy link
Author

@adrianludwin adrianludwin commented Aug 9, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 9, 2019

We don't know what the blocker is yet, and I don't have access to a setup suitable for testing this. The existing bug doesn't have enough details either.

This may not even be considered a bug on their end. Kubernetes / the container runtimes need relatively elevated permissions to run at the moment, I suspect the sandbox is too secure :+)

@adrianludwin

This comment has been minimized.

Copy link
Author

@adrianludwin adrianludwin commented Aug 9, 2019

Re #462: I did see that bug but I thought this was something different. My computer successfully got past the "preparing nodes" step but died on the next "Creating kubeadm config" step.

Can I provide any more details that would help with this? Agreed that there's a chance ChromeOS won't want to "fix" this but I'm motivated to push on it a bit :)

Here's the result of the debug logs:

aludwin@penguin:~$ kind create cluster --loglevel=debug
DEBU[12:04:09] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}] 
Creating cluster "kind" ...
DEBU[12:04:09] Running: /usr/bin/docker [docker inspect --type=image kindest/node:v1.15.0] 
INFO[12:04:09] Image: kindest/node:v1.15.0 present locally  
 βœ“ Ensuring node image (kindest/node:v1.15.0) πŸ–Ό
DEBU[12:04:09] Running: /usr/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[12:04:09] Running: /usr/bin/docker [docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478] 
 βœ“ Preparing nodes πŸ“¦ 
DEBU[12:04:13] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:13] Running: /usr/bin/docker [docker inspect -f {{index .Config.Labels "io.k8s.sigs.kind.role"}} kind-control-plane] 
DEBU[12:04:14] Running: /usr/bin/docker [docker exec --privileged kind-control-plane cat /kind/version] 
 βœ— Creating kubeadm config πŸ“œ 
DEBU[12:04:14] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:14] Running: /usr/bin/docker [docker rm -f -v kind-control-plane] 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

I'll try to reproduce this more precisely.

@adrianludwin

This comment has been minimized.

Copy link
Author

@adrianludwin adrianludwin commented Aug 9, 2019

Running the docker run command on its own, without -d, has the same result I got initially:

aludwin@penguin:~$ docker run -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478
mount: /sys: permission denied.
@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 10, 2019

How was docker installed?
What other security mechanisms does this system have?
Can you do mounts in a container at all?
If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

@adrianludwin

This comment has been minimized.

Copy link
Author

@adrianludwin adrianludwin commented Aug 11, 2019

How was docker installed?

I followed the instructions for Docker-on-Pixelbook (basically sudo apt-get install docker-ce), using the Debian repo (not Ubuntu). I didn't need any of the stuff about a blacklisted syscall - it Just Worked for me when I installed it.

What other security mechanisms does this system have?

I have no idea. AFAIK it's stock ChromeOS but it's managed by my employer (Google) so they may have some additional stuff on it.

Can you do mounts in a container at all?

I can do a bind mount, yes:

$ mkdir /mnt/test
$ mkdir /home/aludwin
$ echo hi > /mnt/test/foo
$ mount --bind /mnt/test /home/aludwin
$ cat /home/aludwin/foo
hi

My VM and container don't have any block devices other than vdb and I'm not sure how to mount that (mount: /home/aludwin: special device vdb does not exist.). Is this enough to answer your question?

If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

Yup:

root@kind-control-plane:/# mount -l
/dev/vdb on / type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=379,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/btrfs/subvolumes/e0cd2ec9e5417357be09b8d36540d00f596f0f0d127a346faba8f67096bbb990)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=1000000,gid=1000000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755,uid=1000000,gid=1000000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,uid=1000000,gid=1000000)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
/dev/vdb on /etc/resolv.conf type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/resolv.conf)
/dev/vdb on /etc/hostname type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hostname)
/dev/vdb on /etc/hosts type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hosts)
/dev/vdb on /usr/lib/modules type btrfs (ro,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/lib/modules)
/dev/vdb on /var/lib/containerd type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/volumes/56ea3686b15c669ecb98b632b1fc1094e1c6bd2b5679f060ce74f4e48b4c457b/_data)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)
@BenTheElder

This comment has been minimized.

@rosera

This comment has been minimized.

Copy link

@rosera rosera commented Aug 30, 2019

Hey, I am highly interested in getting this working on ChromeOS (I raised the original ticket reference https://bugs.chromium.org/p/chromium/issues/detail?id=878034). Thanks very much for the tips regarding debugging - very helpful.

I plan to investigate a bit more over the coming weekend. My thoughts at the moment relate to the container being restricted on the accessible ports available to Crostini. K8s ports are not on the existing list of default ports. I believe you can enable them, so it might be trial and error for this to work. Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Aug 30, 2019

Hi! πŸ‘‹

K8s ports are not on the existing list of default ports

you can configure kind to use a particular port with kind create cluster --config=config.yaml and config.yaml:

apiVersion: kind.sigs.k8s.io/v1alpha3
kind: Config
networking:
  apiServerPort: 6443

Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

That's a bit more of a blocker.

Each kind "node" docker container is --privileged in order to run containerd / pods on the inside, each kubelet needs its own container runtime. To run pods we need proc, cgroups, mounting ...

It might be possible to hack up kind to use usermode Linux instead but that's going to be fairly involved.

If I read the logs from @adrianludwin correctly though, we actually manage to create those containers, but it fails once we try to use mount inside them. Besides the re-mounting we do*, I believe kubelet needs permission to mount in order to setup pods etc.

* (changing /sys to read only, docker forces /sys to a read-write mount if you specify --privileged),

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 3, 2019

dupe: #1137

I don't really have the capacity to dig into this further at the moment, and I don't think any of our other usual contributors do. AFAICT, the chromeOS sandbox is too restrictive to run Kubernetes.

This is probably generally a good thing πŸ™ƒ as kind is certainly not the most secure thing to be doing, but there's not much we can do here.

At this point I'm going to recommend that we add this to known issues and close out this bug in favor of the upstream chromeOS issue.

@cbandy

This comment has been minimized.

Copy link

@cbandy cbandy commented Dec 3, 2019

If I understand correctly, that upstream bug is targetting minikube (rather than kind specifically) and also it might be ... done/terminal/finished/closed?

Status: Verified (Closed)

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 3, 2019

I don't know if the upstream bug is actually fixed, but if you read through the comments the upstream bug expands to more generally trying to run some form of kubernetes-like-thing including kind, minikube, k3s...

kind is not doing anything special for operating systems otherwise, we only require privileged docker containers and whatever kernel features kubernetes requires.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.