ChromeOS compatability (mount: /sys: permission denied.: container exited unexpectedly)

[tstromberg]

At master:

./out/minikube start --vm-driver=docker
😄  minikube v1.7.0-beta.1 on Debian 9.11
✨  Selecting experimental 'docker' driver from user configuration (alternates: [none])
🔥  Creating Kubernetes in docker container with (CPUs=2), Memory=2000MB (6638MB available) ...

💣  Unable to start VM. Please investigate and run 'minikube delete' if possible: create: creating: prepare kic ssh: apply authorized_keys file ownership, output 
** stderr ** 
Error response from daemon: Container dc8e0447bd40351034d19959e58f9ae2b735e9f45110f2dbae4552a8e8d91157 is not running

** /stderr **: chown docker:docker /home/docker/.ssh/authorized_keys: exit status 1
stdout:

stderr:
Error response from daemon: Container dc8e0447bd40351034d19959e58f9ae2b735e9f45110f2dbae4552a8e8d91157 is not running

Looking at the container logs:

$ docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                      PORTS               NAMES
dc8e0447bd40        gcr.io/k8s-minikube/kicbase:v0.0.2   "/usr/local/bin/entr…"   2 minutes ago       ...

$ docker logs dc8e0447bd40
WARN: /dev/kmsg does not exist, symlinking /dev/console
INFO: ensuring we can execute /bin/mount even with userns-remap
INFO: remounting /sys read-only
mount: /sys: permission denied.

[tstromberg]

NOTE: This issue means that minikube effectively will not work on ChromeOS at this time.

[medyagh]

@tstromberg
I belive the source of the problem is, we generate the SSH key using lib machine, and I believe lib machine tries to generate the key into

// ResolveStorePath returns the store path where the machine is
func (d *BaseDriver) ResolveStorePath(file string) string {
	return filepath.Join(d.StorePath, "machines", d.MachineName, file)
}

see the code bellow. that is in kic.go

// prepareSSH will generate keys and copy to the container so minikube ssh works
func (d *Driver) prepareSSH() error {
	keyPath := d.GetSSHKeyPath()
	glog.Infof("Creating ssh key for kic: %s...", keyPath)
	if err := ssh.GenerateSSHKey(keyPath); err != nil {
		return errors.Wrap(err, "generate ssh key")
	}

	cmder := command.NewKICRunner(d.NodeConfig.MachineName, d.NodeConfig.OCIBinary)
	f, err := assets.NewFileAsset(d.GetSSHKeyPath()+".pub", "/home/docker/.ssh/", "authorized_keys", "0644")
	if err != nil {
		return errors.Wrap(err, "create pubkey assetfile ")
	}
	if err := cmder.Copy(f); err != nil {
		return errors.Wrap(err, "copying pub key")
	}
	if rr, err := cmder.RunCmd(exec.Command("chown", "docker:docker", "/home/docker/.ssh/authorized_keys")); err != nil {
		return errors.Wrapf(err, "apply authorized_keys file ownership, output %s", rr.Output())
	}

	return nil

@tstromberg so I belive on chrome os, the user doesnt have access to to that folder return filepath.Join(d.StorePath, "machines", d.MachineName, file)

[afbjorklund]

@medyagh : This is more of a platform issue, when running on Crostini (the Debian "Linux (Beta)")

https://bugs.chromium.org/p/chromium/issues/detail?id=878034

It comes from the kindbase, that the kicbase inherits. It is the mount: /sys: permission denied.

kubernetes-sigs/kind#763

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tstromberg]

I've just switched my work machine to a Pixelbook Go, so I felt I should update this issue with the latest output on the current kernel release:

Linux penguin 5.4.67-09356-gf3ed4c0c4a31 #1 SMP PREEMPT Sun Sep 27 20:35:52 PDT 2020 x86_64 GNU/Linux

😄  minikube v1.16.0 on Debian 10.5
✨  Using the docker driver based on existing profile
💨  For improved Docker performance, enable the overlay Linux kernel module using 'modprobe overlay'
❗  docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3600MB) ...
✋  Stopping node "minikube"  ...
🔥  Deleting "minikube" in docker ...
🤦  StartHost failed, but will try again: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-05T17:43:30.446193388Z  + echo 'INFO: remounting /sys read-only'
2021-01-05T17:43:30.446210943Z  INFO: remounting /sys read-only
2021-01-05T17:43:30.446240749Z  + mount -o remount,ro /sys
2021-01-05T17:43:30.457168392Z  mount: /sys: permission denied.: container exited unexpectedly
🔥  Creating docker container (CPUs=2, Memory=3600MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-05T17:44:32.940820072Z  + echo 'INFO: remounting /sys read-only'
2021-01-05T17:44:32.940831797Z  INFO: remounting /sys read-only
2021-01-05T17:44:32.940841973Z  + mount -o remount,ro /sys
2021-01-05T17:44:32.940849359Z  mount: /sys: permission denied.: container exited unexpectedly

💢  The minikube docker container exited unexpectedly.
💡  If you are still interested to make docker driver work. The following suggestions might help you get passed this issue:

        - Prune unused docker images, volumes, networks and abandoned containers.

                                docker system prune --volumes

        - Restart your docker service

        - Delete and recreate minikube cluster
                minikube delete
                minikube start --driver=docker

❌  Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: create kic node: container name "minikube": log: 2021-01-05T17:44:32.940820072Z  + echo 'INFO: remounting /sys read-only'
2021-01-05T17:44:32.940831797Z  INFO: remounting /sys read-only
2021-01-05T17:44:32.940841973Z  + mount -o remount,ro /sys
2021-01-05T17:44:32.940849359Z  mount: /sys: permission denied.: container exited unexpectedly

[tstromberg]

I did some playing around, and found that disabling the apparently unnecessary remount gets minikube a bit farther: at least into kubeadm, and starting the kubelet. Unfortunately, it fails eventually, for a reason which I have not been able to root cause yet:

Jan 08 21:55:17 minikube kubelet[1813]: E0108 21:55:17.967566 1813 kubelet.go:1823] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]

To get this far, I'm applying the following diff:

diff --git a/deploy/kicbase/entrypoint b/deploy/kicbase/entrypoint
index 50b784ca0..002b8b42d 100755
--- a/deploy/kicbase/entrypoint
+++ b/deploy/kicbase/entrypoint
@@ -71,7 +71,7 @@ fix_mount() {
   # https://systemd.io/CONTAINER_INTERFACE/
   # however, we need other things from `docker run --privileged` ...
   # and this flag also happens to make /sys rw, amongst other things
-  mount -o remount,ro /sys
+  mount -o remount,ro /sys || echo 'failed to make /sys read-only'
 
   echo 'INFO: making mounts shared' >&2
   # for mount propagation

and running the following command:

make && docker build deploy/kicbase -t kicbase:experimental && ./out/minikube start --base-image=kicbase:experimental --container-runtime=containerd --extra-config=kubelet.v=5

The resulting image still works on regular Debian host, but not yet on ChromeOS.

[eddyjlhaigh]

My work machine is a Pixelbook too, and just ran into this issue when starting a new project. I assume the number of people using Minikube while working on Pixelbooks is fairly small, but will this be looked into soon?

[priyawadhwa]

Hey @eddyjlhaigh this is on our roadmap, but we are waiting for the overlayfs kernel module to be enabled on ChromeOS. Once that module is enabled, minikube should be able to run, but I'm not sure when that can be expected to happen. Regardless, this is definitely on our radar.

[eddyjlhaigh]

@priyawadhwa Thanks for the update! :)

[BenTheElder]

I did some playing around, and found that disabling the apparently unnecessary remount gets minikube a bit farther:

This is fairly necessary and why it is necessary is commented in the original source:
https://github.com/kubernetes-sigs/kind/blob/c234d2b77ae36bd24cfa14af9c18b9bb0810a646/images/base/files/usr/local/bin/entrypoint#L53-L58

/sys should not be rw. It's technically possible to run this way but it is not a great idea, see the references above.
It is somewhat less necessary in KIND @ HEAD due to kubernetes-sigs/kind@42761be

See also: kubernetes-sigs/kind#763

[rckgomz]

any news on this?

[afbjorklund]

I don't think minikube will work on ChromeOS before it does with kind, and as far as I know there is not anyone actively looking into it. So I think it is blocked on upstream, but if there is anything that we can do in minikube to move it along then please update.

[anindya6]

Need more eyes on this.

[afbjorklund]

Supporting ChromeOS is not a top priority at the moment, I suppose one could use the "ssh" driver with remote nodes.

[CodeByEddy]

Even if this isn't a priority, is it still on the roadmap?

[sharifelgamal]

Our maintainers don't currently have any bandwidth to work on this, so it's not on the roadmap as such, but if someone were to add a PR to attempt to support ChromeOS, we'd be happy to review.

[ben700]

Just another waiting for fix; I assumed I'd be able to use Google hardware to build Google cloud

[BenTheElder]

This is not something we can fix in KIND or minikube. The chromeOS crostini Linux sandbox is too restrictive to run Kubernetes (which is essential a highly privileged tool to manage distributed Linux) for security reasons.

If you install the latest KIND into a free GCP cloud shell environment you could use it remotely, or GKE.

Or you could run another Linux distro on chrome book hardware in developer mode.

[BenTheElder]

kvm mode can work, see discussion kubernetes-sigs/kind#763 (comment)