minikube start on ubuntu VM with driver=docker reports 403 forbidden from apiserver healthz check; Apiserver is stopped

[rboal]

Steps to reproduce the issue:

1. Using ubuntu VM Ubuntu 20.04.1 LTS \n \l.
2. Installed docker version 19.03.13; it is running.
3. In docker info, have Cgroup Driver: systemd. See attached dockerInfo.txt.
   .
4. Installed and configured minikube per instructions here: https://minikube.sigs.k8s.io/docs/start/, using Debian package.
5. minikube start --driver=docker --alsologtostderr -v=1
6. minikube status reports that apiserver is Stopped
7. minikube stop
8. minikube start --driver=docker --alsologtostderr -v=1
9. minikube status still reports apiserver is Stopped.

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Stopped
kubeconfig: Configured

10. This time there are errors in the log re contacting the apiserver healthz.

Full output of failed command:

See attached file [OutputOfFailedMinikubeStartCommand.txt](https://github.com/kubernetes/minikube/files/5442253/OutputOfFailedMinikubeStartCommand.txt)

**Optional: Full output of `minikube logs` command:**

See attached file [OutputOfFailedMinikubeStartCommand.txt](https://github.com/kubernetes/minikube/files/5442262/OutputOfFailedMinikubeStartCommand.txt)

[rboal]

Sorry, I repeatedly tried to drag and drop the log files for the output of the failed command, and the output of minikube logs, but the results are as above: the links to the files (which seem to be available) are not created properly.

Please tell me what I'm doing wrong and I will try to attach them again. Thanks.

[priyawadhwa]

Hey @rboal could you try running:

minikube delete
minikube start --force-systemd --driver docker

and see if that fixes the issue?

[rboal]

Hello. I started minikube with

minikube start --force-systemd --driver docker --v=5 --alsologtostderr

and results are attached:

OutputOfFailedMinikubeStartCommand-2020-11-09.txt

[mbartosi]

I can confirm the same problem on Debian 10 / Docker 19.03.13 / Minikube 1.14.2 and driver=docker with systemd.

[medyagh]

@mbartosi do you still have this issue with latets version ?
and and does adding --force-systemd flag make a difference?

[mbartosi]

Right now I have one testing host with Minikube 1.15.1 / Kubernetes release 1.19.4 and it started OK after a long fight with startup scripts -- my testing host OS is Gentoo at the moment, not Debian :)
The problem was that kubelet wasn't starting at all because of utterly broken systemd unit, more info here: https://gist.github.com/thanatos/bc1ef8e9e60fa524cb89734a847ca0bc

[priyawadhwa]

Hey @rboal thanks for providing that output. If you're still seeing this issue, could you provide more info about the kubelet in minikube? It looks like for some reason it isn't running.

the output of these would be helpful:

minikube ssh -- sudo systemctl status kubelet
minikube ssh -- sudo  'journalctl -xeu kubelet

[priyawadhwa]

Hey @rboal friendly ping, are you still seeing this issue?

[medyagh]

Hi @bri-pug c, I haven't heard back from you, I wonder if you still have this issue?
Regrettably, there isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but please feel free to reopen whenever you feel ready to provide more information.