Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Kubernetes is starting" indefinitely on macOS Mojave #3327

Closed
3 tasks done
Miyurz opened this issue Nov 9, 2018 · 48 comments
Closed
3 tasks done

"Kubernetes is starting" indefinitely on macOS Mojave #3327

Miyurz opened this issue Nov 9, 2018 · 48 comments

Comments

@Miyurz
Copy link

Miyurz commented Nov 9, 2018

  • I have tried with the latest version of my channel (Stable or Edge)
  • I have uploaded Diagnostics
  • Diagnostics ID: 062CE476-9DB5-4E83-9512-60BFDB5E8527/20181108064441

Expected behavior

Kuberntes should be in running state once I select my context as "docker-for-desktop" from the dropdown

Actual behavior

Its hung in "Kubernetes is starting" forever.

Information

  • macOS Version: MacOS Mojave version 10.14.1

Diagnostic logs

Docker for Mac: version => Version 2.0.0.0-beta1-mac75 (27117) Channel: edge 915f68b131

Steps to reproduce the behavior

  1. Start/restart docker daemon and enable kubernetes
  2. Reinstall docker for Mac with edge channel and try and it still doesn't fix it.
@nicklonginow
Copy link

same issue for me

1 similar comment
@ai-xi
Copy link

ai-xi commented Nov 26, 2018

same issue for me

@Fierozen
Copy link

I have the same issue, almost exactly. I posted it yesterday 11/27/2018. But in addition, I can't turn Enable Kupernetes off. I can uncheck the box, but it keeps getting rechecked somehow. Not even reseting Kupernetes will make it stop.

@abdennour
Copy link

same issue for me

1 similar comment
@bitzhuxb
Copy link

same issue for me

@abdennour
Copy link

I fixed it.

Brief

Add the following lines in /etc/hosts if it is empty.

127.0.0.1 localhost
::1       localhost

Details:

  1. Display the log of docker:
pred='process matches ".*(ocker|vpnkit).*"
  || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'
/usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"
  1. Analyze the log:
    Found
2018-11-30 08:33:37.468984+0300  localhost com.docker.driver.amd64-linux[38047]: Cannot list nodes: Get https://localhost:6443/api/v1/nodes: dial tcp: lookup localhost: no such host

It means, it cannot resolve localhost ???!!

  1. Check /etc/hosts

Oops! it is empty.

  1. Remediation
# cat /etc/hosts

127.0.0.1 localhost
::1       localhost
  1. Watch the log again:
pred='process matches ".*(ocker|vpnkit).*"
  || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'
/usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"

🎉 It works. k8s is now up and running with docker-for-mac.

@riteshapatel
Copy link

riteshapatel commented Dec 3, 2018

Facing the same issue. Apparently, the fix above doesn't apply in my case. I already have entries in /etc/hosts for the localhost. Any other ideas? Kubernetes is stuck on starting....

Thx
screen shot 2018-12-03 at 8 16 49 am
screen shot 2018-12-03 at 7 54 28 am

And here are the entries from /etc/hosts.

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost

@bitzhuxb
Copy link

bitzhuxb commented Dec 4, 2018

Uninstall the docker and install the latest version fixed my issue. But older images and k8s resources have been lost.
image
image

@calmzhu
Copy link

calmzhu commented Jan 1, 2019

same issue for me

@ghost
Copy link

ghost commented Jan 8, 2019

Also have the same issue...

cat /etc/hosts

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost

@SecuraPaWn
Copy link

I have the latest version exact screenshot posted by @bitzhuxb still the same issue.
Please update what is the workaround for this ?

@micuncang
Copy link

duplicated issue with #2990

@arcseldon
Copy link

arcseldon commented Mar 17, 2019

These steps worked:

# cat /etc/hosts

127.0.0.1 localhost
::1       localhost

then:

rm -rf ~/.kube

and finally:

Hitting "Reset Kubernetes cluster" on the Reset tab (under preferences)

Not sure whether all were required...

I also took info from here

@quekshuy
Copy link

quekshuy commented Apr 5, 2019

These steps worked:

# cat /etc/hosts

127.0.0.1 localhost
::1       localhost

then:

rm -rf ~/.kube

and finally:

Hitting "Reset Kubernetes cluster" on the Reset tab (under preferences)

Not sure whether all were required...

I also took info from here

This works.

Some times I've found it's also because the Docker disk image is running low on space. I ran this and "Reset Kubernetes cluster" (as mentioned above):

# Remove all exited Docker containers (note: you might still want some of them so feel free to double check with docker ps -a)
docker rm -v $(docker ps -aq -f 'status=exited')

# Remove unused images
docker image prune

@mpdn
Copy link

mpdn commented Jun 6, 2019

Also experiencing this issue, but none of the proposed fixes here have worked.

Executing this from one of the earlier posts:

pred='process matches ".*(ocker|vpnkit).*"
  || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'
/usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"

Reveals errors like this repeating ad nauseam:

2019-06-06 10:44:43.124744+0200  localhost com.docker.vpnkit[1986]: DNS lookup docker-for-desktop.localdomain A: NoSuchRecord
2019-06-06 10:44:43.899432+0200  localhost com.docker.driver.amd64-linux[1987]: (libsystem_info.dylib) [com.apple.network.libinfo:si_destination_compare] Rule 6, prefer d2, d2 precedence 50 > d1 precedence 35
2019-06-06 10:44:43.899512+0200  localhost com.docker.driver.amd64-linux[1987]: (libsystem_info.dylib) [com.apple.network.libinfo:si_destination_compare] <private>@0 < <private>@0
2019-06-06 10:44:43.901892+0200  localhost com.docker.driver.amd64-linux[1987]: Cannot list nodes: Get https://localhost:6443/api/v1/nodes: EOF

So it looks like some kind of DNS problem, maybe? My /etc/hosts file should be OK:

127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost

@davecombs
Copy link

I'm in the same boat as noctune - nothing worked so far, and I have the same log output about a missing DNS record and 'Cannot list nodes'.

@moochalini
Copy link

moochalini commented Jun 19, 2019

Try to update the hosts file so that it resolves docker-for-desktop to localhost, try adding it like so:

127.0.0.1 localhost docker-for-desktop
255.255.255.255 broadcasthost
::1 localhost

@cbron
Copy link

cbron commented Jun 26, 2019

Fix didn't work for me either, but switching to edge did (version 2.0.5.0)

@davecombs
Copy link

Same experience for me - nothing worked until I switched to edge.

@vietchung
Copy link

rm -rf ~/.kube
then reset Kubernetes cluster
Waiting 2-5 minutes

@wickerx
Copy link

wickerx commented Jul 17, 2019

How I solved my Issue:

0. Open terminal and follow up the logs of Docker for Desktop:
(by tiping following commands):

$ pred='process matches ".*(ocker|vpnkit).*" || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'

$ /usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"

1. Uninstalled Kubernetes from Docker Preferences Panel (if present)
2. Remove .kube file (rm -rf ~/.kube) - if present
3. Edit /etc/hosts. (MUST contain docker-for-desktop name on first localhost line)

  • It will look like this (after edit):

127.0.0.1 localhost docker-for-desktop
255.255.255.255 broadcasthost
::1 localhost

4. Reinstall Kubernetes from Docker Preferences Panel

5. Restart Docker Desktop
(It takes about 1 minute 30 seconds to startup)

  • After some time (and a lot of informations), tailing will show following messages:

2019-07-17 20:23:55.562719-0300 localhost com.docker.driver.amd64-linux[69080]: Kubernetes is up and running

2019-07-17 20:23:55.563460-0300 localhost Docker[68814]: (CoreFoundation) [com.apple.CFBundle:strings] Bundle: , key: kubeStateRunning, value: , table: Localizable, localizationName: (null), result: Kubernetes is running

2019-07-17 20:23:55.565458-0300 localhost Docker[68814]: (CoreFoundation) [com.apple.CFBundle:strings] Bundle: , key: kubeStateRunning, value: , table: Localizable, localizationName: (null), result: Kubernetes is running

6. After services running (Docker and Kubernetes), try access the API:

https://localhost:6443/api/v1

@daisuke-awaji
Copy link

daisuke-awaji commented Jul 22, 2019

It is possible that you simply do not have access to the file. I used to spend an hour in the face of this problem.

image

image

~$ sudo chmod 444 /etc/hosts
~$ ll /etc/hosts
-rwxr--r--  1 root  wheel  258  7 22 13:36 /etc/hosts

Then k8s will start.

@altfatterz
Copy link

for me this worked:

Reset to factory settings in the Docker for Mac

then Enable Kubernetes again

It will take couple of minutes:
Screenshot 2019-07-30 at 17 59 49

No need to add docker-for-desktop into your /etc/hosts

@drenze
Copy link

drenze commented Aug 30, 2019

For me, this was not Mac-specific. It was impacting me on a Windows laptop as well, following the most recent upgrade.

Based on the fact that it resolved itself after working outside the corporate firewall and disconnected from the VPN for a few days, I believe that in my case, it was a firewall issue. Our corporate firewall apparently wasn't allowing Docker to pull the Kubernetes images (I discovered this by accident).

After returning to the office, I was able to test the following workaround, which worked for me:

  1. Disconnect from the corporate network.
  2. Connect to an outside network (i.e., my home network) and disable VPN.
  3. Enable Kubernetes or, if already enabled, restart Docker for Desktop.
  4. Wait for k8s to start.

Once Kubernetes was running, it continued to run, even when reconnected to the corporate network again. I don't know whether I'll have to repeat this after my next update or not.

@nmarus
Copy link

nmarus commented Nov 11, 2019

I had a similar problem above. My issue is that I am using a app to manage my hosts file called gasmask. It prevents external applications from modifying the hosts file. Once disabled, I reset docker, and re-eneabled kubernetes. I notices this now shows in my hosts file and everything started fine. Hope this helps someone else.

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

@prachigup
Copy link

I had a similar issue. I ran diagnostics (available in the troubleshoot menu). From there in the log file, I can see the error : "Error while setting up kubernetes: cannot update the host kube config: open /Users//.kube/config.lock: permission denied".
I fixed this by giving permissions to the .kube folder using command: sudo chmod -R 777 .kube/

@kozchris
Copy link

kozchris commented Mar 8, 2020

I had issue with 2.2.0.3 (42716) and OSX Catalina. Turns out that somehow my /etc/hosts was corrupt with a bunch of garbage unprintable characters just before the section:

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

Deleting the garbage fixed the issue for me.

@vikas027
Copy link

Hitting "Reset Kubernetes cluster" on the Reset tab (under preferences) worked for me on Catalina

@soumendrak
Copy link

soumendrak commented May 19, 2020

Am I the only one for whom none of the above methods worked? :'(

@gstruczynski
Copy link

As per #3649 (comment) , deleting the pki folder fixed this issue

rm -rf ~/Library/Group\ Containers/group.com.docker/pki/

@soumendrak
Copy link

soumendrak commented Jun 1, 2020

Finally the following worked:

  1. Uninstalling the existing Docker Desktop from Mac Catalina.
  2. Removing .docker and .kube folders.
  3. Installing the Docker Desktop 2.0.0.0-mac81 (29211)

@interactwithankush
Copy link

This thread worked like a charm 👍

@2015211289
Copy link

there is an easy way to fix it and it really helps.
1,git clone https://github.com/maguowei/k8s-docker-for-mac.git
2,cd k8s-docker-for-mac/
3, ./load_images.sh when docker desktop is open
4,enter preference, in configure file add https://registry.docker-cn.com to registry-mirrors.
Screen Shot 2020-06-06 at 4 59 57 PM
5. restart Docker Desktop.
If you can't modify preference in step 4, try to enter Troubleshoot and click 'reset Kubernetes cluster'.

@changtraisitinh
Copy link

for me, uninstall docker in install last version

@schemacs
Copy link

Update kubernetesEnabled to false in ~/Library/Group Containers/group.com.docker/settings.json.

@suraneti
Copy link

suraneti commented Sep 3, 2020

Update kubernetesEnabled to false in ~/Library/Group Containers/group.com.docker/settings.json.

This solution worked for me after I've changed kubernetesEnabled to false then start Kubernetes again.

Environment:
macOS Catalina 10.15.6 (19G73)
Docker Desktop Community 2.3.0.3 (45519)

@tlaak
Copy link

tlaak commented Sep 19, 2020

Uninstalling Docker and installing the latest version was the only trick that worked for me.

@anthonyxpalermo
Copy link

Can get it to work.
Error while setting up kubernetes: cannot update the host kube config: Failed to load Kubernetes CA: couldn't load ca certificate authority from /Users/myuser/Library/Group Containers/group.com.docker/pki

@sasi040
Copy link

sasi040 commented Dec 8, 2020

Uninstall the docker and install the latest version fixed my issue. But older images and k8s resources have been lost.
image
image

this worked for me. Thanks.

@NevoEldad
Copy link

for me this worked:

Reset to factory settings in the Docker for Mac

then Enable Kubernetes again

It will take couple of minutes:
Screenshot 2019-07-30 at 17 59 49

No need to add docker-for-desktop into your /etc/hosts

tried all of the above, this one worked for me.

@gwpantazes
Copy link

gwpantazes commented Jan 18, 2021

I had this issue and after trying a few advices in this thread and a few of my own fiddlings, I fixed it. I'll list everything I tried in case this helps people browsing through google, or in case my extra tweaks were the fix.

  1. Tried rm -rf ~/Library/Group\ Containers/group.com.docker/pki/ as per gstruczynski's comment. Restarted my machine, but k8s didn't appear to start or was taking a while and I got impatient to try the next thing.
  2. Updated /etc/hosts/ as per abdennour's comment. I also found an extra mapping to 127.0.0.1 auto-generated by Docker Desktop (perhaps an earlier version?) and deleted that. Restarted my machine, but k8s didn't appear to start or was taking a while and I got impatient to try the next thing.
  3. My ~/.docker/config.json had a context for the docker desktop kubernetes. I switched to that context (my context was on another k8s cluster for work). Once I had switched to that context, I noticed that kubernetes started. I'm not sure if it was because of the context matching up or if it was coincidence, but I wanted to note it here in case that helps anyone.

If I tried anything else, I can't remember it. So whether the fix was the first, second or third thing, I'm not sure, but if anybody is still having issues after deleting the pki folder and double-checking hosts, try setting the context for docker desktop kubernetes and see if that helps.

@dand-or
Copy link

dand-or commented Jan 20, 2021

I was able to solve it by adding the following to hosts
127.0.0.1 kubernetes.docker.internal

When I checked the log along here , it seems that there was a problem with the above name resolution.
$ / usr / bin / log stream --style syslog --level = debug --color = always --predicate’process matches “. * (ocker | vpnkit). *” || (process in {“taskgated-helper” , “launchservicesd”, “kernel”} && eventMessage contains [c] “docker”)'

My version
Kubernetes
v1.19.3

@adancaym
Copy link

adancaym commented Mar 7, 2021

I fixed it.

Brief

Add the following lines in /etc/hosts if it is empty.

127.0.0.1 localhost
::1       localhost

Details:

  1. Display the log of docker:
pred='process matches ".*(ocker|vpnkit).*"
  || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'
/usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"
  1. Analyze the log:
    Found
2018-11-30 08:33:37.468984+0300  localhost com.docker.driver.amd64-linux[38047]: Cannot list nodes: Get https://localhost:6443/api/v1/nodes: dial tcp: lookup localhost: no such host

It means, it cannot resolve localhost ???!!

  1. Check /etc/hosts

Oops! it is empty.

  1. Remediation
# cat /etc/hosts

127.0.0.1 localhost
::1       localhost
  1. Watch the log again:
pred='process matches ".*(ocker|vpnkit).*"
  || (process in {"taskgated-helper", "launchservicesd", "kernel"} && eventMessage contains[c] "docker")'
/usr/bin/log stream --style syslog --level=debug --color=always --predicate "$pred"

🎉 It works. k8s is now up and running with docker-for-mac.

This works for me too

@lavanyapallapa
Copy link

lavanyapallapa commented Mar 23, 2021

Please check the logs in below location.
/Users/username/Library/Containers/com.docker.docker/Data/log/ host/com.docker.driver.amd64-linux.log

see the error and try to resolve the issue.

In my case it said
time="2021-03-24T00:12:05+05:30" level=info msg="cannot get lease for master node: Get \"https://kubernetes.docker.internal:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/docker-desktop\": dial tcp: lookup kubernetes.docker.internal: no such host"

so updated /etc/hosts file with "127.0.0.1 kubernetes.docker.internal"

then it gave other error
time="2021-03-24T00:12:24+05:30" level=info msg="cannot get lease for master node: leases.coordination.k8s.io \"docker-desktop\" not found"

again , updated hosts file with "127.0.0.1 docker-desktop"

So, now my kubernetes on docker desktop :-)

@GwamakaCharles
Copy link

GwamakaCharles commented Apr 7, 2021

I fixed it!
I deleted the config file i.e ~/.kube/config then I uninstalled the app through the Troubleshoot window. I reinstalled and enabled kubernetes which required the internet to download the necessary files and after a while kubernetes was running. I believe its due to the internet connection during the installation of kubernetes files that causes problems.
Uploading Screen Shot 2021-04-07 at 13.57.02.png…

@anjanesh
Copy link

I updated Docker via the update Docker notification on my macOS mojave - but Starting kubernetes is still stuck at starting.

@docker-robott
Copy link
Collaborator

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@docker-robott
Copy link
Collaborator

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

@docker docker locked and limited conversation to collaborators Sep 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests