docker desktop installs kubectl and overwrites existing even though kubernetes integrations are disabled

[zswanson]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

Installing docker desktop for mac, either through the official dmg image or homebrew install docker should install docker desktop, with kubernetes integrations disabled initially (which is the current default behavior) and not install kubectl

Actual behavior

The docker desktop installer installs its own kubectl and creates 2 soft link targets in /usr/local/bin for kubectl and kubectl.docker. Kubernetes integrations in docker desktop are still disabled when inspected immediately after installation. Due to #6327 this version of kubectl doesn't work with corporate vpn and we have to manually remove the soft links. In some cases after either a) a docker desktop update; b) restarting docker desktop, I have also observed that Docker Desktop will re-create the softlinks, leading to the same problem.

Information

• macOS Version:
• Intel chip or Apple chip:
• Docker Desktop Version:

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Note: I have already removed the kubectl softlinks due to the issue mentioned in #6327

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0017: can a VM be started?
[FAIL] DD0015: are the binary symlinks installed? looking for /usr/local/bin/kubectl: lstat /usr/local/bin/kubectl: no such file or directory
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0007: is the backend responding?
[PASS] DD0014: are the backend processes running?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[FAIL] DD0012: is the VM networking working? network checks failed: failed to ping host: exit status 1
[2022-05-27T18:14:11.697727000Z][com.docker.diagnose][I] ipc.NewClient: d1f4489b-diagnose-network -> <HOME>/Library/Containers/com.docker.docker/Data/diagnosticd.sock diagnosticsd
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[	common/pkg/diagkit/gather/diagnose/network.go:34 +0xd4
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x102ec80e0)
[	common/pkg/diagkit/gather/diagnose/test.go:46 +0x44
[common/pkg/diagkit/gather/diagnose.Run.func1(0x102ec80e0)
[	common/pkg/diagkit/gather/diagnose/run.go:17 +0x44
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0x102ec80e0)
[	common/pkg/diagkit/gather/diagnose/run.go:140 +0x84
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x102ec80e0, 0x140007c1728)
[	common/pkg/diagkit/gather/diagnose/run.go:146 +0x3c
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0x20?, 0x140007c1728)
[	common/pkg/diagkit/gather/diagnose/run.go:149 +0x78
[common/pkg/diagkit/gather/diagnose.walkOnce(0x1029ac0a0?, 0x1400036f890)
[	common/pkg/diagkit/gather/diagnose/run.go:135 +0x8c
[common/pkg/diagkit/gather/diagnose.Run(0x102ec8260, 0x10227bfa0?, {0x1400036fb08, 0x1, 0x1})
[	common/pkg/diagkit/gather/diagnose/run.go:16 +0x178
[main.checkCmd({0x140001b6010?, 0x6?, 0x4?}, {0x0, 0x0})
[	common/cmd/com.docker.diagnose/main.go:132 +0xe0
[main.main()
[	common/cmd/com.docker.diagnose/main.go:98 +0x308
[2022-05-27T18:14:11.697828000Z][com.docker.diagnose][I] (5f03f93e) d1f4489b-diagnose-network C->S diagnosticsd POST /check-network-connectivity: {"ips":["192.168.86.23","10.188.128.51"]}
[2022-05-27T18:14:12.254140000Z][com.docker.diagnose][W] (5f03f93e) d1f4489b-diagnose-network C<-S 93d63d94-diagnosticsd POST /check-network-connectivity (556.286041ms): failed to ping host: exit status 1

[PASS] DD0032: do Docker networks overlap with host IPs?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?

Please investigate the following 2 issues:

1 : The test: is the VM networking working?
    Failed with: network checks failed: failed to ping host: exit status 1

VM seems to have a network connectivity issue. Please check your host firewall and anti-virus settings in case they are blocking the VM.

2 : The test: are the binary symlinks installed?
    Failed with: looking for /usr/local/bin/kubectl: lstat /usr/local/bin/kubectl: no such file or directory

The symlinks to the docker CLI etc are needed for docker commands to work.

Steps to reproduce the behavior

1. Install kubectl with homebrew install kubernetes-cli
2. Install docker desktop
3. Inspect that soft links to kubectl for docker now exist in /usr/local/bin, and that $(which kubectl) points to that instead of the homebew installed client

[laurazard]

Hi @zswanson, thanks for the report! I've created an internal ticket so we can address this.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[zswanson]

/remove-lifecycle stale

[zswanson]

@laurazard any update on whether this will be addressed? We deal with a lot of confusion with this on a regular basis from our developer teams. Or if #6327 was resolved it would be less of an issue.

[juliettepaul]

We are running into this as well. My asdf shim for kubectl keeps getting overwritten giving me version discrepancies when I accidentally use docker's installed version

[nicks]

My understanding of the current state of things:

• If you don't have kubectl installed in /usr/bin/local, Docker will install it for you on startup. I don't think there are any plans to change this.
• We've heard reports of Docker Desktop overwriting an existing kubectl shim, but no one who works at Docker has been able to reproduce this problem. I tried with both brew install kubernetes-cli and asdf, and confirmed that DD preserved the existing symlink. If you can repro this problem, detailed repro steps and/or a diagnostic ID would help. (https://docs.docker.com/desktop/troubleshoot/overview/#diagnose)

[juliettepaul]

Thanks for the response! My shim was actually not in /usr/local/bin, but I am now linking it there and and that seems to help with startup. The error came because I was relying on it later in the path, so I can either move my local bin folder earlier in the path or link it to /usr/local/bin. I will make sure it doesn't overwrite it on version upgrade/re-install as well.

[zswanson]

Its been noted that my original description of 'overwrite' is inaccurate, as the homebrew installed kubectl doesn't exist in /usr/bin/local/, but rather that the Docker installed binary is jumping precedence.

[nicks]

Thank you both for confirming!

I'm going to mark this as working as intended. I believe most tool managers (like asdf) should set their PATH precedence of their shims higher than /usr/local/bin. You can also set PATH to make sure your kubectl binary gets loaded first.

Happy to re-open if DD is overwriting preferences in some way that's not intended.

[draco2003]

@nicks could we ask for a solution that handles the existence of kubectl outside of /usr/bin/local better? something like:
lstat "$(which kubectl)"

This would not require the Docker installation of kubectl if it is in the PATH, but outside of /usr/bin/local?

[juliettepaul]

If I may offer an adjustment. If you just place your kubectl path before /usr/local/bin in your PATH even if docker writes kubectl there, it won't get used

[draco2003]

@jayson definitely easy enough for myself or a one off individual that runs into this problem and then discovers it to reorder their PATH.
From a user support perspective it would ideally work as expected by typical usage patterns.

I'd expect the bullet above from @nicks to be more like:
"If you don't have kubectl installed, Docker will install it for you on startup."...
Then docker does something like:
lstat "$(which kubectl)" in order to check existence of kubectl and/or usage of it.

This allows them to still install it in /usr/local/bin if it is required, but if it's available on the path then no need to do so.
Win / Win :)

[nicks]

@draco2003 sure, I can ask about that.

[nicks]

OK, had a quick chat with the team. We're making a bunch of changes to how Docker Desktop installs to require fewer root permissions, so as part of this effort, we're going to change it so that it only installs kubectl if it's not already on PATH. stay tuned!

[nicks]

The team changed the logic in Docker Desktop 4.14 so that it checks for the existence of kubectl on PATH, rather than checking /usr/local/bin. Thanks for the suggestion!

[zswanson]

@nicks we just had an engineer upgrade docker desktop to 4.14.1 (91661) and the docker kubectl softlink was still created on OSX.

#Before docker starts#
$ ls -al /usr/local/bin/kubectl
lrwxr-xr-x  1 mclaussen  admin  51 Dec  1 11:04 /usr/local/bin/kubectl -> /usr/local/Cellar/kubernetes-cli/1.25.4/bin/kubectl
#After docker restarts#
$ ls -al /usr/local/bin/kubectl
lrwxr-xr-x  1 root  admin  55 Dec  1 11:05 /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl

So in this case not only did it not detect that kubectl was on the PATH already, it also overwrote an existing softlink.

I actually don't see anything in the release notes for 4.14.x for this

[nicks]

Hmmm...i wonder if what's happening here is that /usr/local/bin is on the "normal" user's PATH, but not on the root user's PATH. does that sound plausible?

[zswanson]

That sounds likely.
Though also definition of the PATH is going to vary by shell depending on what the docker installer is using. ie I'm on zsh but I know many folks are still on bash.

[MrEyratnz]

@nicks we just had an engineer upgrade docker desktop to 4.14.1 (91661) and the docker kubectl softlink was still created on OSX.

#Before docker starts#
$ ls -al /usr/local/bin/kubectl
lrwxr-xr-x  1 mclaussen  admin  51 Dec  1 11:04 /usr/local/bin/kubectl -> /usr/local/Cellar/kubernetes-cli/1.25.4/bin/kubectl
#After docker restarts#
$ ls -al /usr/local/bin/kubectl
lrwxr-xr-x  1 root  admin  55 Dec  1 11:05 /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl

So in this case not only did it not detect that kubectl was on the PATH already, it also overwrote an existing softlink.

I actually don't see anything in the release notes for 4.14.x for this

I'm experiencing the same in 4.15.0 as well

[djpadz]

Same here on 4.15.0. Before DD start:

(0) djs-16-mbp ~$ ls -l /usr/local/bin/kubectl
lrwxr-xr-x  1 djpadz  admin  43 Dec 16 09:24 /usr/local/bin/kubectl@ -> ../Cellar/kubernetes-cli/1.26.0/bin/kubectl

After:

(0) djs-16-mbp ~$ ls -l /usr/local/bin/kubectl
lrwxr-xr-x  1 djpadz  admin  55 Dec 16 09:31 /usr/local/bin/kubectl@ -> /Applications/Docker.app/Contents/Resources/bin/kubectl

It's easy enough to put a symlink to brew's kubectl in ~/bin and keep that at the front of $PATH, but it's still kind of annoying.

[yuzhouliu9]

Same on Docker desktop upgrade to 4.15.0.
You can reproduce by installing kubectl via these instructions to /usr/local/bin/kubectl. After upgrade to DD 4.15.0, you'll see docker override with a symlink.

> ll /usr/local/bin/kubectl
lrwxr-xr-x  1 yuzhou.liu  admin  55 16 Dec 18:15 /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl

Extremely annoying.

[matthewjmiller1]

Still seeing on 4.16.2.

Before DD upgrade:

$ ls -l /usr/local/bin/kubectl
lrwxr-xr-x 1 millers admin 43 Jan 22 12:26 /usr/local/bin/kubectl -> ../Cellar/kubernetes-cli/1.26.1/bin/kubectl
$

After DD upgrade:

$ ls -l /usr/local/bin/kubectl
lrwxr-xr-x 1 millers admin 55 Jan 22 14:14 /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl
$

[byrongrogan]

@nicks This is still occurring in the same way on current versions. Docker Desktop taking over kubectl locally doesn't seem like reasonable behavior, given that many users need to have specific different versions locally for kubernetes version compatibility.

There doesn't seem to be any way to disable this behavior. Can you provide a workaround for this or remove the unnecessary kubectl install upon upgrades unless Kubernetes integrations are enabled?

[nicks]

@byrongrogan did you see the workaround in this comment? #6328 (comment)

My understanding is that if you have kubectl on PATH, docker won't replace it. (And I haven't seen any repro steps that suggest otherwise.) So the workaround is to make sure you have it somewhere on PATH? That the DD binary can see?

[zswanson]

@nicks that doesn't work, probably because 1) the shell you're using may differ from whatever the docker installer is using 2) it may not be running as our user and gaining our shell envs.

I'm personally of the opinion that the docker installer should ask before installing something like kubectl.

[byrongrogan]

@nicks Homebrew installs kubectl to /usr/local/bin/kubectl on Intel Macs. I can't confirm what the PATH may be in the installer's environment, but I'm consistently seeing the installer clobber this.

Instead of relying on the PATH, which could be different between the installer's env and the user's normal shell, perhaps check to see if the file already exists before overwriting it?

[zswanson]

All of these approaches have problems because the order of the PATH precedence matters and anywhere that Docker installs to might clobber or shadow the users already-existing kubectl install. The only real preventative here is to ask during the install if the user wants docker to install and manage kubectl. (and remember that choice when its performing upgrades)

[nicks]

DD 4.17 will change the logic so that it will check if:

• kubectl is at /usr/local/bin
• kubectl is on PATH where the symlink handler runs (see footnote 1)

and if the answer to both is "no", then DD will not install the symlink.

re: "The only real preventative here is to ask during the install if the user wants docker to install and manage kubectl."

I'm sympathetic to the idea that DD should not install kubectl at all. But given that it does, there's a whole ecosystem of extensions + downstream dependencies that assumes this guarantee.

Adding a dialog box during the install doesn't fix the problem, because most of those millions of users won't know if the tools they're using depend on it being installed. reminds me of this old gem: https://limi.net/checkboxes

footnote 1) this is a bit nuanced, see https://docs.docker.com/desktop/mac/permission-requirements/. Over time, we've been actively shifting which parts run as root vs user. There are broader ecosystem and backwards-compatibility concerns that come up here too.

[chaizeg]

Closing this issue because a fix has been released in Docker Desktop 4.17.0 . See the release notes for more details.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

/lifecycle locked