Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperkit process on MacOS continuously using approx 50% CPU #2601

Open
evoncken opened this issue Feb 13, 2018 · 52 comments

Comments

Projects
None yet
@evoncken
Copy link

commented Feb 13, 2018

Expected behavior

With no user workload / containers running, expect CPU usage to be very low.

Actual behavior

"hyperkit" process using aprox. 50% CPU all the time

Information

Diagnostics ID: 1582F6B0-78F2-4F39-9BDC-67E37D52D34B

Steps to reproduce the behavior

  1. Install Docker for Mac (Edge, with Kubernetes enabled)
  2. Check hyperkit CPU usage in Activity Monitor
@w4-jungyeol

This comment has been minimized.

Copy link

commented Mar 27, 2018

I've also seen the memory usage at the top. About 1.85 GB. Not sure if it is normal.

@richardthombs

This comment has been minimized.

Copy link

commented Apr 4, 2018

Still happening on Docker version 18.03.0-ce, build 0520e24

@vinayan3

This comment has been minimized.

Copy link

commented Apr 26, 2018

Any updates on this? It's a little bit of a blocker for me to keep this on all the time. It drains my laptop battery when unplugged.

@anacrolix

This comment has been minimized.

Copy link

commented Apr 30, 2018

I think it went away when i shut down a container I forgot about.

@atombender

This comment has been minimized.

Copy link

commented May 28, 2018

This urgently needs to be fixed. It's important to note that while there are many issues about Docker for Mac consuming an insane amount of CPU, those reports seem to be related to actual workloads. But this issues when there are no containers running (other than Kubernetes itself). A simple reset and restart will trigger the problem.

@atombender

This comment has been minimized.

Copy link

commented May 28, 2018

Looking at the actual VM, this seems like a typical distribution of CPU usage among the Kubernetes processes:

18774 18757 root     S     153m   4%   0   7% kube-controller-manager --kubeconf
 1130  1015 root     S     787m  20%   0   6% /usr/local/bin/dockerd -H unix:///
 1016   906 root     S     247m   6%   0   4% kubelet --kubeconfig=/etc/kubernet
11142 11124 root     S     721m  18%   0   3% kube-apiserver --advertise-address
14329 14312 root     S    53468   1%   0   1% kube-scheduler --address=127.0.0.1

First, looking at the logs, something is continuously calling Docker's API with requests such as /v1.31/containers/json. I'm going to assume this is the Kubelet. But the Kubelet doesn't log any activity corresponding to the frequency of requests, and neither does kube-controller-manager. apiserver does log a little bit:

2018-05-28T19:03:15.431464400Z I0528 19:03:15.431266       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.compose.docker.com
2018-05-28T19:03:16.387777600Z E0528 19:03:16.387645       1 controller.go:111] loading OpenAPI spec for "v1beta1.compose.docker.com" failed with: OpenAPI spec does not exists
2018-05-28T19:03:16.388078600Z I0528 19:03:16.387994       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.compose.docker.com: Rate Limited Requeue.
2018-05-28T19:03:17.368711500Z I0528 19:03:17.368437       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta2.compose.docker.com
2018-05-28T19:03:17.371758900Z E0528 19:03:17.371636       1 controller.go:111] loading OpenAPI spec for "v1beta2.compose.docker.com" failed with: OpenAPI spec does not exists
2018-05-28T19:03:17.372081200Z I0528 19:03:17.372001       1 controller.go:119] OpenAPI AggregationController: action for item v1beta2.compose.docker.com: Rate Limited Requeue.
2018-05-28T19:04:16.316600800Z I0528 19:04:16.316422       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.compose.docker.com
2018-05-28T19:04:16.321664900Z E0528 19:04:16.321553       1 controller.go:111] loading OpenAPI spec for "v1beta1.compose.docker.com" failed with: OpenAPI spec does not exists
2018-05-28T19:04:16.321878200Z I0528 19:04:16.321822       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.compose.docker.com: Rate Limited Requeue.
2018-05-28T19:04:17.304634900Z I0528 19:04:17.301377       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta2.compose.docker.com
2018-05-28T19:04:17.311260900Z E0528 19:04:17.306398       1 controller.go:111] loading OpenAPI spec for "v1beta2.compose.docker.com" failed with: OpenAPI spec does not exists
2018-05-28T19:04:17.311302000Z I0528 19:04:17.306541       1 controller.go:119] OpenAPI AggregationController: action for item v1beta2.compose.docker.com: Rate Limited Requeue.

It doesn't seem enough to indicate high CPU usage, but the note about rate-limiting is interesting. strace on the API server also shows a continuous stream of activity, though it appears to be binary (gRPC and/or TLS-encrypted), and so I don't know what it is. I do see that the API server has 91 TCP connections open, which seems excessive.

@tristanpemble

This comment has been minimized.

Copy link

commented May 29, 2018

we are aware of this issue and track it internally. The amount of CPU taken varies from system to system and is only present when Kube is enabled. With Kube enabled a couple of fairly chunky containers are running and it is yet unclear if it is these containers taking up the CPU or interactions with HyperKit

from this thread

would like some public tracking of this issue

@jsternberg

This comment has been minimized.

Copy link

commented Jul 12, 2018

I can confirm that this is happening for me too and most of the developers have complained about kubernetes using docker for desktop on their macs causes their computers to get very loud. It happens most frequently when deploying and then begins to quiet down, but it's been over 10 minutes since I used the kubernetes api server, all of the deployed stuff is running without any restarts, and I'm still getting a large amount of cpu used. I get these log lines once a minute from the api server:

I0712 14:28:57.714579       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.compose.docker.com
E0712 14:28:57.721750       1 controller.go:111] loading OpenAPI spec for "v1beta1.compose.docker.com" failed with: OpenAPI spec does not exists
I0712 14:28:57.721794       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.compose.docker.com: Rate Limited Requeue.
I0712 14:28:58.679366       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta2.compose.docker.com
E0712 14:28:58.683655       1 controller.go:111] loading OpenAPI spec for "v1beta2.compose.docker.com" failed with: OpenAPI spec does not exists
I0712 14:28:58.683702       1 controller.go:119] OpenAPI AggregationController: action for item v1beta2.compose.docker.com: Rate Limited Requeue.
@IngaFeick

This comment has been minimized.

Copy link

commented Jul 24, 2018

Same happens to me and pretty much everyone I know who's using Docker for Mac. Strangely it seems to happen mostly when no container is running.

@ypresto

This comment has been minimized.

Copy link

commented Jul 26, 2018

htop in privileged container when Kubernetes is enabled:
image

disabled:
image

@mitchellnemitz

This comment has been minimized.

Copy link

commented Aug 27, 2018

Any movement on this issue? A fresh install of Docker for Mac on a fresh install of OSX this morning sees CPU usage hovering at 35-50% and Activity Monitor's "Energy Impact" rating for Docker sitting around ~70, which is absurdly high.

Is there anything we users can do to help push this one forward? More data collection maybe, collecting logs or providing profiling output? If there's already a fix that needs tested, is there a way for us to chip in and verify it's working with our setups?

@dwaite

This comment has been minimized.

Copy link

commented Aug 28, 2018

On Mojave (18A377a), I am seeing large amounts of CPU usage (and fan spin) from hyperkit on a fresh install of both Docker stable and experimental (18.06.0-ce-mac69 26398). I did nothing outside of a fresh install, and have not enabled Kubernetes.

@jamilbk

This comment has been minimized.

Copy link

commented Sep 11, 2018

Also experiencing this issue making it annoying to do any sort of Kubernetes work on my Macbook Pro. Battery runtime is cut approximately in half from ~4 hours to 2 hours with Kubernetes running in Docker for Mac.

10.13.6
15" MBP mid 2015

@jnicholls

This comment has been minimized.

Copy link

commented Oct 10, 2018

I do hope this is going to be addressed in the near future, even if it requires an upstream patch to one of the k8s components. I see the CPU usage is primarily the kube-controller-manager which spins at about 10% of a core, followed by kube-apiserver at about 7% of a core. etcd and kube-scheduler take another 3% each or so.

HyperKit itself is spinning at 50-60%, so I don't know where the other compute is coming from but it definitely settles down after Kubernetes is disabled.

OSX 10.13.6
Docker 18.09.0-ce-beta1

@if6was9

This comment has been minimized.

Copy link

commented Oct 12, 2018

With a bare kube install, I see ~50% CPU utilization from HyperKit. Inside Hyperkit, the utilization doesn't look that high, but etcd is doing something odd.

2018-10-12_10-04-34

When I shut down Kube, everything goes back to normal.

2018-10-12_10-05-59

I'm running Mojave and Docker Version 18.06.1-ce-mac73 (26764)

@dhs227

This comment has been minimized.

Copy link

commented Oct 12, 2018

The same issue reproduces on my mac:
Mojave
docker-machine version 0.15.0, build b48dc28
hyperkit: 0.20180403
Docker: not installed

@torbjornvatn

This comment has been minimized.

Copy link

commented Oct 24, 2018

Same here:
Mojave
docker-for-mac: 2.0.0.0-beta1-mac75 (27117)
Engine: 18.09.0-ce-beta1
Kubernetes: v1.10.3

Any hope of this getting fixed any time soon?

@teosoft123

This comment has been minimized.

Copy link

commented Oct 28, 2018

Same here.
Mojave
Engine/Client: 18.06.1-ce-mac73 (26764)

@ekhaydarov

This comment has been minimized.

Copy link

commented Oct 31, 2018

im on high sierra

some update somewhere is causing this issue.

please give love to docker for mac, will give love back

@erulabs

This comment has been minimized.

Copy link

commented Oct 31, 2018

Same issue:
osx: High Sierra
docker-for-mac: 2.0.0.0-beta1-mac75 (27117)
kubernetes: v1.10.3

Looks to me like there is some hypervisor related issue - load inside the VM is minimal (according to htop), but from the host (osx) it's eating basically as much CPU as I give it (more cores === more load)

Running dtruss on the hyperkit process shows that it's making and failing a ton of psynch_cvwaits:

psynch_cvsignal(0x1048AF180, 0x552FC100552FC200, 0x552FC100)		 = 257 0
psynch_cvwait(0x1048AF180, 0x552FC101552FC200, 0x552FC100)		 = 0 0
psynch_mutexdrop(0x1048AF130, 0xD7D6EF03, 0xD7D6EE00)		 = 0 0
psynch_mutexwait(0x1048AF130, 0xD7D6EF03, 0xD7D6ED00)		 = -673779965 0
psynch_cvwait(0x1048AF180, 0x552FC201552FC300, 0x552FC200)		 = -1 Err#316
psynch_cvwait(0x1048AF180, 0x552FC301552FC400, 0x552FC200)		 = -1 Err#316
psynch_cvsignal(0x7FA570050410, 0x16B0D40016B0D500, 0x16B0D400)		 = 257 0
psynch_cvwait(0x7FA570050410, 0x16B0D40116B0D500, 0x16B0D400)		 = 0 0
psynch_cvsignal(0x1048AF180, 0x552FC400552FC500, 0x552FC200)		 = 257 0
psynch_cvwait(0x1048AF180, 0x552FC401552FC500, 0x552FC200)		 = 0 0
psynch_cvwait(0x1048AF180, 0x552FC501552FC600, 0x552FC500)		 = -1 Err#316

will continue looking into this and will post updates here. FWIW this issue appears to occur with no pods, no deployments (totally stock kubernetes after enabling in Docker). If it helps anyone's investigation, this also occurs with Kubernetes 1.10 on Minikube via Virtualbox (so my guess is that's not related to hyperkit per say, but rather kubernetes 1.10)

@mattstein

This comment has been minimized.

Copy link

commented Oct 31, 2018

FWIW I'm having similar issues and do not have Kubernetes enabled. macOS 10.14, Docker for mac 18.06.1-ce-mac73 (26764).

Using .raw storage (and :cached), limiting CPUs, and paring down mounted folders are the only things that've helped. I'm only running very light (most often idle) DDEV containers and still seeing com.docker.hyperkit at ~65-130% CPU.

@erulabs

This comment has been minimized.

Copy link

commented Oct 31, 2018

@mattstein got me curious, so I wiped my docker install entirely and re-installed:

base install, never been touched, no kubernetes: 2.5% of CPU (Acceptable!)
click "Enable Kubernetes" -> "apply": 35% of CPU (Not very acceptable 😢)

FWIW, Hyperkit version is v0.20180403-17-g3e954c, but im more and more suspecting kubernetes itself, not osx or hyperkit

@mattstein

This comment has been minimized.

Copy link

commented Oct 31, 2018

@erulabs I've never had kubernetes enabled and yet share the symptoms many are experiencing here, so at least in my case it's definitely osx or hyperkit. I don't know Docker well enough to chase it down any further, but I suspect it has something to do with how disks are mounted and maybe polled for changes (if that's even relevant). The problem seems to be significantly worse when I have front-end build tools (gulp, webpack) watching for file changes in locations that are mounted.

@amasson88

This comment has been minimized.

Copy link

commented Oct 31, 2018

same for me:

dtrace: system integrity protection is on, some features will not be available

SYSCALL(args) 		 = return
dtrace: 10310 dynamic variable drops with non-empty dirty list
psynch_cvsignal(0x7FBA48000600, 0xDEBF2F00DEBF3000, 0xDEBF2F00)		 = 257 0
psynch_cvsignal(0x102AC5180, 0xC28FCA00C28FCB00, 0xC28FC900)		 = 257 0
psynch_cvwait(0x102AC5180, 0xC28FCA01C28FCB00, 0xC28FC900)		 = 0 0
psynch_cvwait(0x102AC5180, 0xC28FCA00C28FCC00, 0xC28FCB00)		 = -1 Err#60
psynch_cvsignal(0x7FBA48000600, 0xDEBF3000DEBF3100, 0xDEBF3000)		 = 257 0
psynch_cvsignal(0x7FBA480007F8, 0xDFB4A000DFB4A100, 0xDFB4A000)		 = 257 0
psynch_cvwait(0x7FBA48000600, 0xDEBF3001DEBF3100, 0xDEBF3000)		 = 0 0
psynch_cvsignal(0x7FBA48000408, 0xE65DA200E65DA300, 0xE65DA200)		 = 257 0
psynch_cvsignal(0x102AC5180, 0xC28FCC00C28FCD00, 0xC28FCB00)		 = 257 0
psynch_cvwait(0x102AC5180, 0xC28FCC01C28FCD00, 0xC28FCB00)		 = 0 0
psynch_mutexdrop(0x7FBA48000318, 0x4DD62603, 0x4DD62500)		 = 0 0
psynch_mutexwait(0x7FBA48000318, 0x4DD62603, 0x4DD62400)		 = 1305880067 0
psynch_cvwait(0x102AC5180, 0xC28FCD01C28FCE00, 0xC28FCD00)		 = -1 Err#316
psynch_cvsignal(0x7FBA48000600, 0xDEBF3100DEBF3200, 0xDEBF3100)		 = 257 0
psynch_cvwait(0x7FBA48000600, 0xDEBF3101DEBF3200, 0xDEBF3100)		 = 0 0
psynch_cvsignal(0x7FBA480007F8, 0xDFB4A100DFB4A200, 0xDFB4A100)		 = 257 0
psynch_cvwait(0x7FBA480007F8, 0xDFB4A101DFB4A200, 0xDFB4A100)		 = 0 0
psynch_cvwait(0x102AC5180, 0xC28FCE01C28FCF00, 0xC28FCD00)		 = -1 Err#316
psynch_cvsignal(0x7FBA48000210, 0x7294E0007294E100, 0x7294E000)		 = 257 0
psynch_mutexdrop(0x7FBA48000120, 0x59BC7C03, 0x59BC7B00)		 = 0 0
psynch_mutexwait(0x7FBA48000120, 0x59BC7C03, 0x59BC7A00)		 = 1505524739 0
@amasson88

This comment has been minimized.

Copy link

commented Oct 31, 2018

CPU usage of Hyperkit drops from 46% to 2% when I disable Kubernetes in Docker pref panel.

@atombender

This comment has been minimized.

Copy link

commented Oct 31, 2018

... something to do with how disks are mounted and maybe polled for changes (if that's even relevant). The problem seems to be significantly worse when I have front-end build tools (gulp, webpack) watching for file changes in locations that are mounted.

The CPU usage problem shows up even when no disks are mounted and all the Kubernetes processes should essentially be idle, so let's not muddy the waters with unrelated factors. There are, certainly, performance issues with volumes and file system watchers, but those things aren't needed to provoke this particular problem.

@mattstein

This comment has been minimized.

Copy link

commented Oct 31, 2018

Fair enough @atombender, I can just confirm on two separate machines that kubernetes isn't required for extraordinary CPU usage from the hyperkit process.

@erulabs

This comment has been minimized.

@atombender

This comment has been minimized.

Copy link

commented Oct 31, 2018

@mattstein: That is something of an understatement. 😬

@erulabs

This comment has been minimized.

Copy link

commented Oct 31, 2018

FWIW, Without kubernetes, docker behaves fine for me, but still spews tons of psynch_cvwait Err#316 while at 3% cpu usage, so I'm going to re-focus on the kubernetes side of the picture for now.

docker stats with kube enabled does show that kube-controller-manager and kube-apiserver are both using 5% of the CPU (those %s dont match up outside and inside the VM tho). I think this idle load on those components is my source of pain. FWIW, in production on AWS on 1.10, I don't get CPU usage like this (and thats when there is actually stuff for kube-apiserver to do!)

@naviat

This comment has been minimized.

Copy link

commented Nov 6, 2018

same here
Docker version 18.06.1-ce, build e68fc7a

@sgloutnikov

This comment has been minimized.

Copy link

commented Nov 16, 2018

I just ran into this and got worried that when I started Kubernetes com.docker.hyperkit was using 50% of my CPU. It turns out I jumped to that conclusion too quickly without reading the activity monitor properly. Please double check this yourself.
screen shot 2018-11-15 at 7 53 55 pm

@jjungnickel

This comment has been minimized.

Copy link

commented Nov 16, 2018

@sgloutnikov Those two measurements are scaled differently. While the process metric is scaled to a single cpu core and can rise above 100%, the summary below is a measurement over all available cores.

@sgloutnikov

This comment has been minimized.

Copy link

commented Nov 16, 2018

@jjungnickel yes, that is exactly what I meant. Some people had posted the per core at 50%, which is different.

@yugansh20

This comment has been minimized.

Copy link

commented Nov 23, 2018

I am facing same problem even without kubernetes. I am running docker edge and after turning off kubernetes also laptop is making sound and CPU usage is high.

One solution I found is to limit CPU for docker under Preferences - Advanced.

I have changed it to 1 CPU and now usage is much lower

@atombender

This comment has been minimized.

Copy link

commented Nov 23, 2018

@yugansh20 This issue is only about Kubernetes. There are other issues, I'm sure, about non-Kubernetes CPU usage.

@yugansh20

This comment has been minimized.

Copy link

commented Nov 23, 2018

@atombender Thank you. I think reducing CPU allocation should help for k8s issue as well.

@atombender

This comment has been minimized.

Copy link

commented Nov 23, 2018

That is true, but if you run multiple workloads, they will want enough CPU so that the VM doesn't become overloaded, leading to pings that start to fail, image download failures, volume timeouts, etc.

@tungd

This comment has been minimized.

Copy link

commented Dec 12, 2018

Personally I think this is a Kubernetes problem. I tried to go back to Minikube with VMware Fusion driver and the vmware-vmx process is also using 25% CPU doing nothing. It might not be as severe as with HyperKit (Minikube + HyperKit is also 50% CPU) but I guess the K8s controller services is the main cause here.

@EduardoLeonMiranda

This comment has been minimized.

Copy link

commented Jan 30, 2019

Same problem here Mojave(18D42) Docker Version 17.12.0-ce-mac46 (21698), I see 90% CPU, any ideas?

@steeve

This comment has been minimized.

Copy link

commented Jan 31, 2019

Same here, 50% CPU while idle with no containers on Mojave and Docker for Mac Version 2.0.1.0 (30090)
I tried to top in side the d4m vm, and the most I see is around 3-4% with kube-apiserver.

@miholeus

This comment has been minimized.

Copy link

commented Feb 9, 2019

Same, CPU rises to 50% when enabling kubernetes local

@prehor

This comment has been minimized.

Copy link

commented Feb 9, 2019

Here are reported similar problems with minikube on Linux with VirtualBox and KVM2 driver: kubernetes/minikube#3207

@wuestkamp

This comment has been minimized.

Copy link

commented Mar 30, 2019

Same issue, constant ~50%

@Foritus

This comment has been minimized.

Copy link

commented Apr 4, 2019

Might be related to this? kubernetes/kubernetes#75565

TL;DR Kuberenetes is very CPU hungry relative to small clusters and this is currently by design

@fennb

This comment has been minimized.

Copy link

commented May 5, 2019

I'm also observing this behaviour too (only when kubernetes is running - approx 45% of host CPU used by com.docker.hyperkit constantly on an early 2015 i5 macbook pro).

I dug in a little bit using a privileged container playing around with kill -STOP & kill -CONT. No single process seems to be responsible for the load, however, if you pause kube-controller, kube-apiserver, kube-schedule and kubelet then (perhaps unsurprisingly) CPU usage drops to almost nothing.

kube-controller appears to be responsible for about 30% of the usage, unsure if this is useful but if I profile a running kube-controller process:

# strace -p 2944 -f -c

strace: Process 2944 attached with 9 threads
strace: Process 2944 detached
strace: Process 3062 detached
strace: Process 3063 detached
strace: Process 3064 detached
strace: Process 3118 detached
strace: Process 3129 detached
strace: Process 3623 detached
strace: Process 3750 detached
strace: Process 3751 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 73.70  244.398693        7598     32168      4775 futex
 21.97   72.853979        7376      9877           pselect6
  2.23    7.398116       11243       658           sched_yield
  1.30    4.302810       20392       211           epoll_wait
  0.44    1.454464        8036       181        90 read
  0.33    1.089365        9903       110           write
  0.02    0.053624       13406         4           setsockopt
  0.01    0.026812       13406         2           epoll_ctl
  0.01    0.026812       13406         2         1 accept4
  0.00    0.013406       13406         1           close
  0.00    0.013406       13406         1           getsockname
------ ----------- ----------- --------- --------- ----------------
100.00  331.631487                 43215      4866 total
@planetf1

This comment has been minimized.

Copy link

commented Jul 10, 2019

FYI Still seeing this on Catalina - around 40-45% idle on a MBP 2016 high-end. It makes k8s unusable or at least tedious in a battery environment. I'm persuing a lab setup. Already have k8s charts, but think I'll need to do a cutdown single docker image (dirty!) or docker compose/stack JUST to address the k8s load which is a real shame vs using std technology.

@jnicholls

This comment has been minimized.

Copy link

commented Jul 10, 2019

@planetf1 I'm confident this is a k8s component and not macOS or Docker. Have you tried KIND? https://github.com/kubernetes-sigs/kind It may not exhibit the same behavior, but I haven't tried it myself personally. You can get k8s setup on a single node in EC2 or Azure for a low cost as well. I'm not sure if the behavior still exists in Kube 1.14+. The latest Docker bundles Kube 1.14.3. I'll give it a try and report back.

@steeve

This comment has been minimized.

Copy link

commented Jul 10, 2019

I switched to https://github.com/rancher/k3s running in docker, and CPU is around 25% on one core. Way better.

@planetf1

This comment has been minimized.

Copy link

commented Jul 10, 2019

Thanks for the tips. Myself I mostly use multiple cloud environments to offload from my mac. Or have k8s running on other linux systems. The requirement here for me, was for a simple entry point for a talk/hands on lab where people would turn up to try out our software... even with developers the -on-ramp time takes a while, hence the thought of using docker desktop for mac & windows with k8s assets I already have. But I think this will cause issues sadly so looking at other options too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.