New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High CPU utilisation for 'com.docker.hyperkit' process #82

Closed
cmacrae opened this Issue Aug 2, 2016 · 32 comments

Comments

Projects
None yet
@cmacrae

cmacrae commented Aug 2, 2016

Expected behaviour

Docker for Mac should launch as it has previously and not start consuming high amounts of CPU resources, even when "idle".

Actual behaviour

Starting Docker for Mac spawns a process 'com.docker.hyperkit' that immediately starts hogging CPU resources. Even when "idle" (i.e: no containers are running, and/or no interaction with docker at all).

Information

  • Diagnostic ID: 5324838F-D30E-435A-995B-48D8B2B150B2
  • Diagnostic information from Docker for Mac:
Docker for Mac: version: win-beta-v1.12.0-rc5-beta21-68-g5b59d37
OS X: version 10.11.6 (build: 15G31)
logs: /tmp/5324838F-D30E-435A-995B-48D8B2B150B2/20160802-202642.tar.gz
[OK]     docker-cli
[OK]     app
[OK]     moby-syslog
[OK]     disk
[OK]     virtualization
[OK]     system
[OK]     menubar
[OK]     osxfs
[OK]     db
[OK]     slirp
[OK]     moby-console
[OK]     logs
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux
@shri3k

This comment has been minimized.

shri3k commented Aug 3, 2016

I'm facing a similar issue. I did see a warning when my machine woke up from sleep that it's a known bug in Yosemite though. Oh and I'm obviously running Yosemite.

Edit- Just FYI, I upgraded to El Capitan 10.11.6 and don't see the issue anymore

@cmacrae

This comment has been minimized.

cmacrae commented Aug 3, 2016

@shri3k - thanks for chiming in. I'm on El Capitan, the latest patch release

@chalmagean

This comment has been minimized.

chalmagean commented Aug 12, 2016

I'm getting 170% CPU load when I build a new image and COPY files into it.

I should also mention that the files I'm COPYing are from a mounted volume and all have the xattrs set to: com.docker.owner: 0:33:33

$ docker -v
Docker version 1.12.0, build 8eab29e, experimental
@AlexChesters

This comment has been minimized.

AlexChesters commented Sep 16, 2016

This is also reproducible on OS X 10.11.5 running Docker 1.12.0.

We're running a JavaScript web app and we only see this error if we attempt to run grunt watch in the Docker container and seems to only be an issue if we are watching lots of files.

@xeor

This comment has been minimized.

xeor commented Sep 16, 2016

Same here.. My mac was suddenly blowing a lot of hot air. Docker was unresponsive, and it started happening without anyone using the mac for some hours.

@dmechas

This comment has been minimized.

dmechas commented Sep 17, 2016

Hi all,

I had some problems here running my app inside docker. I use node as base image, use volume to share my app code between machines, launch app with nodemon to watch 800 files.
I check use nodemon + volumes consumes over 270% of CP

With nodemon and without volumes the CPU usage decrease.
With volumes and without nodemon the CPU usage decrease too.

So the combination between these two "shits" needs to be solve.

My solution: increase the polling-interval of nodemon.

By default it has a polling interval of 100ms.
I tried with 1 sec (1000ms) and his CPU load went from 270% to 90%. With 2 secs it went down to 60%.
There is a command line option --polling-interval which reduces the polling frequency for the legacy watch.
For the nodemon configuration file there is a corresponding pollingInterval option. Both in millisecs.

@harmw

This comment has been minimized.

harmw commented Sep 18, 2016

@dmechas where can I configure startup options of dockerd? I'd expect this to be an option available from the tray-icon but that's not true...

@justincormack

This comment has been minimized.

Member

justincormack commented Sep 18, 2016

@dmechas your issue seems unrelated to the original issue, as it is about performance of large numbers of file watches with nodemon, while the original poster as not apparently running anything at all. Can you open a different issue?

@jascudder

This comment has been minimized.

jascudder commented Sep 20, 2016

Likewise on OSX 10.10.5, CPU utilization is maxing out all cores. Same process and scenario as op.

@brwr

This comment has been minimized.

brwr commented Sep 20, 2016

I had a similar issue when using docker-toolkit. The problem was that the VirtualBox VM had /Users/ as a shared folder. My projects live at /Users/james/workspace/. Removing /Users/ as a shared folder and replacing it with /Users/james/workspace/<project_name/ solved the problem.

When I moved to Docker for Mac a couple of days ago, I started noticing the same thing. Sure enough, /Users/ was attached as a shared folder. Here's what worked for me:

  1. Click the Docker for Mac icon on the Mac toolbar.
  2. Select Preferences ....
  3. Go to the File Sharing section of the preferences.
  4. Look for /Users/ and remove it.
  5. Add the specific folder you want to share -- in my case, /Users/james/workspace/<project_name>/.
  6. Click Apply & Restart.
  7. Watch your CPU utilization drop. This may take 10-15 seconds.

Hope that helps!

@ae6rt

This comment has been minimized.

ae6rt commented Sep 21, 2016

@brwr Any idea why following your instructions should help? I don't see the connection between a mount and high CPU. Thanks.

@brwr

This comment has been minimized.

brwr commented Sep 21, 2016

My understanding is that the high CPU is a result of trying to sync too many files at the same time. Mounting folders like /Users/ and /Volumes/, which typically have a lot of stuff in them, would lead to more files being synced which leads to high CPU usage.

@brwr

This comment has been minimized.

brwr commented Sep 21, 2016

It seems this doesn't actually solve the problem. Even after unmaintained
everything but the project I'm work on, my CPU eventually spikes again.

On Sep 20, 2016 5:48 PM, "Mark Petrovic" notifications@github.com wrote:

Any idea why following your instructions should help? I don't see the
connection between a mount and high CPU. Thanks.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AD5-rDQbXxt5AHuMvYcUKX3A_mrumKiFks5qsH7wgaJpZM4Ja9z1
.

@JohannesHoffmann

This comment has been minimized.

JohannesHoffmann commented Sep 21, 2016

I run a drupal 7 installation via docker. It is a very big website with tones of php files. To generate the Website it loads a lot of the files. My Mac blows itself to the hell.
+1 for a solution!

@mrjackdavis

This comment has been minimized.

mrjackdavis commented Sep 22, 2016

I think you guys are on the right track - for me this seems to be linked to file system events.

If I run bash in a base node image, CPU usage is 3-10%
screen shot 2016-09-22 at 1 35 50 pm

Then, once I run a process which watches the file system (nodemon or similar) inside a base node image, CPU usage climbs to 100-400%.
screen shot 2016-09-22 at 1 34 18 pm

At least it's keeping my hands warm :)

@bestform

This comment has been minimized.

bestform commented Sep 25, 2016

Just wanted to say that I observe the same issue

$ docker -v

Docker version 1.12.1, build 6f9534c, experimental
@mrjackdavis

This comment has been minimized.

mrjackdavis commented Sep 25, 2016

For those who experience high CPU because of FS events (like me, see above). @justincormack I agree with your earlier statement, I doubt this is the fault of docker.

It is, however, a noticeable change from using docker-machine. Reason being, I did not actually observe the high CPU because it was obfuscated by the VM.

I resolved my particular issue by disabling polling on webpack and removing -L, legacy watching, from Nodemon. I had to enable these initially because docker-machine didn't support file system events.

I'd suggest inspecting your config to ensure polling strategies are disabled in lieu of FS events.

@justincormack

This comment has been minimized.

Member

justincormack commented Sep 25, 2016

Yes, I would definitely avoid polling. Making notifications work has been something we have spent a lot of time on, so you do not have to use inefficient polling.

@jonatasbaldin

This comment has been minimized.

jonatasbaldin commented Sep 26, 2016

I'm experiencing the same issue.

Docker version 1.12.1, build 6f9534c
Mac OS X Sierra 10.12
@zachary-russell

This comment has been minimized.

zachary-russell commented Sep 28, 2016

I'm also experiencing this issue.

Docker version 1.12.1, build 6f9534c, experimental

I'm using docker-compose to run a basic WordPress config, doing actions within it is super slow. Note: this does not occur on a linux VPS that i'm running.

Please let me know if I can do anything to help.

@LQkkeN

This comment has been minimized.

LQkkeN commented Sep 29, 2016

I have this issue as well. Any container with Grunt watching mounted files will cause the CPU to run amok. I suspect it's related to the fact that the files are being watched by both Grunt and Docker, concurrently.

OS X 10.11.6
Docker build: 12496

@bestform

This comment has been minimized.

bestform commented Sep 29, 2016

I have this issue as well. Any container with Grunt watching mounted files will cause the CPU to run amok. I suspect it's related to the fact that the files are being watched by both Grunt and Docker

What @mathiaslm89 says is interesting. As I said I observe the same problem and I have a similar situation, where a container is watching the same files as my IDE. This might be nothing, but I suppose it couldn't hurt to mention it. Maybe it is a factor.

@dsheets dsheets removed the area/docker label Sep 29, 2016

@dsheets

This comment has been minimized.

Contributor

dsheets commented Sep 29, 2016

If you are seeing increased CPU utilization while running programs that 'watch' files on shared directories, you are not experiencing the same issue as the original poster in this thread. Many popular file system monitoring applications do not actually use the asynchronous event APIs of the platform they run on -- grunt in particular does not use inotify by default but instead polls the file system with a relatively high frequency to simulate FS event notification. This is a common problem even with native grunt configurations. See gruntjs/grunt-contrib-watch#429 in which someone recommends using the chokidar backend which does use inotify.

See this previous comment in which it is suggested that, whatever file system watching system you are using, you ensure that it is not polling large numbers of files at high frequency and instead uses inotify.

To reiterate, this issue is about the case where the hyperkit process appears to consume a lot of CPU for no apparent reason when Docker is otherwise idle. We believe the root cause for this behavior lies in the new Hypervisor.framework component of OS X which we rely on to access hardware-assisted virtualization features.

Thanks!

@bestform

This comment has been minimized.

bestform commented Sep 29, 2016

See this previous comment in which it is suggested that, whatever file system watching system you are using, you ensure that it is not polling large numbers of files at high frequency and instead uses inotify

Thank you @dsheets for clearing this up. I'll take back my addition then, as the IDE I am using is in fact utilizing inotify to watch files.

@mwarren2

This comment has been minimized.

mwarren2 commented Oct 2, 2016

Two days ago I switched from the beta to Docker for Mac 1.12, and got this same problem, which occurs on every build.
The beta was fine as far as CPU was concerned.

@mttkay

This comment has been minimized.

mttkay commented Oct 21, 2016

I just ran into the same issue with the latest Docker for Mac beta (1.12.2-beta28 (12906))

I ran the diagnosis to see if that comes up with anything, but it appears to fail:

screen shot 2016-10-21 at 10 14 14

Docker seems to be completely stuck when I try to issue any command through the CLI.

Force-killing the hypervisor process seems to recover from the situation.

@anonymuse

This comment has been minimized.

anonymuse commented Oct 27, 2016

Having a similar problem myself and being relatively intolerant of GUI-based intervention, I've found that pkill-ing docker at the command line provides a seamless restart of the Docker engine and associated processes.

If you run
$ sudo pkill docker
the Docker for Mac application will restart all associated processes and allow you to get back to work with (what appears to be) minimal disruption.

There's a hearty YMMV here, though!

@tweep

This comment has been minimized.

tweep commented Oct 28, 2016

Same problem on Mac OS X "El Capitan" 10.11.6 - com.docker.hyperkit is draining my battery by using 100% cpu. Shutting Docker down via icon in menu bar did not work - I had to force-quit the process.

I'm using Docker Version 1.12.1 (build: 12133)

@adammolnar

This comment has been minimized.

adammolnar commented Oct 28, 2016

I had the same issue and used my macbook as a heating in the last days. I'm using a webpack container which is running in docker. In my webpack.dev.js config file I added the following:

var config = {
  ...
  watchOptions: {
    poll: false,
    ignored: /node_modules/
  },
  ...
}

and in the package.jsonof the project I removed the --watch-poll flag in the start script. I don't know which one solved the problem, but now my CPUs running on ~15%. I hope it helps.

@zachary-russell

This comment has been minimized.

zachary-russell commented Oct 28, 2016

@adammolnar are you using just a regular MacBook or a MacBook pro? I'm more curious to see how docker for mac performs on a Macbook as i'm considering it for my next purchase.

@Murazaki

This comment has been minimized.

Murazaki commented Oct 30, 2016

For the same containers, Docker use 200% CPU on my Mac, as opposed to 99% CPU when running through VBox.

macOS 10.12.1 (16B2555)
MacBook Pro 3,1 GHz Intel Core i7, 16 Go 1867 MHz DDR3 memory

$ docker -v
Docker version 1.12.1, build 6f9534c
@ijc

This comment has been minimized.

Contributor

ijc commented Oct 31, 2016

All,

To reiterate and expand upon @dsheets' earlier comment there are many possible reasons why each of you may be seeing high CPU usage, ranging from macOS kernel bugs (in 10.10.x and early 10.11.x in particular) to the use of polling within container (e.g. build systems for various languages) to potential Docker for Mac bugs or inefficiencies. Many of the subsequent comments share little in common with the original report or indeed each other but lack the necessary diagnostic IDs and descriptions of use cases which would enable us to identify what is going on.

So I am going to close and lock this issue. For anyone who is seeing high CPU usage please create a fresh ticket including details of your own specific environment and use case, possible reproduction steps and a diagnostic upload. The easiest way to do this is via the "Diagnostics and Feedback" item in the whale menu and filling in the template with the details.

@cmacrae I'm very sorry that in amongst it all we do not appear to have actually looked at your original issue in any detail. When I come to do so now I find that your diagnostic tarball is not present on our servers for some reason. Since this ticket seems to have become rather unfocused would you mind uploading a new one with a new issue to go with it (assuming you can still reproduce)? I'm very sorry for the inconvenience and the long delay in getting to your issue.

@ijc ijc closed this Oct 31, 2016

@docker docker locked and limited conversation to collaborators Oct 31, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.