Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docker Desktop] Investigate Mac CPU performance #12

Closed
nebuk89 opened this issue Mar 6, 2020 · 119 comments
Closed

[Docker Desktop] Investigate Mac CPU performance #12

nebuk89 opened this issue Mar 6, 2020 · 119 comments
Labels
docker_desktop Improvements or additions to Docker Desktop

Comments

@nebuk89
Copy link
Contributor

@nebuk89 nebuk89 commented Mar 6, 2020

Tell us about your request
CPU performance on Mac is a long standing issue, we have investigated this in the past but want to dedicate time to re-invetsigate this and try and find a fix

Which service(s) is this request for?
Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We keep spiking to 100% CPU and this isn't good, no one likes their laptop taking off

Are you currently working around this issue?
N/A
Additional context
N/A
Attachments
docker/for-mac#1759
docker/for-mac#3499
docker/for-mac#2601

@aguedeney
Copy link

@aguedeney aguedeney commented Apr 3, 2020

Ben, for purposes of documentation of pain points with Docker on Mac, I wanted to make sure there was a reference back to these complaints.

docker/for-mac#3499

Thanks -Alan

@cculianu
Copy link

@cculianu cculianu commented Apr 4, 2020

Please add docker/for-mac#4392 to this list.

@nebuk89
Copy link
Contributor Author

@nebuk89 nebuk89 commented Apr 6, 2020

Thanks @aguedeney and @cculianu - when we move this into in progress I will link back the tickets we are using in for-mac to highlight where we will be discussing with everyone and also where we will be seeking fresh logs for issues :) please feel free to add other cases here as you think appropriate!

@schellj
Copy link

@schellj schellj commented Apr 14, 2020

Please add docker/for-mac#4064.

@mc0
Copy link

@mc0 mc0 commented May 7, 2020

The primary issue with CPU that I have seen relates to volume mounts, transfused, and heavy IO workloads. I have observed this a lot of times and inside the VM it is usually transfused using the CPU.

See this on why mounts use lots of CPU. Namely:

A Linux system call such as open or read that accesses bind-mounted files in a container must be.

- turned into a FUSE message in the Linux VFS
- proxied over a virtio socket by transfused
- forwarded onto a UNIX domain socket by HyperKit
- deserialized, dispatched and executed as a macOS system call by osxfs

The entire process then takes place in reverse to return the result of the macOS system call to the container.

@daveisfera
Copy link

@daveisfera daveisfera commented May 7, 2020

See this on why mounts use lots of CPU.

This definitely makes sense for bind mounts because it's having to go in and out of the VM to interface with the host filesystem, but I'm still seeing high CPU usage after we switched to using named volumes and that should all be inside of the VM, but I'm still seeing high CPU usage when things are sitting idle, so is there any similar :cached/:delegated tags for named volumes to help with performance issues there?

@m8rge
Copy link

@m8rge m8rge commented May 8, 2020

This issue is too broad. Some people may suffer from mounts CPU performance, but other have 100% cpu after prune command in this issue: docker/for-mac#4064

@nebuk89
Copy link
Contributor Author

@nebuk89 nebuk89 commented May 11, 2020

Thanks to all the responses!
We will start our investigation on this in the next ~2 weeks!
Watch out for incoming questions on some of the for-mac issues. We won't respond to posts in here with logs etc but will use this to share updates as we make progress :)

@nebuk89 nebuk89 moved this from Investigating to We're Writing The Code in docker-roadmap May 11, 2020
@alechirsch
Copy link

@alechirsch alechirsch commented May 16, 2020

Hope this helps in some way.

I was running into CPU issues as well and noticed something weird happening with volumes.
I created three volumes in my compose file, one for the project directory, one to ignore node_modules, and another to mount a specific node_module.

It works as expected, a file from the specific node_module that I modified locally was copied to the container. Likewise I modified a file in another node_module and the change was not copied to the container. Everything works fine.

The issue came when I did a docker-compose restart, the other node_module that should have been ignore was copied into the container, and my laptop fans were taking off. Every file in node_modules after that point was copied over to the container even though I specified it not to.

I did investigating a few months ago so the reproduction steps might be off, but there was definitely an issue with volumes copying files that it was not supposed to.

 - volumes
    - ./:/src
    - /src/node_modules
    - ./node_modules/express:/src/node_modules/express

@andig
Copy link

@andig andig commented Jun 4, 2020

I would like to highlight that docker/for-mac#3499 (comment) has a concrete suspicion of potentially bugged code that might be easy to investigate (with OCAML knowledge).

The primary issue with CPU that I have seen relates to volume mounts, transfused, and heavy IO workloads. I have observed this a lot of times and inside the VM it is usually transfused using the CPU.

I would also like to add that this is not the only cause. I do sometimes see docker spike to at remain at 100% without any containers running. Definitely no high IO workload in that case, pure "home" usage.

@schellj
Copy link

@schellj schellj commented Jun 4, 2020

I've also had times where the Docker process was using 100% CPU indefinitely when no containers were running.

@ebriney
Copy link
Member

@ebriney ebriney commented Jun 18, 2020

@schellj , can you do a diagnose when it happens? (if the whale menu is responding of course).
I you cannot use the UI, you can still do it with cli command:
/Applications/Docker.app/Contents/MacOS/com.docker.diagnose gather -upload

@schellj
Copy link

@schellj schellj commented Jun 18, 2020

@ebriney I haven't had any issues with Docker for Mac using a lot of CPU unexpectedly since early May - see docker/for-mac#4064 (comment), but I did a couple of diagnostics when I was - see docker/for-mac#4064 (comment) and docker/for-mac#4064 (comment) (I made some other relevant comments in that issue as well).

@haslo
Copy link

@haslo haslo commented Jun 26, 2020

I installed the edge build (with spindump) and uploaded this diagnostic:
0D3FCEE0-663F-4A5E-8DA2-1B79B6746B54/20200626175354

The Kubernetes containers were running (with nothing to manage), nothing from my side, and Docker used 50% CPU.

@m8rge
Copy link

@m8rge m8rge commented Jun 27, 2020

My Diag Id with spindump 80E081FA-7763-4245-BB3F-15814B31E0BB/20200627174556

@joerobot
Copy link

@joerobot joerobot commented Jun 30, 2020

Another diag ID F2F1BBBF-4BF7-4158-BCFA-0C29469D1E33/20200630084358

@andig
Copy link

@andig andig commented Jun 30, 2020

Here's another one with only 5-10% but it still seems like a lot with no containers running on a decent MBP: A913E1F9-1085-4C2B-A79D-4B6862C0EA1E/20200630095026

@xjrcode
Copy link

@xjrcode xjrcode commented Jul 1, 2020

Another diag C39B37CE-DA64-46FE-B4ED-3A714CE9CC62/20200701075247 with spindump version.

image

Using Big Sur but I feel no differences from Catalina…

@mikepflu
Copy link

@mikepflu mikepflu commented Jul 1, 2020

Another diag: 93B23EAA-7413-4ACB-8278-F008093AD412/20200701154406
I don't think I'm using spindump version, still reading up on this ticket, but my mac is about to take off and I've had a number of crashes lately. I suspect that they're related to docker, but that's just a guess on my part.

@nm777
Copy link

@nm777 nm777 commented Aug 26, 2020

@stephen-turner
My issues with high CPU due to serial console activity appear to still be resolved in this latest edge build.

@chnbz
Copy link

@chnbz chnbz commented Aug 26, 2020

I had an horrible experience with Edge 2.3.5.0. There was no high cpu usage when no container is running, but the cpu usage goes through the roof once I start my container for a simple php web project, the login page takes minutes to load. It didn't make a difference if I had gRPC-FUSE turned on or not. Went back to Edge 2.3.4.0 for now

Diagnostics ID if you guys wanna have a look: 6DF43448-D8E6-475E-BC75-3613D91113E8/20200826150527

@vredeling
Copy link

@vredeling vredeling commented Aug 30, 2020

The new edge build works pretty well for me. I can work for quite some time comfortably. CPU use stays at around 25% with a couple of containers running.

But after a while (2 hours or so) or when waking my machine from sleep, com.docker.hyperkit flares up to 200% - 250% and stays there until I stop all containers. With no containers running I'm at 5% cpu.

@JamesonNetworks
Copy link

@JamesonNetworks JamesonNetworks commented Sep 2, 2020

Concur with @vredeling

Edge 2.3.6.0 is working much better for me. I think there is at least one dangling issue, however: Returning from sleep sends the hyperkit process spinning on all CPUs. Stopping the containers does not help. I am able to stop the bleeding be using Restart from the Docker tray icon. I have gRPC-FUSE enabled. Let me know if there is any information I can provide to help diagnose the issue.

@jsharpe
Copy link

@jsharpe jsharpe commented Sep 2, 2020

With gRPC-FUSE enabled I'm getting some operation denied calls when a bazel build in the docker container attempts to create a symlink within a bind mounted directory.

@stephen-turner
Copy link
Collaborator

@stephen-turner stephen-turner commented Sep 7, 2020

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

@djs55
Copy link

@djs55 djs55 commented Sep 7, 2020

@jsharpe thanks for your feedback -- are you able to create and share a small repro case of the problem? If not, next time it manifests, could you upload a diagnostics report and quote the ID on a fresh ticket on https://github.com/docker/for-mac ? Thanks!

@jsharpe
Copy link

@jsharpe jsharpe commented Sep 7, 2020

@djs55 Yes to repro:

git clone https://github.com/abseil/abseil-cpp.git
cd abseil-cpp
docker run \
  -e USER="$(id -u)" \
  -u="$(id -u)" \
  -v $PWD:/src/workspace \
  -w /src/workspace \
  l.gcr.io/google/bazel:2.2.0 \
  --output_user_root=/src/workspace/build_output \
  build //absl/...

Gives output when grpc-fuse is on:

Extracting Bazel installation...
FATAL: failed to create installation symlink '/src/workspace/build_output/a08c2e4811c846650b733c6fc815a920/install': (error: 1): Operation not permitted

When using osxfs you will see bazel build the software.

This is on edge 2.3.6.0

@JamesonNetworks
Copy link

@JamesonNetworks JamesonNetworks commented Sep 10, 2020

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

@stephen-turner Alright, got an ID for the diag:
12F1947A-2753-47C1-A19E-A206946809E5/20200910125157

This is from version 2.3.6.0

Shutting down all of the containers seems to return the hyperkit process to normal. So this may be more a function of what I'm doing in containers (django services). However, I still do not believe returning from sleep should cause that kind of a meltdown, but at least the process does eventually calm down now.

After sitting with this issue for a couple of weeks, I'm noticing the high CPU calm down after about 30-40 mins. I assume this is actually expected behavior and not an issue. Wanted to come back here and update in case anyone was looking into it. I believe 2.3.7.0 solves all my CPU woes.

I appreciate all that the docker team is doing, thanks for the good work and awesome software everyone!

@j13k
Copy link

@j13k j13k commented Sep 23, 2020

I just thought I'd report a positive outcome from upgrading to the 2.3.7.0 Edge release, following advice in the comments here.

For three days since upgrading from Stable 2.3.0.4 to 2.3.0.5, I had constant high CPU (and associated fan noise), despite virtually idle container activity.

Following the upgrade, I can report that hyperkit %CPU is sitting just under 20%:

image

@vredeling
Copy link

@vredeling vredeling commented Sep 25, 2020

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

I've not encountered this issue any more since I moved to Docker Desktop 2.3.7.0

@djs55
Copy link

@djs55 djs55 commented Sep 30, 2020

The latest CPU-related changes from the edge 2.3.7.0 have now made it to stable 2.4.0.0 for Mac and Windows. If you get a chance to try this, let us know how it goes. If there are still CPU problems please provide as much information as you can: ideally a repro example and a set of diagnostics.

Thanks for all your help so far!

@docker docker deleted a comment from daveisfera Sep 30, 2020
@pkdetlefsen
Copy link

@pkdetlefsen pkdetlefsen commented Oct 1, 2020

I just upgraded to 2.4 for Mac and CPU usage is a lot better for the app I'm running! The code is mounted inside the container during development and the app checks all relevant files (769 atm.) for changes every 5 seconds in order to auto reload. This usually caused a good spike in CPU usage up to around 50% CPU. Now there's a consistently low usage, mostly below 5%, when checking every second!

So I can confirm that IO performance and CPU usage have improved a lot in this update in my case.

Tested on: MacBook Pro (13-inch, 2017, 2,3 GHz Intel Core i5)

@andig
Copy link

@andig andig commented Oct 1, 2020

I'm still getting 2.3.7.0 as latest release, both on stable and edge channel. What's needed to get 2.4 to install?

@rfay
Copy link

@rfay rfay commented Oct 1, 2020

@andig 2.4.0.0 is latest in Stable. 2.3.7.0 was never in Stable and never will be. See the Stable Release Notes for how to download any stable version. Note that you will lose all containers and volumes switching from edge to stable.

@sdbruder
Copy link

@sdbruder sdbruder commented Oct 1, 2020

I've upgraded to 2.4.0.0 and got a lot of ERROR: for <container-name> UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) in our complex local dev stack, we instructed all devs to revert to the previous version which solved the issue (for now).

@aleks-rope
Copy link

@aleks-rope aleks-rope commented Oct 2, 2020

I have updated Docker to 2.4.0 the CPU usage seems to be lower (now only around 10-12%) and everything has been running smooth so far on my MacBook Pro 2.8 GHz Quad-Core Intel Core i7.

@KhaledMohamedP
Copy link

@KhaledMohamedP KhaledMohamedP commented Oct 5, 2020

Yes! this is a giant improvement!!!
Thanks everyone for working on this upgrade 🎊 🙏

@PhilETaylor
Copy link

@PhilETaylor PhilETaylor commented Oct 5, 2020

And even works on MacOS Big Sur ;-)

@rpkoller
Copy link

@rpkoller rpkoller commented Oct 5, 2020

hm the 2.4.0 build was behaving well so far without any fans running extensively all of a sudden without an apparent reason. but today after starting a project and leaving the computer for half an hour, just started up a project in DDEV right before, when i returned the fans were running on max and the cpu was spiking without a stop. :/

setup: macos 10.13.6 on a mbp 13" early 2011 8GB RAM with docker desktop 2.4.0 running a Drupal site with DDEV. i've sampled the .com.docker.backend process which was constantly at +/- 100% and com.docker.hyperkit which was around 10 to 30%... the samples:
samples.zip

p.s. and according to the deprecation notice for high sierra on the first startup of the minor 2.4.0 release it was the last release for macos 10.3 so no 2.4.x bug fix release anymore?

@nebuk89
Copy link
Contributor Author

@nebuk89 nebuk89 commented Oct 8, 2020

Hey all,

We have now shipped the changes from our investigation over the last 3 months as part of the 2.4 Desktop release.
This does not mean we have got all of the bugs potentially causing this (we fixed a new one yesterday! ) but it is the point where we are no longer going to be driving a significant investigation into multiple potential root causes to this.

We will do a blog post in a few weeks detailing the things we found/changes we made to give you some insight into what we did and why.

If you are still having issues, I would ask that you create new tickets on for-mac rather than comment on old tickets as we are going to close these to keep our backlog a bit tidier and not mix up the messages (and it is easier for us to get signal on the scale of issues this way).

Thanks again for your patience and an even bigger thank you to all of you who took the time to test builds & upload diagnostics!

@nebuk89 nebuk89 closed this as completed Oct 8, 2020
@nebuk89 nebuk89 moved this from Developer Preview/Edge to Shipped! Enjoy! in docker-roadmap Oct 8, 2020
@sashaaKr
Copy link

@sashaaKr sashaaKr commented Apr 12, 2021

@nebuk89 can you share link to blog post if it was written?

@stephen-turner
Copy link
Collaborator

@stephen-turner stephen-turner commented Apr 12, 2021

@sashaaKr Unfortunately we never got round to writing the blog post.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docker_desktop Improvements or additions to Docker Desktop
Projects
docker-roadmap
  
Shipped! Enjoy!
Development

No branches or pull requests