Docker healthcheck causes higher than usual idle CPU usage

[tanc]

Bug Report

Tell us about your setup

Lando: v3.0.0-beta.47
OS: MacOS 10.13.5

Tell us about your .lando.yml

recipe: drupal8
config:
  webroot: web
  via: nginx

Tell us generally about your bug

Not a bug but I noticed Lando idles at a higher CPU than the other Docker based environment I have been testing (Docker for Drupal).

When Docker for Mac is first started it's hyperkit process idles at around 2-3% CPU on my machine.

When a Drupal 8 recipe based Lando controlled set of containers is started it idles at 15-20% CPU when using NGINX.

The default Drupal 8 recipe using Apache idles at around 7-8% CPU.

I've ssh'd into the app container but its not obvious which process is using CPU.

In comparison I've spun up a Drupal 8 set of containers using Docker for Drupal and this idles at basically the same amount as not having any containers spun up (around 2-3% CPU).

Any idea what is causing this CPU load? It would be nice if there was no (extra) load having a Lando site up so that battery lasts longer on laptops and heat is kept to a minimum.

[tanc]

This gif shows the nginx and mariadb containers using CPU when idle, using docker stats:

.lando.yml:

name: mytest
recipe: drupal8
config:
  webroot: gitroot/web
  database: mariadb
  xdebug: true
  via: nginx

[Etroid]

Noticing the same issue when running a pantheon D8 recipe.

[sherakama]

I am pretty sure this is a Docker/OSX bug. See this thread: docker/for-mac#1759

I ran in to this issue as well and have updated my OS and Docker Edge to the latest. This has helped with the CPU usage during idle times significantly for me. One other factor is that often I override the database image to use a specific version of mysql. There are special flags to help with performance when doing this and you can append a :cached or :delegated flag to your image.

eg:

services:
  database:
    type: mysql:5.7:delegated

[tanc]

@sherakama can you point to the docs (or code) where it explains the :cached and :delegated flag in Lando? (I know about the flag generally in Docker).

I'm not sure the high idle CPU is related to that issue, although I'll keep an eye on it. With no containers running the idle CPU is very low. With Docker for Drupal or DDEV containers running idle is also very low. When running Lando containers I see idle higher. So it seems like there is some kind of polling or other regular task running that keeps CPU higher.

[justageek]

My see 250% plus cpu load for com.docker.hyperkit when starting lando, primarily with sites using the drupal8 recipe, I've been trying to troubleshoot but no luck so far, I'll be happy to contribute information and tests if anyone has any progress,

Lando - v3.0.0-beta.47

Docker
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:05:26 2018
OS/Arch: darwin/amd64
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:13:46 2018
OS/Arch: linux/amd64
Experimental: true

[justageek]

This has solved my issues: https://coderwall.com/p/pl3rfg/fix-docker-for-mac-high-cpu-usage

[iBobik]

I think it is not because Docker for Mac, but because of some strange conf from Lando. I usually have stated multiple Drupal projects but only some use Lando. And only Lando's database eats ~5% of CPU while idle, other databases are on zero. Happens on both MariaDB and MySQL.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[iBobik]

This issue has not been solved yet. 18. 3. 2019 v 19:01, stale[bot] <notifications@github.com>:

…

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[dasginganinja]

I'm not noticing this issue when I use the lamp recipe. When using the drupal8 recipe I see lots of CPU comparatively.

Stats from docker stats:

lamp:

drupal8:

[ccharlton]

I've been using 'docker pause INSTANCE_1 INSTANCE_2 INSTANCE_3' for the services I'm not using at that time. It barely helps.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[iBobik]

Still draining our batteries. 16. 5. 2019 v 0:14, stale[bot] <notifications@github.com>:

…

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[iBobik]

Still draining our batteries.

…

15. 6. 2019 v 10:27, stale[bot] ***@***.***>: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this <https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea> if you are wondering why we auto close issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1088?email_source=notifications&email_token=AAEV6WHRJEUWC6Q2MUT27XLP2SRWZA5CNFSM4FI6AX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYTGTY#issuecomment-502346575>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEV6WDVUYCSWAF7XMBDGCDP2SRWZANCNFSM4FI6AX6Q>.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[iBobik]

Idle Lando still have probably bigger carbon footprint than Bitcoin mining ;-)

…

15. 7. 2019 v 18:24, stale[bot] ***@***.***>: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this <https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea> if you are wondering why we auto close issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1088?email_source=notifications&email_token=AAEV6WHUJ7KWKJOZDVM2OFLP7SQFDA5CNFSM4FI6AX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6HA7A#issuecomment-511471740>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEV6WB522PCQHNODFWROYLP7SQFDANCNFSM4FI6AX6Q>.

[MatthieuScarset]

Indeed I confirm that using Lando drupal8 recipe use a lot of ressources compare to other Docker-based solution on Ubuntu .

Just a though here.. but is there anyway to limit memory and CPUs when running lando start ?

Something like:

lando start --memory=”1g” --memory-swap=”2g” --cpus=".5"

Example of high CPU use by two database containers of one single project (idle):

[pirog]

@iBobik obviously you've never mined bitcoin before.

@MatthieuScarset
on macOS/Windows this happens automatically and is configurable generally through the Docker Destkop UI. If you are on Linux you probably need to configure your docker daemon to operate within specific resources thresholds.

you also might be able to use Lando’s service override functionality to specify resource restraints at the docker-compose level and on per service basis

[iBobik]

This issue is not complain on CPU consumption when on load, but when idle - no requests to webserver neither to database, but is still do something. This is huge energy consumption cumulatively. Only workaround is to stop Lando when not using the project.

[pirog]

@iBobik yup, we can read.

[iBobik]

Probably this one: https://github.com/bitnami/bitnami-docker-mysql

…

8. 10. 2019 v 11:11, Ricardo Amaro ***@***.***>: I may have now a way to fix this permanently for a Drupal setup also. Where should I send the PR to? Where are the mysql templates? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1088?email_source=notifications&email_token=AAEV6WFAB7M7NNC4AJ5YSJLQNRFC7A5CNFSM4FI6AX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEATPS4A#issuecomment-539425136>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEV6WASHJJJWV5QTNFYWNDQNRFC7ANCNFSM4FI6AX6Q>.

[ccharlton]

@ricardoamaro got details (or a Gist) to share for us curious?

[ricardoamaro]

it's here #1088 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[iBobik]

Workaround `healthcheck:` works but it will be better if databases will be configured like this by default in Lando.

…

29. 11. 2019 v 14:01, stale[bot] ***@***.***>: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this <https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea> if you are wondering why we auto close issues. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1088?email_source=notifications&email_token=AAEV6WEDP7H6SE5J5USEWTLQWEHDHA5CNFSM4FI6AX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFOZXRY#issuecomment-559782855>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEV6WHRISADVJCIL2BGD2DQWEHDHANCNFSM4FI6AX6Q>.

[RobertAKARobin]

Every time I run Lando my computer (a 3-month-old Macbook Pro) sounds like a vacuum cleaner and gets all toasty. It would be great to have a fix merged in.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[RobertAKARobin]

I use Lando very often and so would like to see this fixed.

[sryanb]

I'm seeing remarkable improvements from disabling healthcheck and then running lando rebuildas described above.

Lando version: v3.0.0-rc.22
Docker Desktop: 2.1.0.5 (configured to use 1 CPU, 8GB RAM)
Macbook Pro 16" (2019) running Catalina (10.15.2)

[Dopeyr]

This has had a positive impact on my machine too. I was running 2 solr instances, 1 database and limiting the interval to every 60s on all services has reduced the idle load from about 60% CPU to less than 10%, typically 5%.

Does anyone know if these healthchecks are supposed to keep running? I understand they are using in the startup sequence, but are they meant to keep running post start? I don't see any messages from lando if I stop a container via docker for example.

[pirog]

So we definitely don't want to get rid of the healthcheck because we want to be able to tell the user that something is not working correctly. Also this is super helpful to the Lando team for debugging and support purposes.

That said, i don't think we need the persistent running-all-the-time healthcheck that docker provides out of the box. It also doesn't look like you can configure the docker healthcheck to do something like "run 10 times at 5 second intervals on start" which is unfortunate.

This makes me think the best-of-both-worlds resolution here would be to reimplement the healthchecks at the lando level so that we can run them post-start with some reasonable retry and backoff until they fail/succeed and then we leave the container be. Guessing that would allow the UX on the healthchecks to stay the same but would drastically cut idle CPU after an app has booted up correctly.

[pirog]

Alright, a reimplementation of the healthcheck has been merged into master with #2136

This replace the docker healthcheck with a lando one. the relevant consequence there is the healthcheck stops after the app has started which should cut down on the idle cpu usage. guessing this will be available in the next edge release.