Docker Machine is not running properly after update to macOS Catalina #83

mediaessenz · 2019-11-06T19:28:58Z

After the first generation of a new machine (in my case done by "dinghy", a reverse proxy solution) everything seems to work normal.
But after stopping the machine and try to start it again, its not working anymore.
The startup ends up with this error message

Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded
Traceback (most recent call last):
	9: from /usr/local/bin/_dinghy_command:12:in `<main>'
	8: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/base.rb:440:in `start'
	7: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor.rb:359:in `dispatch'
	6: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/invocation.rb:126:in `invoke_command'
	5: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/command.rb:27:in `run'
	4: from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:93:in `up'
	3: from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:271:in `start_services'
	2: from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:25:in `up'
	1: from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:126:in `system'
/usr/local/Cellar/dinghy/4.6.5/cli/dinghy/system.rb:18:in `system': Failure calling `docker-machine start dinghy` (System::Failure)

After starting the machine with debug option I get ten times this fault, before the upper error comes up again:

(dinghy) Calling .GetSSHHostname
(dinghy) DBG | executing: /usr/local/bin/prlctl list dinghy --output status --no-header
(dinghy) DBG | executing: /usr/local/bin/prlctl list -i dinghy
(dinghy) DBG | Found lease: 10.211.55.32 for MAC: 001C4208D8F8, expiring at 1571651690, leased for 1800 s.
(dinghy) DBG |
(dinghy) DBG | Found IP lease: 10.211.55.32 for MAC address 001C4208D8F8
(dinghy) DBG |
(dinghy) Calling .GetSSHPort
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@10.211.55.32 -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/local/bin/ssh <nil>}
About to run SSH command:
if ! type netstat 1>/dev/null; then ss -tln; else netstat -tln; fi
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN

Watching the bootup sequence by opening the miniature window inside the parallel controll panel I see several errors and warning during the first bootup:

...
unable to write 'random state'
...
unable to write 'random state'
...
Device "eth1" does not exists.
...
unable to write 'random state'
...
unable to write 'random state'

Independent of this messages the machine works as expected until I stop and start it again.
If I doing this, I see only one warning during the bootup inside the parallels window:

warning: unable to find partition with the swap label (boot2dockerswap) or TYPE=swap (so Docker will likely complain about swap)
- this could also mean TCL already mounted it! (see 'free' or '/proc/swaps')

I have two macs (both already updated to macOS Catalina) with exact the same problem (after updating the system).

There is also an issue I posted at the dinghy repo, but the autor means the problem stuck inside docker-machine or this parallels connector:
codekitchen/dinghy#290

The text was updated successfully, but these errors were encountered:

KatSick · 2019-11-07T15:33:55Z

any updates on this? I have similar problem. before latest Catalina updates all went well, but now on come commands like docker-compose up I see: [1] 26388 abort docker-compose up and that's all

mediaessenz · 2019-11-07T15:41:46Z

@KatSick Maybe this helps: Homebrew/homebrew-core#45687 (comment)

romankulikov · 2019-11-09T07:46:36Z

@KatSick Maybe this helps: Homebrew/homebrew-core#45687 (comment)

@mediaessenz , does this workaround help for you?

mediaessenz · 2019-11-10T20:28:29Z

@KatSick Maybe this helps: Homebrew/homebrew-core#45687 (comment)

@mediaessenz , does this workaround help for you?

Yes

mediaessenz · 2019-11-15T12:12:25Z

An Update:
I got a brand new iMac yesterday and started to set up the system to my needs.
Because of the problems with restarting a docker machine I described here, I made the desicion not to use a timemachine backup of my old mac.
I installed only some basic stuff (browser, iterm, parallels desktop 15 and brew) before I used brew to install docker, docker-compose, docker-machine and docker-machine-parallels.
After this I created a new docker-machine and still have the same problem like before.
The machine comes up the first time without problems and end in the same error described up after stop and trying to start it again.
Also the messages shown in the parallels window I described up are the same.

mediaessenz · 2019-11-18T08:55:04Z

Am I really the ony one on this planet who have problems with using docker machine together with parallels desktop on macOS Catalina?

mediaessenz · 2019-11-18T09:53:37Z

After comparing the debug output of the first (working) start (create command) and the second (not working) start (start command) I found this different which may can help to identify the problem:

Start:

...
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@10.211.55.8 -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /usr/bin/sethostname dinghy && echo "dinghy" | sudo tee /var/lib/boot2docker/etc/hostname
SSH cmd err, output: <nil>: Setting hostname to dinghy Done.
dinghy

(dinghy) Calling .GetSSHHostname
(dinghy) DBG | executing: /usr/local/bin/prlctl list dinghy --output status --no-header
(dinghy) DBG | executing: /usr/local/bin/prlctl list -i dinghy
(dinghy) DBG | Found lease: 10.211.55.8 for MAC: 001C42CE0C5E, expiring at 1574071941, leased for 1800 s.
(dinghy) DBG |
(dinghy) DBG | Found IP lease: 10.211.55.8 for MAC address 001C42CE0C5E
(dinghy) DBG |
(dinghy) Calling .GetSSHPort
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@10.211.55.8 -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if ! type netstat 1>/dev/null; then ss -tln; else netstat -tln; fi
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
...

Start

...
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@10.211.55.6 -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if ! type netstat 1>/dev/null; then ss -tln; else netstat -tln; fi
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN

(dinghy) Calling .GetSSHHostname
(dinghy) DBG | executing: /usr/local/bin/prlctl list dinghy --output status --no-header
(dinghy) DBG | executing: /usr/local/bin/prlctl list -i dinghy
(dinghy) DBG | Found lease: 10.211.55.6 for MAC: 001C4295E032, expiring at 1574070104, leased for 1800 s.
(dinghy) DBG |
(dinghy) DBG | Found IP lease: 10.211.55.6 for MAC address 001C4295E032
(dinghy) DBG |
(dinghy) Calling .GetSSHPort
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@10.211.55.6 -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if ! type netstat 1>/dev/null; then ss -tln; else netstat -tln; fi
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
...

romankulikov · 2019-11-18T14:36:48Z

Well, it looks like the issue is in broken IP address reporting from virtual machine. I'm investigating it.

mediaessenz · 2019-11-25T12:50:37Z

Any news about this?

romankulikov · 2019-11-27T09:13:52Z

It looks like this is duplicate of #docker/machine#3595. For example, switching to older version of boot2docker works for me:

$ dinghy create --provider parallels --boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v18.06.1-ce/boot2docker.iso

romankulikov · 2019-11-27T12:13:21Z

It looks like this is duplicate of #docker/machine#3595. For example, switching to older version of boot2docker works for me:
$ dinghy create --provider parallels --boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v18.06.1-ce/boot2docker.iso

Well, now after more investigation I tend to reference this issue down to the lack of entropy at the start of virtual machine. It is described in #boot2docker/boot2docker#1322 (comment). Watching at /var/lib/boot2docker/log/docker.log inside virtual machine this message is printed at the start of dockerd:

crypto/rand: blocked for 60 seconds waiting to read random data from the kernel

–dockerd hangs at the start of the system which results in docker-machine create command failure (and consequent certificate issues).

Issue is reproduced on the recent boot2docker versions, like current 19.03.5, based on Linux kernel 4.14. Version 18.06.1 works because it is based on Linux kernel 4.9 where entropy pool state during VM boot is better.

mediaessenz · 2019-11-27T18:19:26Z

@romankulikov You are my HERO!!!
After switching to the older image my problems are gone!
Thank You very much for the energy you put into this issue!

iby · 2020-02-29T17:43:17Z

@romankulikov Sorry in advance if the question is dumb. I understand this is an issue with boot2docker image, but isn't this a show stopper? Aren't other drivers affected by this and if no, is there a way Parallels driver can adapt? If I understand correctly the referencing PR had something to do with fixing this, but the latest version (on Mojave) still shows the same behaviour (can create, cannot restart).

romankulikov · 2020-03-03T10:26:54Z

I've tried Parallels Desktop 15.1.2 and VirtualBox 6.1.4 with Boot2Docker v19.03.5–both work ok for me at the moment. At least in this case of creating and starting "dinghy" machine. @ianbytchek, can you please share your setup for me to reproduce the problem?

Speaking about the lack of entropy when starting the guest OS on the one hand it does look like a showstopper. On the other hand it doesn't look a like an easy thing to fix. On the third hand issue is currently addressed in modern Linux kernels:
https://lwn.net/Articles/808575/
https://git.kernel.org/linus/50ee7529ec4500c88f8664560770a7a1b65db72b

Not sure where to move forward.

romankulikov · 2020-03-03T11:13:31Z

Well, it looks like boot2docker starting from 19.03.5 has a backported patch from Linux kernel 5.4 with entropy fixes. So my picture of the problem is broken. And I need to know if one can reproduce the issue with recent boot2docker image.

mediaessenz · 2020-06-05T13:53:19Z

Unfortunately, at least for me, the problem still exists with the latest boot2docker image (19.03.5)

josefglatz · 2020-06-05T13:53:40Z

@romankulikov Tried it with boot2docker iso image version 19.03.5 and get still same error es @mediaessenz already mentioned after a dinghy stop && dinghy up:

Starting the dinghy VM...

Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded
/usr/local/Cellar/dinghy/4.6.5/cli/dinghy/system.rb:18:in `system': Failure calling `docker-machine start dinghy` (System::Failure)
	from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:126:in `system'
	from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:25:in `up'
	from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:271:in `start_services'
	from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:93:in `up'
	from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/command.rb:27:in `run'
	from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor.rb:359:in `dispatch'
	from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/base.rb:440:in `start'
	from /usr/local/bin/_dinghy_command:12:in `<main>'

legal90 · 2020-06-05T15:20:13Z

Well, it looks like boot2docker starting from 19.03.5 has a backported patch from Linux kernel 5.4 with entropy fixes.

boot2docker 19.03.5 was released from the earlier state, before that fix. That's why the issue still persists.

And, unfortunately, it seems there will be no releases anymore 😭 : boot2docker/boot2docker#1408
Together with docker/machine#4537, it looks like the sunset of the entire Docker Machine project.

romankulikov · 2020-06-24T13:59:55Z

boot2docker 19.03.5 was released from the earlier state, before that fix. That's why the issue still persists.

Yeah :-(

@legal90, how should we proceed with this issue? From my point it should be fixed only on guest OS (i.e. boot2docker) side. Is forking boot2docker an option?

legal90 · 2020-06-24T14:10:54Z

@romankulikov Building and releasing a custom boot2docker.iso might be an option, but in this case all users will have to specify the custom URL to it using --parallels-boot2docker-url flag. Let's see if there will be any fork continued by the community.

I asked here if there is any chance for the patch to be released: boot2docker/boot2docker#1403 (comment)

legal90 · 2020-06-30T21:53:46Z

v19.03.12, the final release of boot2docker was published today: https://github.com/boot2docker/boot2docker/releases/tag/v19.03.12

It includes the fix boot2docker/boot2docker#1403 and this issue should be solved there. I checked it on the test vm by doing docker-machine restart several times and it works as expected - no "Maximum number of retries (10) exceeded" errors anymore.
@mediaessenz, please, verify it in your setup

mediaessenz · 2020-07-01T19:43:53Z

YES, IT WORKS !

Thanks a lot to all involved people!

mediaessenz mentioned this issue Nov 6, 2019

dinghy halt && dinghy up result in error codekitchen/dinghy#290

Open

legal90 mentioned this issue Nov 27, 2019

Remove haveged in favor of backported upstream kernel commit boot2docker/boot2docker#1403

Merged

legal90 closed this as completed Jul 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Machine is not running properly after update to macOS Catalina #83

Docker Machine is not running properly after update to macOS Catalina #83

mediaessenz commented Nov 6, 2019 •

edited

KatSick commented Nov 7, 2019

mediaessenz commented Nov 7, 2019

romankulikov commented Nov 9, 2019

mediaessenz commented Nov 10, 2019

mediaessenz commented Nov 15, 2019

mediaessenz commented Nov 18, 2019

mediaessenz commented Nov 18, 2019

romankulikov commented Nov 18, 2019

mediaessenz commented Nov 25, 2019

romankulikov commented Nov 27, 2019

romankulikov commented Nov 27, 2019

mediaessenz commented Nov 27, 2019

iby commented Feb 29, 2020

romankulikov commented Mar 3, 2020

romankulikov commented Mar 3, 2020

mediaessenz commented Jun 5, 2020

josefglatz commented Jun 5, 2020

legal90 commented Jun 5, 2020 •

edited

romankulikov commented Jun 24, 2020

legal90 commented Jun 24, 2020 •

edited

legal90 commented Jun 30, 2020

mediaessenz commented Jul 1, 2020

Docker Machine is not running properly after update to macOS Catalina #83

Docker Machine is not running properly after update to macOS Catalina #83

Comments

mediaessenz commented Nov 6, 2019 • edited

KatSick commented Nov 7, 2019

mediaessenz commented Nov 7, 2019

romankulikov commented Nov 9, 2019

mediaessenz commented Nov 10, 2019

mediaessenz commented Nov 15, 2019

mediaessenz commented Nov 18, 2019

mediaessenz commented Nov 18, 2019

romankulikov commented Nov 18, 2019

mediaessenz commented Nov 25, 2019

romankulikov commented Nov 27, 2019

romankulikov commented Nov 27, 2019

mediaessenz commented Nov 27, 2019

iby commented Feb 29, 2020

romankulikov commented Mar 3, 2020

romankulikov commented Mar 3, 2020

mediaessenz commented Jun 5, 2020

josefglatz commented Jun 5, 2020

legal90 commented Jun 5, 2020 • edited

romankulikov commented Jun 24, 2020

legal90 commented Jun 24, 2020 • edited

legal90 commented Jun 30, 2020

mediaessenz commented Jul 1, 2020

mediaessenz commented Nov 6, 2019 •

edited

legal90 commented Jun 5, 2020 •

edited

legal90 commented Jun 24, 2020 •

edited