Latest docker release (20.10.0) doesn't work with docker-machine

[joelgriffith]

Hey there! I've noticed that the latest docker release (20.10.0 as of this writing) breaks with docker-machine's create phase. The following no longer works:

$ docker-machine create --driver digitalocean --digitalocean-access-token REDACTED --engine-storage-driver overlay2 --digitalocean-image ubuntu-18-04-x64 --digitalocean-size s-1vcpu-1gb --digitalocean-region sfo2 dm-create-test
Running pre-create checks...
Creating machine...
(dm-create-test) Creating SSH key...
(dm-create-test) Creating Digital Ocean droplet...
(dm-create-test) Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

This has failed for about the last few hours. Seems like pinning to a custom engine URL works, but I imagine most folks will use this with a broken experience across all drivers. I've currently worked around this by setting:

--engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh"

[joelgriffith]

https://get.docker.com/ is the default install URL, fwiw

[demahum]

Interestingly, I just tried with::

docker-machine create --driver google --google-project blablabla --google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20201201 anyname

And it worked fine.

After doing docker-machine ssh anyname, I get:

sudo docker version
Client: Docker Engine - Community
 Version:           20.10.0
...
Server: Docker Engine - Community
 Engine:
  Version:          20.10.0

I tried with 18, 19 and 20 local Docker versions. Docker machine is 0.16.0.

Do you see anything I'm missing (except that you used DO and I used GCP)?

[joelgriffith]

Thanks for checking this out. I haven't tried with other providers, just DO, so the problem might be there. Will check again later

[joelgriffith]

Looks like, according to JonasProgrammer/docker-machine-driver-hetzner#54, that Hetzner doesn't work as well.

[cardoe]

Interestingly, I just tried with::

docker-machine create --driver google --google-project blablabla --google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20201201 anyname

And it worked fine.

After doing docker-machine ssh anyname, I get:

sudo docker version
Client: Docker Engine - Community
 Version:           20.10.0
...
Server: Docker Engine - Community
 Engine:
  Version:          20.10.0

Querying the docker version will work but attempt to run a container on that box now and it will fail.

[demahum]

@cardoe, not really:

$ docker-machine ssh anothertest
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1030-gcp x86_64)
...
Last login: Wed Dec  9 18:10:01 2020 from 34.90.202.113
docker-user@anothertest:~$ sudo docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

docker-user@anothertest:~$

[pierotofy]

Confirm, latest update broke docker-machine create.

[lcampanis]

Confirm as well

[xandrox]

Me too, my workaround for gitlab runner and AWS driver was also to use engine-install-url with a fixed version script.

[sebastien-gallet]

docker-machine -D create ... --driver ...
-D enable debug mode, so we can see where it breaks.

In the case of rackspace (and I suspect digitalocean as well), we discovered that the system was not allowing external connections to the docker service because the firewall was blocking the port 2376.

As a workaround we created an image with an additional firewall rule to allow 2376 on an external interface.
Using that image, we were able to create a machine successfully.

Best would be to have the provision files updated with the proper fixes.

[GeorgFleig]

In my case (Gitlab runner at DigitalOcean, Ubuntu 20.04 LTS) the update to Docker 20.10.0 broke the runners as well. After a bit of debugging it seemed to me as if the docker daemon was only listening on the socket, not the port, when docker-machine started checking for the running docker daemon. According to the logs, half a minute after docker-machine gave up (10 retries), the docker daemon was restarted and also listened on port 2376.

I connected locally to the docker daemon and was able to run containers.

Haven't investigated further and just changed to Docker 19.03 with the parameter suggested bei @joelgriffith which fixes the issue for now.

[thaJeztah]

This may be related to moby/moby#41767 (systemctl docker start get stuck in cloud-init), for which we'll be publishing updated packages