Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342

mike-potter · 2020-05-12T14:17:08Z

Description

After upgrading to Docker Desktop 2.3.0.2 for Mac OSX, local *.docksal sites are not accessible after restarting Docker Desktop. After doing a fin start for the project, a fin system reset command is required to get the sites working.

Steps to reproduce the issue:

Ensure running Docker Desktop 2.3.0.2
Restart Docker Desktop
In a project folder, use fin start to start the docker containers.
Try visiting the http://project.docksal site in browser. Does not respond, or gives connection reset error.
Run fin system reset
Now visit the site and you'll see it working.

Output of fin config:

fin config output

❯ fin config
---------------------
COMPOSE_PROJECT_NAME_SAFE: myproject
COMPOSE_FILE:
/Users/mpotter/.docksal/stacks/volumes-none.yml
/Users/mpotter/.docksal/stacks/stack-default.yml
/Users/mpotter/Projects/myproject/.docksal/docksal.yml
ENV_FILE:
/Users/mpotter/Projects/myproject/.docksal/docksal.env

PROJECT_ROOT: /Users/mpotter/Projects/myproject
DOCROOT: build/docroot
VIRTUAL_HOST: myproject.docksal
VIRTUAL_HOST_ALIASES: *.myproject.docksal
IP: 192.168.64.100
MYSQL: 192.168.64.100:32768

Docker Compose configuration
---------------------
services:
  cli:
    dns:
    - 192.168.64.100
    - 8.8.8.8
    environment:
      BLACKFIRE_CLIENT_ID: REDACTED
      BLACKFIRE_CLIENT_TOKEN: REDACTED
      COMPOSER_ALLOW_XDEBUG: '1'
      COMPOSER_DISABLE_XDEBUG_WARN: '1'
      DOCROOT: build/docroot
      DRUSH_ALLOW_XDEBUG: '1'
      DRUSH_OPTIONS_URI: http://myproject.docksal
      GIT_USER_EMAIL: REDACTED
      GIT_USER_NAME: REDACTED
      HOST_GID: '20'
      HOST_UID: '501'
      MYSQL_DATABASE: default
      MYSQL_HOST: db
      MYSQL_PASSWORD: REDACTED
      MYSQL_PORT: '3306'
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: REDACTED
      SECRET_ACAPI_EMAIL: null
      SECRET_ACAPI_KEY: null
      SECRET_PLATFORMSH_CLI_TOKEN: null
      SECRET_SSH_PRIVATE_KEY: null
      SECRET_TERMINUS_TOKEN: REDACTED
      SSH_AUTH_SOCK: /.ssh-agent/proxy-socket
      VIRTUAL_HOST: myproject.docksal
      XDEBUG_CONFIG: remote_connect_back=0 remote_host=192.168.64.1
      XDEBUG_ENABLED: '1'
    hostname: cli
    image: docksal/cli:2-php7.3
    labels:
      io.docksal.shell: bash
      io.docksal.user: docker
    volumes:
    - docksal_ssh_agent:/.ssh-agent:ro
    - cli_home:/home/docker:rw
    - project_root:/var/www:rw,nocopy,cached
  db:
    dns:
    - 192.168.64.100
    - 8.8.8.8
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: null
      MYSQL_DATABASE: default
      MYSQL_INITDB_SKIP_TZINFO: null
      MYSQL_ONETIME_PASSWORD: null
      MYSQL_PASSWORD: REDACTED
      MYSQL_RANDOM_ROOT_PASSWORD: null
      MYSQL_ROOT_PASSWORD: REDACTED
      MYSQL_USER: REDACTED
    hostname: db
    image: docksal/mysql:5.6-1.4
    ports:
    - 0:3306/tcp
    volumes:
    - db_data:/var/lib/mysql:rw
    - project_root:/var/www:ro,nocopy,cached
  web:
    depends_on:
      cli:
        condition: service_started
    dns:
    - 192.168.64.100
    - 8.8.8.8
    environment:
      APACHE_BASIC_AUTH_PASS: null
      APACHE_BASIC_AUTH_USER: null
      APACHE_DOCUMENTROOT: /var/www/build/docroot
      APACHE_FCGI_HOST_PORT: cli:9000
      MYSQL_DATABASE: default
      MYSQL_HOST: db
      MYSQL_PASSWORD: REDACTED
      MYSQL_PORT: '3306'
      MYSQL_USER: REDACTED
    hostname: web
    image: docksal/apache:2.4-2.3
    labels:
      io.docksal.cert-name: none
      io.docksal.permanent: "false"
      io.docksal.project-root: /Users/mpotter/Projects/myproject
      io.docksal.virtual-host: myproject.docksal,*.myproject.docksal,myproject.docksal.*
    volumes:
    - project_root:/var/www:ro,nocopy,cached
version: '2.1'
volumes:
  cli_home: {}
  db_data: {}
  docksal_ssh_agent:
    external: true
    name: docksal_ssh_agent
  project_root: {}
---------------------

Output of fin sysinfo:

fin sysinfo output

❯ fin sysinfo output
███  OS
Darwin Mac OS X 10.15.4
Darwin MacBook-Pro.local 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64

███  ENVIRONMENT
MODE : Docker Desktop

███  FIN
fin version: 1.93.0

███  DOCKER COMPOSE
EXPECTED VERSION: 1.24.1
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

███  DOCKER
EXPECTED VERSION: 19.03.4

Client: Docker Engine - Community
Version:           19.03.4
API version:       1.40
Go version:        go1.12.10
Git commit:        9013bf5
Built:             Fri Oct 18 15:48:38 2019
OS/Arch:           darwin/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
Version:          19.03.8
API version:      1.40 (minimum version 1.12)
Go version:       go1.12.17
Git commit:       afacb8b
Built:            Wed Mar 11 01:29:16 2020
OS/Arch:          linux/amd64
Experimental:     true
containerd:
Version:          v1.2.13
GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version:          1.0.0-rc10
GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version:          0.18.0
GitCommit:        fec3683
Kubernetes:
Version:          Unknown
StackAPI:         Unknown

███  DOCKSAL: PROJECTS
project             STATUS                   virtual host                                                project root
myproject           Up 8 minutes (healthy)   myproject.docksal,*.myproject.docksal,myproject.docksal.*   /Users/mpotter/Projects/myproject

███  DOCKSAL: VIRTUAL HOSTS
*.myproject.docksal
myproject.docksal.*
myproject.docksal

███  DOCKSAL: DNS
Successfully requested http://dns-test.docksal

███  DOCKER: RUNNING CONTAINERS
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                   PORTS                                               NAMES
20ddd9361e55        docksal/ssh-agent:1.3     "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)                                                       docksal-ssh-agent
9cc73bcab695        docksal/dns:1.1           "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)   127.0.0.1:32769->53/udp                             docksal-dns
c876ec3d7ce6        docksal/vhost-proxy:1.5   "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)   127.0.0.1:32777->80/tcp, 127.0.0.1:32776->443/tcp   docksal-vhost-proxy
02a65d329e82        docksal/apache:2.4-2.3    "httpd-foreground"       58 minutes ago      Up 8 minutes (healthy)   80/tcp, 443/tcp                                     myproject_web_1
d5048f92e71c        docksal/cli:2-php7.3      "/opt/startup.sh sup…"   58 minutes ago      Up 8 minutes (healthy)   22/tcp, 3000/tcp, 8080/tcp, 9000/tcp                myproject_cli_1
032ae42fcb0c        docksal/mysql:5.6-1.4     "docker-entrypoint.s…"   2 months ago        Up 8 minutes (healthy)   0.0.0.0:32768->3306/tcp                             myproject_db_1

███  DOCKER: NETWORKS
NETWORK ID          NAME                     DRIVER              SCOPE
ed5e46220f39        _default                 bridge              local
02eb22180d2a        bridge                   bridge              local
565cb66920ab        host                     host                local
eb62cb6479ca        none                     null                local
cc1281ee2ea2        myproject_default        bridge              local

███  DOCKER DESKTOP
EXPECTED VERSION: 2.0.0.3
2.3.0.2

███  HDD Usage
Filesystem                                                                          Size   Used  Avail Capacity     iused      ifree %iused  Mounted on
/dev/disk1s1                                                                       932Gi   10Gi  509Gi     2%      487536 9767490624    0%   /
devfs                                                                              228Ki  228Ki    0Bi   100%         788          0  100%   /dev
/dev/disk1s2                                                                       932Gi  409Gi  509Gi    45%     7242504 9760735656    0%   /System/Volumes/Data
/dev/disk1s5                                                                       932Gi  2.0Gi  509Gi     1%           2 9767978158    0%   /private/var/vm
map auto_home                                                                        0Bi    0Bi    0Bi   100%           0          0  100%   /System/Volumes/Data/home

The text was updated successfully, but these errors were encountered:

lmakarov · 2020-05-12T19:34:16Z

Looks like yet another regression from upstream... docker/for-mac#4541

crud42 · 2020-05-13T07:39:23Z

I can confirm here that things are working again after updating to current docker Desktop for Mac Version 2.3.0.2.

Upgraded to Docker Desktop 2.3.0.2
Applied fin update
Applied fin system reset
Cleared all browser caches (chrome can get you in trouble with this) and restart browser
Applied fin project restart

mike-potter · 2020-05-13T13:08:02Z

@42robotsCS to test this specific issue you need to perform your steps above, but then restart the Docker Desktop. Follow the steps given in the OP. The fin system reset is needed after each Docker Desktop restart. This has now been confirmed by several of my co-workers.

@lmakarov yes, I meant to post that link also. I agree that is looks very much like still a problem upstream. At least in this case there is a workaround for it in Docksal.

lmakarov · 2020-05-13T19:36:02Z

I've looked at what's going on with the new Docker Desktop. Below are my findings.

After the upgrade from upgrade from Docker Desktop 2.1.x.x to 2.3.0.2, docksal-vhost-proxy and docksal-dns system service are down:

$ fin docker ps --all --filter "label=io.docksal.group=system"
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                        PORTS                                                    NAMES
bdbd74a004ef        docksal/ssh-agent:1.3     "docker-entrypoint.s…"   2 hours ago         Up 25 minutes (healthy)                                                                docksal-ssh-agent
63fe50d9404f        docksal/dns:1.1           "docker-entrypoint.s…"   2 hours ago         Exited (255) 25 minutes ago   192.168.64.100:53->53/udp                                docksal-dns
379b7ec3d3c2        docksal/vhost-proxy:1.5   "docker-entrypoint.s…"   2 hours ago         Exited (255) 25 minutes ago   192.168.64.100:80->80/tcp, 192.168.64.100:443->443/tcp   docksal-vhost-proxy

Checking why:

$ fin docker inspect --format='{{json .State.Error}}' docksal-vhost-proxy
"driver failed programming external connectivity on endpoint docksal-vhost-proxy (b2b26ec1e94d7f77360c0c7cb93be52f5c148b9246e0350d52ed28c855536cfd): Error starting userland proxy: listen tcp 192.168.64.100:443: bind: cannot assign requested address"

$ fin docker inspect --format='{{json .State.Error}}' docksal-dns
"driver failed programming external connectivity on endpoint docksal-dns (06f3c22dd6f3f05b89934339afaec061633acd2c685ce4e7b8584c7979e33c36): Error starting userland proxy: listen udp 192.168.64.100:53: bind: cannot assign requested address"

That ^ is exact what #1268 was about.

Resetting and rechecking system services:

$ fin system reset
Configuring network settings...
Configuring NFS shares...
NFS shares are already configured
Resetting Docksal system services...
 * proxy
 * dns
   upstream 8.8.8.8
Enabling automatic *.docksal DNS resolver...
Clearing DNS cache...
Password:
 * ssh-agent
Identity added: id_ecdsa (id_ecdsa)
Identity added: id_rsa (id_rsa)

$ fin docker ps --all --filter "label=io.docksal.group=system"
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                    PORTS                                               NAMES
fe5f97e02763        docksal/ssh-agent:1.3     "docker-entrypoint.s…"   6 seconds ago       Up 6 seconds (healthy)                                                        docksal-ssh-agent
07ccea062af2        docksal/dns:1.1           "docker-entrypoint.s…"   15 seconds ago      Up 14 seconds (healthy)   127.0.0.1:32768->53/udp                             docksal-dns
8cce9ef0dfac        docksal/vhost-proxy:1.5   "docker-entrypoint.s…"   15 seconds ago      Up 15 seconds (healthy)   127.0.0.1:32771->80/tcp, 127.0.0.1:32770->443/tcp   docksal-vhost-proxy

Note how the PORTS section now reads 127.0.0.1:32771->80/tcp, 127.0.0.1:32770->443/tcp instead of 192.168.64.100:80->80/tcp, 192.168.64.100:443->443/tcp for docksal-vhost-proxy.

Trying to talk to vhost-proxy:

# Standard Docksal IP
$ curl -I http://192.168.64.100
curl: (52) Empty reply from server

# Localhost (Mac), just in case
$ curl -I http://127.0.0.1
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

Starting any Docksal project, then checking vhost-proxy again:

$ fin project start
...

$ curl -I http://192.168.64.100
curl: (52) Empty reply from server

Resetting docksal-vhost-proxy with a project running:

$ fin system reset
...

$ curl -I http://192.168.64.100
HTTP/1.1 404 Not Found
Server: openresty/1.15.8.2
Date: Wed, 13 May 2020 18:36:43 GMT
Content-Type: text/html
Connection: keep-alive

Success! Finally able to connect to vhost-proxy from the host.

Stopping the project (all active projects), then resetting the vhost-proxy and checking:

$ fin project stop --all
$ fin system reset
$ curl -I http://192.168.64.100
curl: (52) Empty reply from server

We are back to square one with vhost-proxy not working.

It looks like vhost-proxy is only accessible from the host when it is connected to at least one project network.

Creating a bridge network manually and connecting vhost-proxy to it does not help:

$ fin docker network create docksal-vhost-proxy
$ fin docker network connect docksal-vhost-proxy docksal-vhost-proxy
$ curl -I http://192.168.64.100
curl: (52) Empty reply from server

lmakarov · 2020-05-13T19:43:12Z

Here's a workaround for the time being for this issue (and actually, for #1268 on Docker desktop 2.2.x.x versions):

$ fin config set --global DOCKSAL_VHOST_PROXY_IP=0.0.0.0
$ fin system reset vhost-proxy

This removes the IP binding, so vhost-proxy will be exposed on all local interfaces.

With this in place, no additional steps are required after restarting Docker Desktop.

mike-potter · 2020-05-14T14:09:33Z

Hmm, this didn't work for me. I did the two commands mentioned above. Then I restarted Docker Desktop and did a fin start in my project. It still did not respond in the browser. I still had to do a fin system reset to get it working.

I'll try to get time to do more debugging using some of the other commands you used above to try and determine why this didn't work for me.

lmakarov · 2020-05-14T17:16:29Z

@mike-potter There is one more issue that gets in the way here - the docksal-dns container always binds to an IP and there is no way now to tell it bind to 0.0.0.0 I'm thinking of doing a hotfix release to address that.

lmakarov · 2020-05-14T23:56:45Z

Should be fixed in v1.13.3 (just released).

mike-potter mentioned this issue May 12, 2020

Updating to Docker Desktop 2.2.0.0 breaks Docksal #1268

Closed

lmakarov self-assigned this May 12, 2020

lmakarov added the 🏷bug label May 12, 2020

lmakarov added this to To do in 1.14.0 via automation May 12, 2020

lmakarov mentioned this issue May 13, 2020

Add 'expose Docksal to the outer world' to the docs #1303

Open

lmakarov mentioned this issue May 14, 2020

Release 1.13.3 #1345

Merged

lmakarov removed this from To do in 1.14.0 May 14, 2020

lmakarov added this to Done in 1.13.3 May 14, 2020

lmakarov closed this as completed May 14, 2020

lmakarov mentioned this issue Mar 9, 2021

docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342

Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342

mike-potter commented May 12, 2020

lmakarov commented May 12, 2020

crud42 commented May 13, 2020 •

edited

mike-potter commented May 13, 2020

lmakarov commented May 13, 2020

lmakarov commented May 13, 2020 •

edited

mike-potter commented May 14, 2020

lmakarov commented May 14, 2020

lmakarov commented May 14, 2020

Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342

Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342

Comments

mike-potter commented May 12, 2020

lmakarov commented May 12, 2020

crud42 commented May 13, 2020 • edited

mike-potter commented May 13, 2020

lmakarov commented May 13, 2020

lmakarov commented May 13, 2020 • edited

mike-potter commented May 14, 2020

lmakarov commented May 14, 2020

lmakarov commented May 14, 2020

crud42 commented May 13, 2020 •

edited

lmakarov commented May 13, 2020 •

edited