docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

charginghawk · 2021-03-09T18:57:56Z

Description

After updating to Docker for Mac 3.2, everyone on my team has started getting this error:

docker: Error response from daemon: driver failed programming external connectivity on endpoint docksal-dns: Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use.
 ERROR:  Failed starting the DNS service.

Fix seems to be downgrading or adding DOCKSAL_DNS_DISABLED=1 to docksal-local.env (though this will cause local .docksal domains to not resolve).

Steps to reproduce the issue:

Update to Docker for Mac 3.2
fin system start
Get error

Describe the results you received:

docker: Error response from daemon: driver failed programming external connectivity on endpoint docksal-dns: Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use.
 ERROR:  Failed starting the DNS service.

Describe the results you expected:

System starts normally.

Output of fin config:

fin config output

# Note that fin config fails without the DOCKSAL_DNS_DISABLED=1 fix.
% fin config                    
---------------------
COMPOSE_PROJECT_NAME_SAFE: foobar
COMPOSE_FILE:
/Users/foobar/.docksal/stacks/volumes-nfs.yml
/Users/foobar/Projects/foobar/.docksal/docksal.yml
/Users/foobar/Projects/foobar/.docksal/docksal-local.yml
ENV_FILE:
/Users/foobar/Projects/foobar/.docksal/docksal.env
/Users/foobar/Projects/foobar/.docksal/docksal-local.env

PROJECT_ROOT: /Users/foobar/Projects/foobar
DOCROOT: docroot
VIRTUAL_HOST: foobar.docksal
VIRTUAL_HOST_ALIASES: *.foobar.docksal
IP: 192.168.64.100
MYSQL:

Docker Compose configuration
---------------------
services:
  cli:
    build:
      context: /Users/foobar/Projects/foobar/.docksal/cli
    dns:
    - 8.8.8.8
    - 9.9.9.9
    - 8.8.8.8
    - 9.9.9.9
    environment:
      HOST_GID: '1053198925'
      HOST_UID: '1786867176'
      PHP_IDE_CONFIG: serverName=foobar.docksal
      XDEBUG_CONFIG: idekey=PHPSTORM remote_host=192.168.64.1
      XDEBUG_ENABLED: '1'
    extra_hosts:
      federated-search.docksal: 192.168.64.100
    hostname: cli
    volumes:
    - docksal_ssh_agent:/.ssh-agent:ro
    - project_root:/var/www:rw,nocopy,cached
  db:
    dns:
    - 8.8.8.8
    - 9.9.9.9
    - 8.8.8.8
    - 9.9.9.9
    environment:
      MYSQL_DATABASE: default
      MYSQL_PASSWORD: user
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: user
    hostname: db
    image: docksal/db:1.0-mysql-5.7
    ports:
    - published: 0
      target: 3306
  mail:
    dns:
    - 8.8.8.8
    - 9.9.9.9
    healthcheck:
      interval: 10s
    hostname: mail
    image: mailhog/mailhog
    labels:
      io.docksal.cert-name: none
      io.docksal.virtual-host: mail.foobar.docksal,mail.foobar.docksal.*
      io.docksal.virtual-port: '8025'
    logging:
      options:
        max-file: '10'
        max-size: 1m
    volumes:
    - project_root:/var/www:ro,nocopy,cached
  memcached:
    dns:
    - 8.8.8.8
    - 9.9.9.9
    - 8.8.8.8
    - 9.9.9.9
    environment:
      MEMCACHED_MEMORY_LIMIT: '1024'
    hostname: memcached
    image: memcached
    ports:
    - target: 11211
  solr:
    hostname: solr
    image: docksal/solr:1.0-solr4
    labels:
      io.docksal.virtual-host: solr.foobar.docksal
      io.docksal.virtual-port: '8983'
    ports:
    - published: 8983
      target: 8983
    volumes:
    - /Users/foobar/Projects/foobar/.docksal/etc/solr/conf:/var/lib/solr/conf:ro
  web:
    depends_on:
      cli:
        condition: service_started
    dns:
    - 8.8.8.8
    - 9.9.9.9
    - 8.8.8.8
    - 9.9.9.9
    environment:
      APACHE_DOCUMENTROOT: /var/www/docroot
      APACHE_FILE_PROXY: https://www.foobar.com
      foobar_LEGACY_BASE_URL: https://local.foobar.com
    extra_hosts:
      federated-search.docksal: 192.168.64.100
    hostname: web
    image: docksal/web:apache2.2
    labels:
      io.docksal.project-root: /Users/foobar/Projects/foobar
      io.docksal.virtual-host: foobar.docksal,*.foobar.docksal,local.d8.foobar.com,192.168.0.182
    volumes:
    - project_root:/var/www:ro
version: '2.1'
volumes:
  cli_home: {}
  db_data: {}
  docksal_ssh_agent:
    external: true
    name: docksal_ssh_agent
  project_root:
    driver: local
    driver_opts:
      device: :/Users/foobar/Projects/foobar
      o: addr=192.168.64.1,vers=3,nolock,noacl,nocto,noatime,nodiratime,actimeo=1
      type: nfs

---------------------

Output of fin sysinfo:

fin sysinfo output

% fin sysinfo               

███  FIN
fin version: 1.103.0

███  OS
Darwin Mac OS X 10.15.7
Darwin L20541MUS.local 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64

███  ENVIRONMENT
MODE : Docker Desktop
DOCKER_HOST : 
DOCKSAL_NFS_PATH : /Users

NFS EXPORTS:

# <ds-nfs docksal
/Users 192.168.64.1 192.168.64.100 -alldirs -maproot=0:0
# ds-nfs>

███  DOCKER
EXPECTED CLIENT VERSION: 19.03.13
EXPECTED SERVER VERSION: 19.03.13

Client: Docker Engine - Community
Version:           19.03.13
API version:       1.40
Go version:        go1.13.15
Git commit:        4484c46d9d
Built:             Wed Sep 16 16:58:31 2020
OS/Arch:           darwin/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
Version:          20.10.5
API version:      1.41 (minimum version 1.12)
Go version:       go1.13.15
Git commit:       363e9a8
Built:            Tue Mar  2 20:15:47 2021
OS/Arch:          linux/amd64
Experimental:     false
containerd:
Version:          1.4.3
GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version:          1.0.0-rc92
GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version:          0.19.0
GitCommit:        de40ad0

███  DOCKER COMPOSE
EXPECTED VERSION: 1.27.4
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

███  DOCKSAL: PROJECTS
project                    STATUS                      virtual host                                                                          project root
foobar                  Exited (255) 5 hours ago    foobar.docksal,*.foobar.docksal,local.d8.foobar.com,192.168.0.182                 /Users/foobar/Projects/foobar
portal                     Exited (255) 6 weeks ago    portal.docksal,*.portal.docksal,local.portal.foobar.com                                 /Users/foobar/Projects/portal
federated-search-service   Exited (255) 8 days ago     federated-search.docksal,*.federated-search.docksal,local.federated-search.foobar.com   /Users/foobar/Projects/federated-search-service
2-foobar                Exited (255) 4 months ago   2-foobar.docksal,*.2-foobar.docksal,2-local.d8.foobar.com                         /Users/foobar/Projects/2-foobar

███  DOCKSAL: VIRTUAL HOSTS
No virtual hosts found

███  DOCKSAL: NETWORKING

DOCKSAL_IP: 192.168.64.100
DOCKSAL_HOST_IP: 192.168.64.1
DOCKSAL_VHOST_PROXY_IP: 0.0.0.0
DOCKSAL_DNS_IP: 0.0.0.0
DOCKSAL_DNS_DISABLED: 0
DOCKSAL_NO_DNS_RESOLVER: 0
DOCKSAL_DNS_UPSTREAM: 
DOCKSAL_DNS_DOMAIN: docksal

███  DOCKSAL: CONNECTIVITY

Host to 192.168.64.100:   PASS
Container to 192.168.64.100:  PASS
Container to 192.168.64.1:  PASS

Checking connectivity to http://dns-test.docksal...
Host: FAIL

Debug info:
----------
+ cat /etc/resolv.conf
+ grep 192.168.64.100
+ ping -c 1 -W 1 dns-test.docksal
ping: cannot resolve dns-test.docksal: Unknown host
+ nslookup -timeout=1 dns-test.docksal 192.168.64.100
;; connection timed out; no servers could be reached

+ set +x
----------

Containers: FAIL

███  DOCKER: RUNNING CONTAINERS
CONTAINER ID        IMAGE                     COMMAND                  CREATED              STATUS                        PORTS                                      NAMES
2030d15ee129        docksal/vhost-proxy:1.6   "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   docksal-vhost-proxy
b5c68523fd42        docksal/ssh-agent:1.3     "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes (healthy)                                                   docksal-ssh-agent

███  DOCKER: NETWORKS
NETWORK ID          NAME                               DRIVER              SCOPE
dd1f9cb71bac        2-foobar_default                bridge              local
52c4bd4de3ff        _default                           bridge              local
05ed7bc26ef5        accounts_default                   bridge              local
674e0be66f7d        bridge                             bridge              local
12dc4e2b8716        federated-search-service_default   bridge              local
29742dd446e1        host                               host                local
4f191baac94d        none                               null                local
9cfa87086106        foobar_default                  bridge              local
5562b6e83608        portal_default                     bridge              local

███  DOCKER DESKTOP
EXPECTED VERSION: 2.5.0.1
DETECTED VERSION: 3.2.1

███  HDD Usage
Filesystem      Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5   466Gi   11Gi  171Gi     6%  488372 4881964508    0%   /
devfs          186Ki  186Ki    0Bi   100%     643          0  100%   /dev
/dev/disk1s1   466Gi  279Gi  171Gi    63% 4715499 4877737381    0%   /System/Volumes/Data
/dev/disk1s4   466Gi  5.0Gi  171Gi     3%       5 4882452875    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
map -fstab       0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/Network/Servers
/dev/disk1s3   466Gi  504Mi  171Gi     1%      54 4882452826    0%   /Volumes/Recovery

The text was updated successfully, but these errors were encountered:

danshumaker · 2021-03-09T19:44:43Z

I too am getting this error after the upgrade to 3.2.1.

sean-e-dietrich · 2021-03-09T21:16:33Z

We received this issue the current way we found around it was the following:

fin config set --global DOCKSAL_DNS_IP=192.168.64.100
fin system reset

lmakarov · 2021-03-09T22:34:02Z

The good news

NetworkSettings are incorrect/unusable when binding on specific IPs docker/for-mac#4541 was finally fixed upstream, which means Updating to Docker Desktop 2.2.0.0 breaks Docksal #1268 and Need to use "fin system reset" with new Docker Desktop 2.3.0.2 for Mac #1342 no longer need a workaround (binding to 0.0.0.0 instead of 192.168.64.100)

With the latest Docker Desktop v3.2.1 (61626), biding to a specific IP works again:

fin config set --global DOCKSAL_VHOST_PROXY_IP=192.168.64.100
fin config set --global DOCKSAL_DNS_IP=192.168.64.100
fin system reset

The bad news

binding to 0.0.0.0:53/udp is now broken
last year's workaround for Docker Desktop v2.2.0.0+ has to be rolled back. Binding docksal-dns to a specific IP is now required.

Use the approach suggested by @sean-e-dietrich above until we rollout a new/hotfix release.

Alternatively (and recommended), disable docksal-dns altogether and use the docksal.site base domain for projects:

fin config set --global DOCKSAL_DNS_DISABLED=1
fin system reset

Then, access projects at http://myproject.docksal.site

alejodaraio · 2021-03-10T16:30:22Z

Thanks @lmakarov ur workaround works like a charm !

crud42 · 2021-03-11T08:31:33Z

Applied the workaround/solution by @sean-e-dietrich and @lmakarov but i don't get any connection calling http://site.docksal.
Not sure if the ip binding is correct - is 192.168.64.100 only by example or do i miss something?

I am on Docker Desktop v3.2.1 (61626) and docksal/cli:2.12-php7.3

If i remove proxy settings from global docksal.env and restart, it works but the error message remains in the console
(Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use.)

Current global docksal.env:

DOCKER_NATIVE=1
DOCKSAL_UUID=188da947-46fe-ba6d-1b3c-296916a9a3fc
DOCKSAL_DNS_DISABLED="0"

Best

alejodaraio · 2021-03-11T13:17:52Z

Applied the workaround/solution by @sean-e-dietrich and @lmakarov but i don't get any connection calling http://site.docksal.
Not sure if the ip binding is correct - is 192.168.64.100 only by example or do i miss something?

I am on Docker Desktop v3.2.1 (61626) and docksal/cli:2.12-php7.3

If i remove proxy settings from global docksal.env and restart, it works but the error message remains in the console
(Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use.)

Current global docksal.env:

DOCKER_NATIVE=1
DOCKSAL_UUID=188da947-46fe-ba6d-1b3c-296916a9a3fc
DOCKSAL_DNS_DISABLED="0"

Best

You need to use http://YOUR_SITE.docksal.site with the @lmakarov solution.

crud42 · 2021-03-11T14:42:55Z

@alejodaraio @lmakarov I reproduced all the steps above, but i don't get my projects up and running.
I guess i have to go for downgrading for now.
Maybe local settings in env and yaml interfere, but changing/removing those is not an option right now.

crud42 · 2021-03-12T10:07:30Z

Just to mention the current issue over at docker/roadmap#183
The introduced auto update in Docker Desktop (without an option to disable it) is more or less a reason why this happened so unexpectedly at all.

crud42 · 2021-03-16T07:52:37Z

@lmakarov it looks like the issue is resolved also in Docker for Mac as mentioned in docker/for-win#10601 (comment) 👍 Should work, could not test yet.

crud42 · 2021-03-16T08:03:32Z

Hi all, i can confirm that things are up and running again with latest release for docker desktop version 3.2.2 (61853) !

danshumaker · 2021-03-16T12:10:22Z

Thanks everyone for your hard work! 3.2.2 works for me without any other changes.

lmakarov · 2021-03-18T23:58:02Z

Closing, since the issue was resolved upstream => upgrade to Docker Desktop 3.2.2 (61853).

robcast mentioned this issue Mar 18, 2021

ARM64 architecture support in Docksal #1421

Closed

10 tasks

lmakarov closed this as completed Mar 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

charginghawk commented Mar 9, 2021

danshumaker commented Mar 9, 2021

sean-e-dietrich commented Mar 9, 2021

lmakarov commented Mar 9, 2021

alejodaraio commented Mar 10, 2021

crud42 commented Mar 11, 2021 •

edited

alejodaraio commented Mar 11, 2021

crud42 commented Mar 11, 2021 •

edited

crud42 commented Mar 12, 2021 •

edited

crud42 commented Mar 16, 2021 •

edited

crud42 commented Mar 16, 2021

danshumaker commented Mar 16, 2021

lmakarov commented Mar 18, 2021

docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

docksal-dns "address already in use" error after updating to Docker for Mac 3.2 #1471

Comments

charginghawk commented Mar 9, 2021

danshumaker commented Mar 9, 2021

sean-e-dietrich commented Mar 9, 2021

lmakarov commented Mar 9, 2021

alejodaraio commented Mar 10, 2021

crud42 commented Mar 11, 2021 • edited

alejodaraio commented Mar 11, 2021

crud42 commented Mar 11, 2021 • edited

crud42 commented Mar 12, 2021 • edited

crud42 commented Mar 16, 2021 • edited

crud42 commented Mar 16, 2021

danshumaker commented Mar 16, 2021

lmakarov commented Mar 18, 2021

crud42 commented Mar 11, 2021 •

edited

crud42 commented Mar 11, 2021 •

edited

crud42 commented Mar 12, 2021 •

edited

crud42 commented Mar 16, 2021 •

edited