Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ping -4 works but ping does not work. -> vpnkit dns: extremely slow, with internal unexpected error from dnssd: Timeout (-65568) #5516

Closed
3 tasks done
jjqq2013 opened this issue Mar 29, 2021 · 15 comments

Comments

@jjqq2013
Copy link

jjqq2013 commented Mar 29, 2021

  • I have tried with the latest version of Docker Desktop
  • I have tried disabling enabled experimental features
  • I have uploaded Diagnostics
  • Diagnostics ID: F66AAB1D-A0F9-452D-A643-4C613333DA3C/20210329013206

Expected behavior

DNS queries get response quickly

Actual behavior

DNS queries are very slow

Information

  • macOS Version: macOS Catalina 10.15.7 (19H524)
  • Docker Desktop Version: 3.2.2(61853)

Steps to reproduce the behavior

  1. Connect to my company via "Cisco AnyConnect Secure Mobility Client" 4.8.03052
  2. run ping A_FQDN_IN_VPN or nslookup A_FQDN_IN_VPN, in any docker container such as
```
docker run alpine ping A_FQDN_IN_VPN
docker run ubuntu ping A_FQDN_IN_VPN
...
```
  1. The ping A_FQDN_IN_VPN always cause 15 seconds to show the IP. The ping -4 A_FQDN_IN_VPN will immediately show IP. The nslookup A_FQDN_IN_VPN cause 5 seconds with following answer:
```
/ # nslookup A_FQDN_IN_VPN
Server:   192.168.65.5
Address:  192.168.65.5:53

Non-authoritative answer:
Name: A_FQDN_IN_VPN
Address: 100.79.222.2

*** Can't find A_FQDN_IN_VPN: No answer
```  
  1. I have posted the detail strace to Built-in DNS server extremely slow for large responses #4430 (comment), turns out the sendto and recvfrom syscall were very slow(5 seconds).
  2. The Docker internal log is here:
==> /Users/user_name/Library/Containers/com.docker.docker/Data/log/host/com.docker.backend.log <==
time="2021-03-29T10:16:15+09:00" level=error msg="unexpected error from dnssd: Timeout (-65568)"
goroutine 42834 [running]:
github.com/docker/pinata/common/pkg/dns.lookupOne(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006d0006, 0xc0005ea300, 0x26, 0x5615320, 0xc00096ecf0, 0x5615320)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup_darwin.go:26 +0x20e
github.com/docker/pinata/common/pkg/dns.doLookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x6, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x0, ...)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:90 +0x21a
github.com/docker/pinata/common/pkg/dns.distinguishNodataNxdomain(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006dfb40, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x5613760)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:58 +0x67
github.com/docker/pinata/common/pkg/dns.lookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x1c, 0xc000185680, 0x0, 0xc0006dfc78, 0x40533ac, 0x18)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:53 +0x138
github.com/docker/pinata/common/pkg/dns.(*SystemResolver).Answer(0x6210f00, 0xc000b62090, 0x25, 0xc00001001c, 0x0, 0x0, 0x0, 0x5613760, 0xc0001a3a28)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/system.go:20 +0x172
github.com/docker/pinata/common/pkg/dns.createReply(0xc000300300, 0x2, 0x2, 0xc0005f2990, 0x55f7946)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:164 +0x23d
github.com/docker/pinata/common/pkg/dns.NewServer.func1(0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:35 +0x4a
github.com/miekg/dns.HandlerFunc.ServeDNS(0xc000300340, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:37 +0x44
github.com/miekg/dns.(*ServeMux).ServeDNS(0xc000300320, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/serve_mux.go:103 +0x5d
github.com/miekg/dns.(*Server).serveDNS(0xc00024a000, 0xc000599000, 0x36, 0x200, 0xc000ab6380)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:650 +0x2fd
github.com/miekg/dns.(*Server).serveUDPPacket(0xc00024a000, 0xc00046a9b0, 0xc000599000, 0x36, 0x200, 0x5652458, 0xc0007deb60, 0x0, 0x5620910, 0xc000968b40)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:590 +0xed
created by github.com/miekg/dns.(*Server).serveUDP
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:520 +0x395
  1. If I change the /etc/resolv.conf in the container to make it same as the host side, then everything works well.

Additional information: I've managed to dump to DNS UDP data and time, note the timestamp on right side, the timestamp of first response is 5 seconds later of the request:
Screen Shot 2021-03-29 at 11 52 00

BTW, my Macbook has installed "Symantec Endpoint Protection" (an anti-virus software).

Other notable things:

  • On Windows PC, there is no such issue.
  • ping an VPN IP from a container always fails, seems like some data are blocked.
    EDIT: sorry fo confusing, after restart Docker, ping IP works, but ping fqdn still does not work.
  • all host side dns servers can not be pinged from the container but can be pinged in the host.
    EDIT: sorry fo confusing, after restart Docker, all host side dns servers can be pinged.
@jjqq2013 jjqq2013 changed the title vpnkit dns: unexpected error from dnssd: Timeout (-65568) vpnkit dns: extremely slow, with internal log unexpected error from dnssd: Timeout (-65568) Mar 29, 2021
@jjqq2013 jjqq2013 changed the title vpnkit dns: extremely slow, with internal log unexpected error from dnssd: Timeout (-65568) vpnkit dns: extremely slow, with internal unexpected error from dnssd: Timeout (-65568) Mar 29, 2021
@jjqq2013
Copy link
Author

jjqq2013 commented Mar 30, 2021

I have narrowed down this issue to the Docker’s built-in DNS server not responding to DNS AAAA query at all.

Here are the proof:

  • Some host name are pingable, such as ping host.vpn.domain2. The nslookup completed two querie(Query 0 and Query 1), one for type=A, another for type=AAAA, though no IPv6 record.
$__ docker run -it --rm alpine nslookup -debug host.vpn.domain2
Server:   192.168.65.5
Address:  192.168.65.5:53

Query #0 completed in 1ms:
Non-authoritative answer:
Name: host.vpn.domain2
Address: 10.6.16.233

Query #1 completed in 4ms:
Non-authoritative answer:
*** Can't find host.vpn.domain2: No answer
  • Some host name are not pingable, such as ping host.vpn.domain1, but ping -4 host.vpn.domain1 works. The nslookup just completed one query (type=A), no response for type=AAAA. The nslookup -type=A host.vpn.domain1 works
$__ docker run -it --rm alpine nslookup -debug host.vpn.domain1
Server:   192.168.65.5
Address:  192.168.65.5:53

Query #0 completed in 2ms:
Non-authoritative answer:
Name: host.vpn.domain1
Address: 100.79.28.134


*** Can't find host.vpn.domain1: No answer

Note that it did not show "Query 1".

$__ docker run -it --rm alpine nslookup -debug -type=A host.vpn.domain1
Server:   192.168.65.5
Address:  192.168.65.5:53

Query #0 completed in 2ms:
Non-authoritative answer:
Name: host.vpn.domain1
Address: 100.79.28.134
$__ docker run -it --rm alpine nslookup -debug -type=AAAA host.vpn.domain1
;; connection timed out; no servers could be reached
$__ docker run -it --rm alpine nslookup -debug -type=ANY host.vpn.domain1
;; connection timed out; no servers could be reached

Each failed ping or nslookup will cause following error log "unexpected error from dnssd: Timeout (-65568)"

==> /Users/user_name/Library/Containers/com.docker.docker/Data/log/host/com.docker.backend.log <==
time="2021-03-29T10:16:15+09:00" level=error msg="unexpected error from dnssd: Timeout (-65568)"
goroutine 42834 [running]:
github.com/docker/pinata/common/pkg/dns.lookupOne(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006d0006, 0xc0005ea300, 0x26, 0x5615320, 0xc00096ecf0, 0x5615320)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup_darwin.go:26 +0x20e
github.com/docker/pinata/common/pkg/dns.doLookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x6, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x0, ...)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:90 +0x21a
github.com/docker/pinata/common/pkg/dns.distinguishNodataNxdomain(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006dfb40, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x5613760)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:58 +0x67
github.com/docker/pinata/common/pkg/dns.lookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x1c, 0xc000185680, 0x0, 0xc0006dfc78, 0x40533ac, 0x18)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:53 +0x138
github.com/docker/pinata/common/pkg/dns.(*SystemResolver).Answer(0x6210f00, 0xc000b62090, 0x25, 0xc00001001c, 0x0, 0x0, 0x0, 0x5613760, 0xc0001a3a28)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/system.go:20 +0x172
github.com/docker/pinata/common/pkg/dns.createReply(0xc000300300, 0x2, 0x2, 0xc0005f2990, 0x55f7946)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:164 +0x23d
github.com/docker/pinata/common/pkg/dns.NewServer.func1(0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:35 +0x4a
github.com/miekg/dns.HandlerFunc.ServeDNS(0xc000300340, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:37 +0x44
github.com/miekg/dns.(*ServeMux).ServeDNS(0xc000300320, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/serve_mux.go:103 +0x5d
github.com/miekg/dns.(*Server).serveDNS(0xc00024a000, 0xc000599000, 0x36, 0x200, 0xc000ab6380)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:650 +0x2fd
github.com/miekg/dns.(*Server).serveUDPPacket(0xc00024a000, 0xc00046a9b0, 0xc000599000, 0x36, 0x200, 0x5652458, 0xc0007deb60, 0x0, 0x5620910, 0xc000968b40)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:590 +0xed
created by github.com/miekg/dns.(*Server).serveUDP
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:520 +0x395

However, during the above ping or nslookup, I have confirmed that the built-in DNS server has forwarded query to the actual DNS server and GOT response successfully! See following screenshots (sorry for confusing, I used other two host name in actual tests).
image
image

@kongslund
Copy link

I experience similar behavior running macOS 11.2.3. My laptop is connected to a VPN using the built-in Cisco VPN client.

When running Docker Desktop 3.2.2, the following pull command towards an internal Artifactory instance took 26 seconds to complete for an alpine:latest image that was already up-to-date:

$ docker image pull docker.artifactory.tlt.local/alpine:latest
latest: Pulling from alpine
Digest: sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06
Status: Image is up to date for docker.artifactory.tlt.local/alpine:latest
docker.artifactory.tlt.local/alpine:latest

The Console log for process:docker shows multiple dnssd timeouts while running the above command:

Console log when pull an image

After upgrading to Docker Desktop 3.3.0, I was entirely unable to pull an image from our local Artifactory Docker repository.

$ docker image pull docker.artifactory.tlt.local/alpine:latest
Error response from daemon: Get https://docker.artifactory.tlt.local/v2/: Bad Gateway

@jjqq2013
Copy link
Author

I want to find the relevant source code of this issue and fix them, but could not find it! I often feel that the "open source" is not that open, sometimes you just can not easily find the source code.
Can there anyone tell me where can I get the source code listed in following stack trace?

==> /Users/user_name/Library/Containers/com.docker.docker/Data/log/host/com.docker.backend.log <==
time="2021-03-29T10:16:15+09:00" level=error msg="unexpected error from dnssd: Timeout (-65568)"
goroutine 42834 [running]:
github.com/docker/pinata/common/pkg/dns.lookupOne(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006d0006, 0xc0005ea300, 0x26, 0x5615320, 0xc00096ecf0, 0x5615320)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup_darwin.go:26 +0x20e
github.com/docker/pinata/common/pkg/dns.doLookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x6, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x0, ...)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:90 +0x21a
github.com/docker/pinata/common/pkg/dns.distinguishNodataNxdomain(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0xc0006dfb40, 0xc0006dfb40, 0x0, 0x0, 0x0, 0x5613760)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:58 +0x67
github.com/docker/pinata/common/pkg/dns.lookup(0x56458f8, 0xc0001b2008, 0xc000b62090, 0x25, 0x1c, 0xc000185680, 0x0, 0xc0006dfc78, 0x40533ac, 0x18)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/lookup.go:53 +0x138
github.com/docker/pinata/common/pkg/dns.(*SystemResolver).Answer(0x6210f00, 0xc000b62090, 0x25, 0xc00001001c, 0x0, 0x0, 0x0, 0x5613760, 0xc0001a3a28)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/system.go:20 +0x172
github.com/docker/pinata/common/pkg/dns.createReply(0xc000300300, 0x2, 0x2, 0xc0005f2990, 0x55f7946)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:164 +0x23d
github.com/docker/pinata/common/pkg/dns.NewServer.func1(0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/common/pkg/dns/server.go:35 +0x4a
github.com/miekg/dns.HandlerFunc.ServeDNS(0xc000300340, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:37 +0x44
github.com/miekg/dns.(*ServeMux).ServeDNS(0xc000300320, 0x5655420, 0xc000ab6380, 0xc0005f2990)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/serve_mux.go:103 +0x5d
github.com/miekg/dns.(*Server).serveDNS(0xc00024a000, 0xc000599000, 0x36, 0x200, 0xc000ab6380)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:650 +0x2fd
github.com/miekg/dns.(*Server).serveUDPPacket(0xc00024a000, 0xc00046a9b0, 0xc000599000, 0x36, 0x200, 0x5652458, 0xc0007deb60, 0x0, 0x5620910, 0xc000968b40)
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:590 +0xed
created by github.com/miekg/dns.(*Server).serveUDP
  /Users/administrator/jenkins/workspace/desktop_desktop-build_3.2.x/src/github.com/docker/pinata/vendor/github.com/miekg/dns/server.go:520 +0x395

@chichungliu42
Copy link

I have the same problem like @kongslund running macOS 10.14.6 and Docker desktop 3.3.1.

@intersailengineering
Copy link

Same problem with macOS 10.15.7 and Docker Desktop 3.3.1.
It runs without any problems downgrading to Docker Desktop 3.2.2.

@jjqq2013
Copy link
Author

jjqq2013 commented Jul 2, 2021

how is it going now? Docker Desktop 3.5.1 still has the issue.
Really hope open some part of sources, let us fix it together!

@jjqq2013 jjqq2013 changed the title vpnkit dns: extremely slow, with internal unexpected error from dnssd: Timeout (-65568) ping -4 works but ping does not work. -> vpnkit dns: extremely slow, with internal unexpected error from dnssd: Timeout (-65568) Jul 2, 2021
@grendach
Copy link

grendach commented Jul 26, 2021

Facing with the same issue as @jjqq2013
Please help to solve it.


MacOS: Mojave 10.14.6
Docker desktop: 3.5.1
Docker Engine: 20.10.7
Also when trying to pull containers from compaty artifactory, pulling is extremely slow.
Took 80 seconds to pull private_docker_image_1
Took 46 seconds to pull private_docker_image_2


Docker desktop: 2.5.0 --> working much more faster + ping also working with it.
Took 5 seconds to pull private_docker_image_1
Took 4 seconds to pull private_docker_image_2

@jjqq2013
Copy link
Author

jjqq2013 commented Aug 5, 2021

this is really an important issue. Docker 3.5.2 still has the same issue. A ping A_FQDN_IN_VPN takes 15 seconds to get first response.

@thaJeztah
Copy link
Member

/cc @djs55

@grendach
Copy link

Did anyone find out any fix for this issue so far ?

@jjqq2013
Copy link
Author

I have narrowed down this issue to the Docker’s built-in DNS server not responding to DNS AAAA query at all.

@grendach , if you can change your DNS server's configuration to answer the AAAA query, even with a clear answer "no record found", then this issue will not happen.

@docker-robott
Copy link
Collaborator

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@kongslund
Copy link

I have narrowed down this issue to the Docker’s built-in DNS server not responding to DNS AAAA query at all.

@grendach , if you can change your DNS server's configuration to answer the AAAA query, even with a clear answer "no record found", then this issue will not happen.

This is not what I experience.

On my Mac, time dig AAAA artifactory.tlt.local returns immediately with no record found.

Within an Ubuntu 20.04 container, the same command takes 15 seconds to return with connection timed out; no servers could be reached. If I bump dig's timeout to 10 seconds with time dig AAAA artifactory.tlt.local +time=10 then it returns after 15 seconds with no record found.

@kongslund
Copy link

I've come to realize that .local plays a role in this issue.

This query takes 15 seconds to return.

time dig AAAA artifactory.tlt.local +time=10

This query returns immediately when using our internal DNS a.b.c.d directly instead of going through Docker DNS. It means the issue is not with our internal DNS server.

time dig AAAA artifactory.tlt.local +time=10 @a.b.c.d

When querying something without .local, the result comes immediately.

time dig AAAA google.com +time=10

Could it have something to do with the lack of a search domain defined in /etc/resolv.conf inside the Linux VM managed by Docker Desktop? As an example, when using systemd on Ubuntu 20.04, it only resolves .local requests if a search domain is set, e.g. search tlt.local in /etc/resolv.conf.

Is there is way to configure the search domain for the Linux VM?

@docker-robott
Copy link
Collaborator

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

@docker docker locked and limited conversation to collaborators Jan 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants