Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

While on VPN, moby can't resolve private DNS #997

Closed
favoretti opened this issue Nov 30, 2016 · 25 comments

Comments

@favoretti
Copy link

commented Nov 30, 2016

Expected behavior

Moby should use the /etc/resolv.conf of the host machine

Actual behavior

It doesn't :)
Unless I log in to moby and put private DNS server ahead of 192.168.65.* block.

Information

1502FA18-9B27-4FE6-BE34-C318CB0BA4A5

Steps to reproduce the behavior

  1. Connect to a VPN that would modify your /etc/resolv.conf
  2. Try logging in into a private docker registry:
    docker login -u vlazarenko@ebay.com -p somethingfake https://registry.ecg.so/v1/
  3. Observe the following:
    Error response from daemon: Get https://registry.ecg.so/v1/users/: dial tcp: lookup registry.ecg.so on 192.168.65.1:53: no such host
@ebriney

This comment has been minimized.

Copy link
Member

commented Dec 2, 2016

It should work, we are relying on scutil --dns output to generate dns entries on moby.
We are investigating why it don't work.

@favoretti

This comment has been minimized.

Copy link
Author

commented Dec 3, 2016

FWIW, scutil --dns sees newly assigned private DNS servers ok:

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : lair
  nameserver[0] : 192.168.2.254
  nameserver[1] : fd00::5e49:79ff:fe2b:9da1
  if_index : 4 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : Reachable, Directly Reachable Address

resolver #2
  nameserver[0] : 10.250.16.37
  nameserver[1] : 10.250.16.38
  if_index : 10 (utun1)
  flags    : Scoped, Request A records
  reach    : Reachable
@favoretti

This comment has been minimized.

Copy link
Author

commented Dec 3, 2016

But even if I reboot moby while on VPN, the actual /etc/resolv.conf is missing them:

/ # cat /etc/resolv.conf
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
domain fritz.box
search lair
nameserver 192.168.65.1
# /etc/resolv.conf.tail can replace this line
/ #
@rogaha

This comment has been minimized.

Copy link

commented Dec 9, 2016

@djs55 any thoughts?

@djs55

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2016

Sorry for the delay in getting back to you. The /etc/resolv.conf in the VM points at a resolver running on the host which is part of Docker for Mac -- this means we can more easily adapt when the host network changes.

Looking at the bug report I think the following sequence happened:

  • the query for registry.ecg.so was forwarded only to 192.168.2.254
  • a "No such name" response was received from 192.168.2.254
  • the VM tried appending the search domain to create a query for registry.ecg.so.lair
  • the query for registry.ecg.so.lair was sent to both 10.250.16.37 and 10.250.16.38
  • more "No such name" responses were received

I suspect we've misinterpreted the contents of the SC database. It looks like your global/default DNS server is 192.168.2.254 while your VPN servers are 10.250.16.37 and 10.250.16.38. The VPN servers have SupplementalMatchDomains which include lair (but not so). The current logic in the DNS forwarder says:

  • if the DNS domain matches a SupplementalMatchDomains entry, send it only there
  • otherwise send it only to the other (i.e. global) servers

Could you confirm for me that registry.ecg.so resolves correctly via 10.250.16.37 and 10.250.16.38 using a command on the Mac like

dig registry.ecg.so @10.250.16.37

I'll investigate further. Thanks for the bug report!

@favoretti

This comment has been minimized.

Copy link
Author

commented Dec 13, 2016

No worries about delay whatsoever, thanks for looking into it!

Yes, registry.ecg.so resolves correctly:

[none][16:28:34] vlazarenko@laptop (~)$ dig registry.ecg.so @10.250.16.37

; <<>> DiG 9.8.3-P1 <<>> registry.ecg.so @10.250.16.37
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53130
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;registry.ecg.so.		IN	A

;; ANSWER SECTION:
registry.ecg.so.	59	IN	A	10.32.160.107

;; Query time: 200 msec
;; SERVER: 10.250.16.37#53(10.250.16.37)
;; WHEN: Tue Dec 13 18:01:23 2016
;; MSG SIZE  rcvd: 49
@favoretti

This comment has been minimized.

Copy link
Author

commented Dec 13, 2016

Just to be sure we're on the same page here: 192.168.2.254 is the 'public internet DNS'. 10.32.*.* are the private VPN DNS. Those don't have any supplementary domains, at least according to the scutil output.

@ebuildy

This comment has been minimized.

Copy link

commented Jan 3, 2017

Maybe as a workaround, will be nice to edit /etc/hosts from Docker UI or a small tutorial to explain how to deal with it.

What a shame D4M is not working in professionnal environment ;-( (I mean with VPN, custom registry etc...)

@djs55

This comment has been minimized.

Copy link
Contributor

commented Jan 3, 2017

I'm hoping to work on improved VPN support over the next few betas... this is one of the issues I'm hoping to fix. Thanks everyone for your patience so far!

@ebuildy

This comment has been minimized.

Copy link

commented Jan 3, 2017

Sure no problem! I guess you guys have a ton of work to do + manage huge community, maybe a minimalist web UI to indicate whats going on / road-map could help and prevent spam..

Do you know any good workaround or tutorial? was thinking about running a DNS (via Docker) on the machine

@favoretti

This comment has been minimized.

Copy link
Author

commented Jan 3, 2017

@ebuildy Run screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty to get moby console, then edit /etc/hosts, then hit Ctrl-A Ctrl-D to detach. Done.

@favoretti

This comment has been minimized.

Copy link
Author

commented Jan 3, 2017

@djs55 Feel free to use me as a guinea pig for SSL-based VPN if you want to verify whether fix works in this particular case :)

@ebuildy

This comment has been minimized.

Copy link

commented Jan 3, 2017

thanks @favoretti , this is what I am doing already, at home. But I manage also a team of 12 french devs (using windows, mac & linux), so I am searching for a very clean solution without any hacks to do.

I do already docker run --rm -v /etc/resolv.conf:/r debian:8 bash -c 'echo "nameserver X.X.X.X" > /r' to fix DNS server, will be helpfull to have more entreprise feature, such as configuration file well documented.

same issue to setup NFS share, this is crucial for pro. adoption.

At least, if Docker team could blog something about these small hacks, this could help a lot people to trust and adopt Docker at work.

@djs55

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2017

I've merged a potential fix for this into the master branch, which should be released as part of beta 37, due in a couple of weeks. Sorry for the delay -- I'll ping you again when there's something you can test.

@ebuildy

This comment has been minimized.

Copy link

commented Jan 12, 2017

Great to hear!

As a workaround, If we can configure daemon.json to setup a DNS it's ok, (for now, we cannot with the stable release) /:

Is windows version have the same problem?

Thanks you,

@Vanuan

This comment has been minimized.

Copy link

commented Jan 20, 2017

As Docker 1.13 was released, this has become critical. Is there a way to cherry-pick this fix to the stable version? There's no way to go back to Docker 1.12.

@djs55

This comment has been minimized.

Copy link
Contributor

commented Jan 20, 2017

The fix is in the master branch, which is due for release in the beta branch next week in beta 39 (hopefully). Once it has been in the beta for a week, it's a candidate for cherry-picking into the next stable update. We want to be very careful with stable updates.

If you want to experiment in the meantime, try following the instructions in this comment: #1103 (comment) -- the newer build of the networking component should be compatible with the stable release (although I've not tested it). Take a backup of the file before you replace it. Normally we don't recommend this kind of hybrid configuration, but if you're blocked then it's worth a shot. If you try it and it doesn't work, please upload fresh diagnostics and ping me -- there's still time to fix bugs before the next beta release.

@remear

This comment has been minimized.

Copy link

commented Jan 20, 2017

I'd like to add my experience with this so far as additional info in case it's useful.

I started experiencing this last night while on VPN using a beta32 (? i don't remember the exact version) and attempting to both push to and re-authenticate to my private registry . It just suddenly started responding with no such host messages after having been on VPN for a while and everything working without issue. I'm currently in the building, not on VPN, and have upgraded to 1.13.0-beta38 (15084), but I still experience this issue. Applying the slirp and restarting didn't have any noticeable effect.

scutil --dns results show what should be the DNS servers that have the entry for the private registry (see below). login appears to succeed on 1.13.0-beta38 (15084), push responds with no such host, when last night on the prevision version I only received a no such host response for both login and push.

DNS configuration

resolver #1
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
Reachable

resolver #2
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 4 (en0)
  flags    : Scoped, Request A records, Request AAAA records
Reachable
@semifor

This comment has been minimized.

Copy link

commented Jan 20, 2017

I had the same issue after an upgrade to 1.13. Subsequently upgraded to 1.13.0-beta38. Same problem. Followed the instructions at #1103 (comment). That resolved the issue for me. I'm using Cisco AnyConnect for access to a private data center.

@remear

This comment has been minimized.

Copy link

commented Jan 20, 2017

I also primarily use Cisco AnyConnect for VPN. When this started happening last night I created a VPN configuration via Network Preferences. It worked for a short while and then went back to no such host responses with no change in VPN connectivity. Subsequent attempts with the non-Cisco VPN configuration only resulted in no such host responses.

@ebuildy

This comment has been minimized.

Copy link

commented Jan 20, 2017

With new D4M 1.13, it looks ok ! good job guys, such a good new.

I use Viscosity as VPN client with split DNS.

@favoretti

This comment has been minimized.

Copy link
Author

commented Jan 21, 2017

Thank you guys! Works with my F5 SSL VPN as well!

@justincormack

This comment has been minimized.

Copy link
Member

commented Jan 21, 2017

Great! I will close this. We understand that there may be other VPN related issues still in different setups, but we hope this now works for most people. If you still have issues please open a new issue for the specific problem.

@Vanuan

This comment has been minimized.

Copy link

commented Jan 22, 2017

Is there a webpage listing d4m beta release history?

@bigkraig

This comment has been minimized.

Copy link

commented Jan 24, 2017

this is worse with Cisco any connect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.