Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hyperkit: external DNS resolution fails: conflict with DNS server running on host #3036

Open
makcumka2000 opened this issue Aug 1, 2018 · 42 comments
Labels
area/dns DNS issues co/hyperkit Hyperkit related issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-faq-entry Things that could use documentation in a FAQ os/macos priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@makcumka2000
Copy link

makcumka2000 commented Aug 1, 2018

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:

Environment: MacOS High Sierra

Minikube version (use minikube version): 0.25.0 and above

  • OS (e.g. from /etc/os-release): MacOS High Sierra, version 10.13.6
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperkit
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): .minikube/cache/iso/minikube.25.0.0.iso
  • Install tools: curl -#Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  • Others:

What happened: dns in minikube does not work. DNS cluster can not start, because can not download images from registry. 192.168.64.1 - it is my host IP

nslookup ya.ru
Server:    192.168.64.1
Address 1: 192.168.64.1

nslookup: can't resolve 'ya.ru'

What you expected to happen: DNS works. If I change DNS server in minikube to 8.8.8.8, it works.

su root
rm -f /etc/resolv.conf && echo nameserver 8.8.8.8 > /etc/resolv.conf
nslookup ya.ru
Server:    8.8.8.8
Address 1: 8.8.8.8 google-public-dns-a.google.com

Name:      ya.ru
Address 1: 87.250.250.242 ya.ru
Address 2: 2a02:6b8::2:242 ya.ru

How to reproduce it (as minimally and precisely as possible):

minikube stop && minikube delete
minikube start --vm-driver hyperkit

Output of minikube logs (if applicable):

Anything else do we need to know:

@Kerwood
Copy link

Kerwood commented Aug 9, 2018

I have exactly the same issue

  • OS: MacOS High Sierra, version 10.13.6
  • VM Driver: hyperkit
  • ISO: minikube-v0.28.1.iso

@maxmarkus
Copy link

maxmarkus commented Aug 24, 2018

Same here, had that issue a couple month ago alrady and just tried again after upgrading all components it is still there:

  • OS: MacOS Sierra
  • VM Driver: hyperkit
  • ISO: minikube-v0.28.2.iso

@tstromberg tstromberg added co/hyperkit Hyperkit related issues area/dns DNS issues area/networking networking issues os/macos kind/bug Categorizes issue or PR as related to a bug. labels Sep 18, 2018
@tstromberg
Copy link
Contributor

Interesting - it sounds like kube-dns is effectively broken within minikube. https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ has some tips on how to debug internal DNS service issues. Do you mind sharing the output of:

kubectl get svc --namespace=kube-system

kubectl get ep kube-dns --namespace=kube-system

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c kubedns

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c dnsmasq

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c sidecar

Thanks!

@tstromberg tstromberg added co/kube-dns Issues relating to kube-dns (not the addon) and removed area/dns DNS issues labels Sep 19, 2018
@tstromberg
Copy link
Contributor

Also, please check if the host (Mac OS) has anything listening on port 53 already, or specifically, if dnsmasq is running.

@maxmarkus
Copy link

maxmarkus commented Sep 20, 2018

In my case, I have something listening on :53 and this process running:
/usr/local/opt/dnsmasq/sbin/dnsmasq --keep-in-foreground

kubedebug.log


Update, fix to my issue:
dnsmasq was running and ignoring the dns request, since it was not listening to the right local addresses

If you are using dnsmasq in your setup and cluster creation fails (stuck at kube-dns initialization) you might need to add listen-address=127.0.0.1,192.168.64.1 to dnsmasq.conf.

Note: If dnsmasq.conf contains listen-address=127.0.0.1 kubernetes discovers dns at 127.0.0.1:53 and tries to use it using bridge ip address, but dnsmasq replies only to reqests from 127.0.0.1

My /usr/local/etc/dnsmasq.conf looks like that:

address=/kube.local/192.168.64.1
listen-address=127.0.0.1,192.168.64.1

@makcumka2000
Copy link
Author

Also, please check if the host (Mac OS) has anything listening on port 53 already, or specifically, if dnsmasq is running.

Yep, I am using dnsmasq on local host, it is neccesary for routing *.kube hosts to minikube

@maxmarkus
Copy link

It seems there is progress ongoing .. there is an open PR with a workaround: https://github.com/kubernetes/minikube/pull/3003/files

@tstromberg tstromberg changed the title DNS does not work in minikube with hyper kit driver external DNS resolution does not work with hyper kit driver Sep 27, 2018
@tstromberg tstromberg added the area/dns DNS issues label Sep 27, 2018
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 26, 2018
@tstromberg tstromberg added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. and removed more-info-needed area/networking networking issues co/kube-dns Issues relating to kube-dns (not the addon) lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 22, 2019
@jk2K
Copy link

jk2K commented Mar 16, 2019

https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver

there is a problem with the current solution, dnsmasq will fail to start when minikube is not running, 192.168.64.1 does not exist

finally I decided to stop my dnsmasq when running minikube, this is a easy way

@tstromberg tstromberg changed the title external DNS resolution does not work with hyper kit driver hyperkit: external DNS resolution fails when dnsmasq is running May 22, 2019
@tstromberg tstromberg added the r/2019q2 Issue was last reviewed 2019q2 label May 22, 2019
@lambda-9
Copy link
Contributor

lambda-9 commented Jul 12, 2019

I am using minikube using the hyperkit driver on macOS and experiencing the same problems with dnscrypt-proxy, which listens on 127.0.0.1:53 (udp+tcp). This binary is part of the Cisco Umbrella (corporate OpenDNS) roaming client. I cannot modify the configuration used by dnscrypt-proxy. The dnsmasq.conf changes linked are insufficient to resolve the issue for others.

Is there no way to apply a persistent DNS configuration change to minikube and the system pods?

~
❯ sudo lsof -i :53
Password:
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnscrypt- 44992 nobody    7u  IPv4 0x94dd190be9fa26c5      0t0  UDP localhost:domain
dnscrypt- 44992 nobody    8u  IPv4 0x94dd190bff44e9ed      0t0  TCP localhost:domain (LISTEN)

~
❯ ps ax 44992
  PID   TT  STAT      TIME COMMAND
44992   ??  Ss     0:02.69 /Library/Application Support/OpenDNS Roaming Client/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/Library/Application Support/OpenDNS Roaming Client/libdcplugin_erc.so -d

~
❯ minikube ssh -- 'cat /etc/resolv.conf; echo "Ping by Name:" && ping -v -c 4 google.com; echo "Ping by IP:" && ping -c 4 8.8.8.8'
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.64.1
Ping by Name:
ping: bad address 'google.com'
Ping by IP:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=51 time=23.489 ms
64 bytes from 8.8.8.8: seq=1 ttl=51 time=25.994 ms
64 bytes from 8.8.8.8: seq=2 ttl=51 time=27.537 ms
64 bytes from 8.8.8.8: seq=3 ttl=51 time=25.740 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 23.489/25.690/27.537 ms

@tstromberg
Copy link
Contributor

I've upped the priority based on the number of votes. Help wanted!

@medyagh medyagh removed the good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. label Sep 2, 2021
@sharifelgamal sharifelgamal added this to the 1.24.0 milestone Sep 2, 2021
@sharifelgamal sharifelgamal self-assigned this Sep 2, 2021
@spowelljr spowelljr added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Oct 13, 2021
@spowelljr spowelljr modified the milestones: 1.24.0, 1.25.0-candidate Nov 5, 2021
@martingaleh
Copy link

I imagine a lot of people will have this type of problem now that the docker desktop issue is forcing our hands. Here's my error to add to the list; hope its helpful. It seems like its' saying the same thing everyone else is saying.
minikube start --driver=hyperkit --container-runtime=docker
😄 minikube v1.24.0 on Darwin 11.4
✨ Using the hyperkit driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🏃 Updating the running hyperkit "minikube" VM ...
❗ The image 'k8s.gcr.io/kube-proxy:v1.23.2' was not found; unable to add it to cache.
❗ The image 'k8s.gcr.io/kube-controller-manager:v1.23.2' was not found; unable to add it to cache.
❗ The image 'k8s.gcr.io/kube-scheduler:v1.23.2' was not found; unable to add it to cache.
❗ The image 'k8s.gcr.io/kube-apiserver:v1.23.2' was not found; unable to add it to cache.
❗ This VM is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳 Preparing Kubernetes v1.23.2 on Docker 20.10.8 ...
❌ Unable to load cached images: loading cached images: stat /Users/kobebryant/.minikube/cache/images/k8s.gcr.io/kube-controller-manager_v1.23.2: no such file or directory

❌ Exiting due to K8S_INSTALL_FAILED: updating control plane: downloading binaries: downloading kubeadm: download failed: https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm.sha256: getter: &{Ctx:context.Background Src:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm.sha256 Dst:/Users/kobebryant/.minikube/cache/linux/v1.23.2/kubeadm.download Pwd: Mode:2 Umask:---------- Detectors:[0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8] Decompressors:map[bz2:0x70f4bb8 gz:0x70f4bb8 tar:0x70f4bb8 tar.bz2:0x70f4bb8 tar.gz:0x70f4bb8 tar.xz:0x70f4bb8 tar.zst:0x70f4bb8 tbz2:0x70f4bb8 tgz:0x70f4bb8 txz:0x70f4bb8 tzst:0x70f4bb8 xz:0x70f4bb8 zip:0x70f4bb8 zst:0x70f4bb8] Getters:map[file:0xc001177190 http:0xc000435900 https:0xc000435920] Dir:false ProgressListener:0x70b6540 Insecure:false Options:[0x5452be0]}: invalid checksum: Error downloading checksum file: bad response code: 404

@rgr-ralf-it
Copy link

In my case I had unused bind9 aside dnsmasq coming with NetworkManager

To solve it, did:

sudo apt purge bind9
sudo systemctl restart network-manager
sudo systemctl restart Network-Manager

@jackychow
Copy link

xxx:~ $ sudo lsof -i :53
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 70594 _mdnsresponder   52u  IPv6 0x33aa4c2d2a962605      0t0  UDP *:domain
mDNSRespo 70594 _mdnsresponder   53u  IPv4 0x33aa4c2d2a5abe4d      0t0  TCP *:domain (LISTEN)
mDNSRespo 70594 _mdnsresponder   54u  IPv6 0x33aa4c2d244e9b7d      0t0  TCP *:domain (LISTEN)

looks like in my case it's mDNSResponder that's using port 53?

@pasuder
Copy link

pasuder commented Jan 28, 2022

As a workaround, following change on running minikube may be done:

minikube ssh -- 'echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf'

Instead of 1.1.1.1 could be any DNS server setup.

Update: above workaround looks temporary, more persistent workaround here: #13497 (comment)

@nikolay-te
Copy link

I was having the same issue on OS X running minikube with also having dnscrypt-proxy which config I can't change.
Adding this to my /etc/pf.conf seems to have fixed it (bridge100 is the hyperkit created bridge):

rdr on bridge100 proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53

@dlitster
Copy link

dlitster commented Feb 18, 2022

@nikolay-te Discovered that pf is angry if you don't put the rule in the right order:

sudo pfctl -f /etc/pf.conf
/etc/pf.conf:24: Rules must be in order: options, normalization, queueing, translation, filtering

So I:
/etc/pf.conf

set limit {tables 10000, table-entries 400000}
scrub-anchor "cisco.anyconnect.vpn"
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr on bridge100 inet proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53
dummynet-anchor "com.apple/*"
anchor "cisco.anyconnect.vpn"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

sudo pfctl -f /etc/pf.conf

I'm on a Mac that's smothered with Cisco Umbrella sauce. BTW, I can't run minikube AT ALL unless I disconnect from my corporate VPN

This allowed my minikube VM to hit my corporate private registry url via Umbrella's DNS

I also have dnscrypt-proxy running:

ps aux | grep crypt
nobody           78236   0.0  0.0  5928120   4492   ??  SN   11:53AM   0:00.78 /opt/cisco/anyconnect/bin/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/opt/cisco/anyconnect/lib/libdcplugin_erc.so -d

@roy-work
Copy link

Can someone on this thread that doesn't have "Cisco Umbrella sauce" (sadly, that includes me) do a sudo lsof -i :53, and post the output?

In particular, I'm wondering if minikube's DNS server, whatever that is, is trying to bind to all interfaces, instead of just the bridge. If I'm looking at this right, I'm wondering why there should be a conflict here: Umbrella only binds to localhost, so a bind to the bridge shouldn't be affected by it. But, if minikube were instead trying to bind to all interfaces, then that would be a different story. The lsof from a normal setup should help answer that.

@hemchander007
Copy link

Can make this work with driver 'hyperkit' on MAC

  • minikube ssh
  • sudo su -
  • vi /etc/systemd/resolved.conf , look for #DNS= , uncomment and add entry 8.8.8.8
    like this --> DNS=8.8.8.8 save it and close
  • systemctl restart systemd-resolved

Post that in the minikube /etc/resolv.conf we will see a new entry of nameserver 8.8.8.8 along with 192.168.64.1

$ cat /etc/resolv.conf | grep name
nameserver 8.8.8.8
nameserver 192.168.64.1

NOTE : If we directly edit file "/etc/resolv.conf" with 8.8.8.8, It will revert back to old setting since its been maintained by the systemd-resolved.service .

  • now we can pull images :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dns DNS issues co/hyperkit Hyperkit related issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-faq-entry Things that could use documentation in a FAQ os/macos priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests