hyperkit dns: failed to pull image, read udp guest:45449->host:53: read: connection refused

[j4k]

Hi, I seem to get issues pulling containers from various remote hosts
minikube start --vm-driver=hyperkit

$   Downloading Kubernetes v1.14.0 images in the background ...
>   Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
-   "minikube" IP address is 192.168.64.15
-   Configuring Docker as the container runtime ...
-   Version of container runtime is 18.06.2-ce
:   Waiting for image downloads to complete ...
-   Preparing Kubernetes environment ...
-   Pulling images required by Kubernetes v1.14.0 ...
X   Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.15:45449->192.168.64.1:53: read: connection refused
, error: exit status 1
: Process exited with status 1
-   Launching Kubernetes v1.14.0 using kubeadm ...
:   Waiting for pods: apiserver proxy etcd scheduler controller dns
-   Configuring cluster permissions ...
-   Verifying component health .....
+   kubectl is now configured to use "minikube"
=   Done! Thank you for using minikube!``` 

and then: 

`eval (minikube docker-env)`
`docker pull registry`
Gives me the output
```Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.15:60324->192.168.64.1:53: read: connection refused```
If I `minikube ssh` and change /etc/resolv.conf to use 8.8.8.8 I can then pull images from dockerhub freely on the host using the docker daemon but minikube is still unable to pull them internally for a deployment

macOS Mojave Version 10.14 (18A391)
Minikube v1.0.0
Latest hyperkit

[j4k]

Another thing related - when building a go container manually using the minikube docker daemon I appear to be getting
fatal: unable to access 'https://github.com/t-tiger/gorm-bulk-insert/': Could not resolve host: github.com during go mod. Any help would be greatly appreciated!

[tstromberg]

Thanks for the bug report! I'm pretty sure this is hyperkit's DNS support misbehaving here. Do you mind sharing the output from your Mac of the following commands:

ps -afe | grep dns

ls -la /usr/local/bin/docker-machine-driver-hyperkit

and:

grep -c minikube /usr/local/bin/docker-machine-driver-hyperkit

I wouldn't be surprised if this command solves it for you, but I'm also not 100% sure it will:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

then:

minikube delete

[j4k]

ps -afe | grep dns
0 90453 245 0 9:38am ?? 0:02.33 dnssd://HP%20Color%20LaserJet%20MFP%20M477fdn%20(EF7D5C)._ipps.tcp.local./?uuid=564e424b-4b37-4237-3544-f430b9ef7d5c 131 jack Screenshot 2018-10-19 at 15.23.47 1 AP_ColorMatchingMode=AP_ApplicationColorMatching AP_D_InputSlot= collate com.apple.print.DocumentTicket.PMSpoolFormat=application/pdf com.apple.print.JobInfo.PMApplicationName=Preview com.apple.print.JobInfo.PMJobName=Screenshot\ 2018-10-19\ at\ 15.23.47 com.apple.print.JobInfo.PMJobOwner=Jack com.apple.print.PrinterInfo.PMColorDeviceID..n.=60817 com.apple.print.PrintSettings.PMColorSpaceModel..n.=2 com.apple.print.PrintSettings.PMCopies..n.=1 com.apple.print.PrintSettings.PMCopyCollate..b. com.apple.print.PrintSettings.PMFirstPage..n.=1 com.apple.print.PrintSettings.PMLastPage..n.=2147483647 com.apple.print.PrintSettings.PMPageRange..a.0..n.=1 com.apple.print.PrintSettings.PMPageRange..a.1..n.=2147483647 DestinationPrinterID=HP_Color_LaserJet_MFP_M477fdn__EF7D5C media=A4 noPDFIsProtected..b. pserrorhandler-requested=standard job-uuid=urn:uuid:61d7540a-34f7-37ce-4fa2-97728cbf2c3b job-originating-host-name=localhost date-time-at-creation= date-time-at-processing= time-at-creation=1542890756 time-at-processing=1553679506 document-name-supplied=Screenshot\ 2018-10-19\ at\ 15.23.47 job-impressions=1 com.apple.print.PrintSettings.PMTotalSidesImaged..n.=1 sides=one-sided Duplex=None com.apple.print.PrintSettings.PMTotalBeginPages..n.=1 PageSize=A4```

ls -la /usr/local/bin/docker-machine-driver-hyperkit
-rwsr-xr-x 1 root wheel 27784436 27 Mar 22:02 /usr/local/bin/docker-machine-driver-hyperkit

grep -c minikube /usr/local/bin/docker-machine-driver-hyperkit
9160

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit; sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 26.4M 100 26.4M 0 0 2588k 0 0:00:10 0:00:10 --:--:-- 3208k

minikube delete
x Deleting "minikube" from hyperkit ...

• The "minikube" cluster has been deleted.

minikube start --vm-driver=hyperkit
o minikube v1.0.0 on darwin (amd64)
$ Downloading Kubernetes v1.14.0 images in the background ...

Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...

• "minikube" IP address is 192.168.64.16
• Configuring Docker as the container runtime ...
• Version of container runtime is 18.06.2-ce
  : Waiting for image downloads to complete ...
• Preparing Kubernetes environment ...
• Pulling images required by Kubernetes v1.14.0 ...
  X Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
  stdout:
  stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.16:33279->192.168.64.1:53: read: connection refused
  , error: exit status 1
  : Process exited with status 1
• Launching Kubernetes v1.14.0 using kubeadm ...
  : Waiting for pods: apiserver proxy etcd scheduler controller dns
• Configuring cluster permissions ...
• Verifying component health .....

• kubectl is now configured to use "minikube"
  = Done! Thank you for using minikube!

eval (minikube docker-env)
skaffold dev
Generating tags...

• dynamic-pricing -> dynamic-pricing:3f7dda0-dirty
  Tags generated in 144.137359ms
  Starting build...
  Found [minikube] context, using local docker daemon.
  Building [dynamic-pricing]...
  Sending build context to Docker daemon 80.9kB
  Step 1/13 : FROM golang:1.12 as builder
  FATA[0004] exiting dev mode because first run failed: build failed: building [dynamic-pricing]: build artifact: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.16:49171->192.168.64.1:53: read: connection refused

[j4k]

Thanks for the prompt reply!

[tstromberg]

Well, there goes my theory! I'm going to de-dupe this against #3579 and we can sort it out there.

[rupokify]

Try stopping the dnsmasq service first as suggested here: #4594 (comment)

This worked for me.